Producing and consuming messages in Python is commonly done using Apache Kafka. Python applications act as:<\/p>\n\n\n\n
This is a core concept in real-time data pipelines.<\/p>\n\n\n\n
The most commonly used library is:<\/p>\n\n\n\n
pip install kafka-python<\/pre>\n\n\n\nKafka Producer in Python<\/h1>\n\n\n\n
A producer sends messages to a Kafka topic.<\/p>\n\n\n\n
Example: Sending Messages<\/h2>\n\n\n\n
from kafka import KafkaProducer
import jsonproducer = KafkaProducer(
bootstrap_servers='localhost:9092',
value_serializer=lambda v: json.dumps(v).encode('utf-8')
)data = {
\"user_id\": 101,
\"action\": \"purchase\",
\"amount\": 250
}producer.send('sales_topic', value=data)
producer.flush()print(\"Message sent successfully\")<\/pre>\n\n\n\nExplanation:<\/h3>\n\n\n\n
\n
bootstrap_servers<\/code> \u2192 Kafka broker address<\/li>\n\n\n\nsales_topic<\/code> \u2192 Topic name<\/li>\n\n\n\nvalue_serializer<\/code> \u2192 Converts data to bytes<\/li>\n<\/ul>\n\n\n\nKafka Consumer in Python<\/h1>\n\n\n\n
A consumer reads messages from a Kafka topic.<\/p>\n\n\n\n
Example: Receiving Messages<\/h2>\n\n\n\n
from kafka import KafkaConsumer
import jsonconsumer = KafkaConsumer(
'sales_topic',
bootstrap_servers='localhost:9092',
auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='sales-group',
value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)for message in consumer:
print(\"Received:\", message.value)<\/pre>\n\n\n\nExplanation:<\/h3>\n\n\n\n
\n
auto_offset_reset='earliest'<\/code> \u2192 Start reading from beginning<\/li>\n\n\n\ngroup_id<\/code> \u2192 Consumer group name<\/li>\n\n\n\nvalue_deserializer<\/code> \u2192 Convert bytes back to JSON<\/li>\n<\/ul>\n\n\n\nHow It Works<\/h1>\n\n\n\n
\n
- Producer sends event to Kafka topic<\/li>\n\n\n\n
- Kafka stores event in partition<\/li>\n\n\n\n
- Consumer reads event using offset<\/li>\n\n\n\n
- Offset moves forward after processing<\/li>\n<\/ol>\n\n\n\n
Real-Time Example Use Case<\/h1>\n\n\n\n
E-commerce System:<\/p>\n\n\n\n
\n
- Producer \u2192 Sends order events<\/li>\n\n\n\n
- Consumer \u2192 Updates dashboard<\/li>\n\n\n\n
- Another consumer \u2192 Detects fraud<\/li>\n\n\n\n
- Another consumer \u2192 Stores data in warehouse<\/li>\n<\/ul>\n\n\n\n
One message, multiple consumers.<\/p>\n\n\n\n
Message Format Best Practice<\/h1>\n\n\n\n
Always send:<\/p>\n\n\n\n
\n
- JSON format<\/li>\n\n\n\n
- Include timestamp<\/li>\n\n\n\n
- Include unique ID<\/li>\n\n\n\n
- Avoid sending large payloads<\/li>\n<\/ul>\n\n\n\n
Example:<\/p>\n\n\n\n
{
\"order_id\": \"ORD1001\",
\"timestamp\": \"2026-03-03T10:30:00\",
\"amount\": 500
}<\/pre>\n\n\n\nError Handling Example<\/h1>\n\n\n\n
Producer with basic error handling:<\/p>\n\n\n\n
try:
producer.send('sales_topic', value=data)
producer.flush()
except Exception as e:
print(\"Error:\", e)<\/pre>\n\n\n\nScaling with Consumer Groups<\/h1>\n\n\n\n
If multiple consumers share the same
group_id<\/code>:<\/p>\n\n\n\n\n
- Kafka distributes partitions among them<\/li>\n\n\n\n
- Enables parallel processing<\/li>\n\n\n\n
- Improves performance<\/li>\n<\/ul>\n\n\n\n
Interview Answer (Short Version)<\/h1>\n\n\n\n
Producing and consuming messages in Python involves using the kafka-python library to send messages to Kafka topics as a producer and read them as a consumer. It enables real-time data processing and event-driven architectures.<\/p>\n\n\n\n
Final Summary<\/h1>\n\n\n\n
Producing and Consuming in Python allows you to:<\/p>\n\n\n\n
\n
- Build real-time pipelines<\/li>\n\n\n\n
- Process live events<\/li>\n\n\n\n
- Scale using consumer groups<\/li>\n\n\n\n
- Integrate with data warehouses and dashboards<\/li>\n<\/ul>\n\n\n\n
It is a fundamental skill in modern streaming and data engineering systems.<\/p>\n\n\n