This mini project will help you understand how to build a simple real-time streaming pipeline using Apache Kafka and Python.<\/p>\n\n\n\n
Project Title: Real-Time Order Processing System<\/strong><\/p>\n\n\n\n Build a streaming pipeline that:<\/p>\n\n\n\n Order Generator (Python Producer) Install Kafka and Python library:<\/p>\n\n\n\n Make sure Kafka broker is running at:<\/p>\n\n\n\n Create a topic named:<\/p>\n\n\n\n (Use Kafka CLI or your Kafka UI tool.)<\/p>\n\n\n\n This script sends a new order every 2 seconds.<\/p>\n\n\n\n You can extend the project by:<\/p>\n\n\n\n This mini project simulates:<\/p>\n\n\n\n In this streaming mini project, we simulated real-time order data using a Kafka producer and processed it with a Kafka consumer to detect high-value transactions. This demonstrates event-driven architecture and real-time data processing.<\/p>\n\n\n\n This Streaming Mini Project helps you practice:<\/p>\n\n\n\n It is a perfect beginner project for understanding modern streaming architecture in Data Engineering.<\/p>\n\n\nProject Objective<\/h1>\n\n\n\n
\n
Architecture Overview<\/h1>\n\n\n\n
\u2193
Kafka Topic (orders_topic)
\u2193
Python Consumer
\u2193
Console Output \/ Alerts<\/p>\n\n\n\nStep 1: Setup Requirements<\/h1>\n\n\n\n
pip install kafka-python<\/pre>\n\n\n\n
localhost:9092<\/pre>\n\n\n\n
Step 2: Create Kafka Topic<\/h1>\n\n\n\n
orders_topic<\/pre>\n\n\n\n
Step 3: Build Order Producer (Simulated Real-Time Data)<\/h1>\n\n\n\n
from kafka import KafkaProducer
import json
import time
import random
from datetime import datetimeproducer = KafkaProducer(
bootstrap_servers='localhost:9092',
value_serializer=lambda v: json.dumps(v).encode('utf-8')
)while True:
order = {
\"order_id\": random.randint(1000, 9999),
\"user_id\": random.randint(1, 100),
\"amount\": random.randint(50, 1000),
\"timestamp\": datetime.now().isoformat()
} producer.send('orders_topic', value=order)
print(\"Sent:\", order)
time.sleep(2)<\/pre>\n\n\n\nStep 4: Build Order Consumer<\/h1>\n\n\n\n
from kafka import KafkaConsumer
import jsonconsumer = KafkaConsumer(
'orders_topic',
bootstrap_servers='localhost:9092',
auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='order-processing-group',
value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)for message in consumer:
order = message.value
print(\"Received:\", order) if order[\"amount\"] > 800:
print(\"ALERT: High value order detected!\")<\/pre>\n\n\n\nWhat This Project Demonstrates<\/h1>\n\n\n\n
\n
Optional Enhancements<\/h1>\n\n\n\n
\n
Real-World Mapping<\/h1>\n\n\n\n
\n
Interview Explanation<\/h1>\n\n\n\n
Final Summary<\/h1>\n\n\n\n
\n