This mini project will help you understand how to build a simple real-time streaming pipeline using Apache Kafka and Python.

Project Title: Real-Time Order Processing System

Project Objective

Build a streaming pipeline that:

• Simulates live order data
• Sends data to Kafka (Producer)
• Processes data in real time (Consumer)
• Prints alerts for high-value orders

Architecture Overview

Order Generator (Python Producer)
↓
Kafka Topic (orders_topic)
↓
Python Consumer
↓
Console Output / Alerts

Step 1: Setup Requirements

Install Kafka and Python library:

pip install kafka-python

Make sure Kafka broker is running at:

localhost:9092

Step 2: Create Kafka Topic

Create a topic named:

orders_topic

(Use Kafka CLI or your Kafka UI tool.)

Step 3: Build Order Producer (Simulated Real-Time Data)

from kafka import KafkaProducer
import json
import time
import random
from datetime import datetimeproducer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)while True:
    order = {
        "order_id": random.randint(1000, 9999),
        "user_id": random.randint(1, 100),
        "amount": random.randint(50, 1000),
        "timestamp": datetime.now().isoformat()
    }    producer.send('orders_topic', value=order)
    print("Sent:", order)
    time.sleep(2)

This script sends a new order every 2 seconds.

Step 4: Build Order Consumer

from kafka import KafkaConsumer
import jsonconsumer = KafkaConsumer(
    'orders_topic',
    bootstrap_servers='localhost:9092',
    auto_offset_reset='earliest',
    enable_auto_commit=True,
    group_id='order-processing-group',
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)for message in consumer:
    order = message.value
    print("Received:", order)    if order["amount"] > 800:
        print("ALERT: High value order detected!")

What This Project Demonstrates

• Real-time message production
• Event streaming using Kafka
• Real-time processing logic
• Alert system implementation
• Consumer groups

Optional Enhancements

You can extend the project by:

• Storing orders in a database
• Sending alerts via email
• Writing processed data to a file
• Adding multiple consumers
• Creating a dashboard
• Deploying to cloud

Real-World Mapping

This mini project simulates:

• E-commerce live orders
• Fraud detection systems
• Financial transaction monitoring
• Live business dashboards

Interview Explanation

In this streaming mini project, we simulated real-time order data using a Kafka producer and processed it with a Kafka consumer to detect high-value transactions. This demonstrates event-driven architecture and real-time data processing.

Final Summary

This Streaming Mini Project helps you practice:

• Kafka basics
• Producer and consumer implementation
• Real-time data simulation
• Event-driven logic

It is a perfect beginner project for understanding modern streaming architecture in Data Engineering.