Kafka Crash Course: Learn with a Parent's Return to Office Mandate Problem

The Problem: Companies are mandating return-to-office. Parents now face a coordination challenge:

• School bus drops kids at 3:15 PM at the community bus stop
• Parents need to be there, but meetings run over
• Group chats don’t work - messages get buried, no confirmation

Real scenario:

3:10 PM - Sarah's meeting runs over
3:11 PM - Posts in group chat: "Can someone watch Jake?"
3:15 PM - Bus arrives, no response yet

Neighbors want to help. They just need a reliable system.

Why Kafka Fits This Problem

Before: Tightly Coupled Services

Parent App → Notification Service → Database → Neighbor App

Problems:

• Notification service crashes = everything stops
• Parent waits for entire chain to respond
• Neighbor offline = message lost forever

With Kafka: Decoupled

Parent App → Kafka ← Neighbor Apps

Benefits:

• Parent sends alert, doesn’t wait
• Message stored safely in Kafka
• Neighbors read when ready (even if offline before)
• Multiple neighbors can all see it
• Add new features without breaking existing ones

Think of Kafka as a bulletin board. Pin a message, walk away. Everyone sees it. First person to help responds.

Let’s Build It

What we need:

• Docker (to run Kafka)
• Python (to write producer/consumer)

Virtual Environment Setup

1. Create the project folder and navigate to it:

mkdir bus-stop-kafka
cd bus-stop-kafka

1. Create a virtual environment:

python3 -m venv venv

1. Activate the virtual environment:

source venv/bin/activate

1. Install librdkafka (required C library for macOS):

brew install librdkafka

1. Upgrade pip and install dependencies:

pip install --upgrade pip
pip install confluent-kafka

The virtual environment is now set up and isolated from your system Python installation.

Start Kafka

Create docker-compose.yml:

version: '3.8'
services:
  kafka:
    image: confluentinc/cp-kafka:7.8.3
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_KRAFT_MODE: "true"
      CLUSTER_ID: "bus-stop-demo"
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: "/var/lib/kafka/data"
    volumes:
      - kafka-data:/var/lib/kafka/data

volumes:
  kafka-data:

Start it:

docker-compose up -d
sleep 30  # Wait for Kafka to start

Producer (Sarah Sends Alert)

Create producer.py:

from confluent_kafka import Producer
import json

# Connect to Kafka
producer = Producer({'bootstrap.servers': 'localhost:9092'})

# Create alert message
alert = {
    'parent_name': 'Sarah',
    'child_name': 'Jake',
    'location': 'Oak Street Bus Stop',
    'message': 'Meeting ran over, will be 10 mins late'
}

# Send to Kafka topic
producer.produce(
    topic='bus-stop-alerts',           # Topic name
    value=json.dumps(alert).encode()   # Convert to bytes
)

producer.flush()  # Ensure it's sent

print(f"✅ Alert sent: {alert['parent_name']} needs help")

Run it (ensure your virtual environment is activated):

python producer.py
# Output: ✅ Alert sent: Sarah needs help

What happened:

1. Connected to Kafka at localhost:9092
2. Created JSON message with alert details
3. Sent to topic called bus-stop-alerts
4. Kafka stored it

Consumer (Mike Receives Alert)

Create consumer.py:

from confluent_kafka import Consumer
import json

# Connect to Kafka
consumer = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'neighbors',              # Consumer group
    'auto.offset.reset': 'earliest'       # Read from beginning
})

consumer.subscribe(['bus-stop-alerts'])
print("🔔 Listening for alerts...\n")

try:
    while True:
        msg = consumer.poll(1.0)  # Check every second
        
        if msg is None:
            continue
        
        if msg.error():
            print(f"Error: {msg.error()}")
            continue
        
        # Got a message!
        alert = json.loads(msg.value().decode())
        
        print(f"🚨 {alert['parent_name']} needs help!")
        print(f"   Child: {alert['child_name']}")
        print(f"   Location: {alert['location']}")
        print(f"   Message: {alert['message']}\n")

except KeyboardInterrupt:
    print("Stopped")
finally:
    consumer.close()

Run it (ensure your virtual environment is activated):

python consumer.py

Output:

🔔 Listening for alerts...

🚨 Sarah needs help!
   Child: Jake
   Location: Oak Street Bus Stop
   Message: Meeting ran over, will be 10 mins late

What happened:

1. Consumer connected to Kafka
2. Subscribed to bus-stop-alerts topic
3. Read the message Sarah sent
4. Keeps running, waiting for more

Understanding Kafka Concepts

Topics

• Like folders for messages
• We used: bus-stop-alerts
• Organizes different types of messages

Producers

• Send messages to topics
• Don’t wait for consumers
• Don’t know who will read it

Consumers

• Read messages from topics
• Can start from beginning or latest
• Keep polling for new messages

Consumer Groups

• Multiple consumers with same group.id
• Kafka distributes messages among them
• Load balancing automatically

Try This: Messages Persist

Shows: Messages don’t disappear

1. Start consumer, then stop it (Ctrl+C)

2. Send 3 alerts:

   python producer.py
   python producer.py
   python producer.py
   

3. Start consumer again

Result: Consumer shows all 3 alerts!

Why this matters: If Mike’s phone was off when Sarah sent alert, he still sees it when phone turns back on.

Important: Consumer Offset Tracking

Question: “If I sent 1 alert earlier and 3 alerts now, why don’t I see all 4 alerts?”

Answer: Kafka tracks where each consumer group left off reading using offsets.

Here’s what happens:

1. First run: producer.py sends alert #1
2. First run: consumer.py reads alert #1, Kafka marks “neighbors group read up to offset 0”
3. Second run: producer.py sends alerts #2, #3, #4
4. Second run: consumer.py only shows #2, #3, #4 (skips #1 because it was already read)

This is a feature, not a bug! Imagine if neighbors saw every alert from the past month every time they checked.

To see ALL messages from the beginning:

Option 1 - Change consumer group name (line 197 in consumer.py):

'group.id': 'neighbors-v2',  # New group = starts fresh

Option 2 - Delete the consumer group offset tracking:

docker exec -it kafka kafka-consumer-groups \
  --bootstrap-server localhost:9092 \
  --delete --group neighbors

Try This: Multiple Neighbors

Shows: Multiple consumers share work

1. Open 3 terminals
2. Run python consumer.py in each (with venv activated)
3. Send alerts from 4th terminal

Result: Each consumer gets different messages (load balancing)

Why this matters: Multiple neighbors at bus stop, all see alerts, first one responds.

Important: Partitions Enable Load Balancing

Question: “All messages go to one consumer. Is load balancing actually working?”

Answer: With the default setup (1 partition), load balancing cannot work. Here’s why:

The Partition Rule:

Maximum parallel consumers = Number of partitions

By default, bus-stop-alerts has 1 partition, so:

• Consumer #1 gets partition 0 (receives all messages)
• Consumer #2 gets nothing (no partitions left)
• Consumer #3 gets nothing

To see actual load balancing:

1. Delete the topic:

   docker exec -it kafka kafka-topics \
     --bootstrap-server localhost:9092 \
     --delete --topic bus-stop-alerts
   

2. Recreate with 3 partitions:

   docker exec -it kafka kafka-topics \
     --bootstrap-server localhost:9092 \
     --create --topic bus-stop-alerts \
     --partitions 3 \
     --replication-factor 1
   

3. Run 3 consumers in separate terminals:

   python consumer.py  # Terminal 1
   python consumer.py  # Terminal 2
   python consumer.py  # Terminal 3
   

4. Send multiple alerts:

   python producer.py  # Run this 6+ times
   

Now you’ll see: Messages distributed across all 3 consumers!

Key insight: More partitions = more parallelism. This is how Kafka scales to handle massive throughput.

The Power of Kafka

Real-World Flow

Sarah (3:10 PM)
  ↓ sends alert
Kafka (stores it)
  ↓ notifies consumers
Mike (3:11 PM) - sees alert
Lisa (3:11 PM) - sees alert
David (3:12 PM) - phone was locked, sees it now
  ↓
Mike responds "I'll watch Jake"
  ↓ sends confirmation through Kafka
Sarah (3:12 PM) - sees confirmation

Why This Architecture Works

Decoupling:

• Services don’t talk directly
• Add/remove services without breaking others

Persistence:

• Messages stored on disk
• Survive crashes and restarts

Scalability:

• Add more consumers = faster processing
• Add more producers = handle more load

Reliability:

• One service down? Others keep working
• Messages don’t get lost

Real-World Use Cases

Same pattern, different problems:

E-commerce:

• Order placed → Kafka
• Payment service charges card
• Inventory service updates stock
• Email service sends confirmation

Uber:

• Ride requested → Kafka
• Driver matching finds nearby driver
• Pricing calculates fare
• Notifications alert driver

Your bus stop:

• Alert sent → Kafka
• Notification service alerts neighbors
• Database logs the event
• Analytics tracks usage

All use the same Kafka pattern you just learned.

Common Questions

“Why not just use a database?”

• Database: Consumer constantly polls “any new data?”
• Kafka: Consumer waits, Kafka notifies when ready
• Result: Real-time, less load

“Why not just use REST API?”

• REST: Consumer must be online NOW
• Kafka: Consumer reads when ready
• Result: More reliable, works offline

“When should I use Kafka?”

• ✅ High message volume
• ✅ Multiple systems need same data
• ✅ Can’t lose messages
• ✅ Need message history

What You Built

bus-stop-kafka/
├── docker-compose.yml  # Kafka setup
├── producer.py         # Send alerts
├── consumer.py         # Receive alerts
├── venv/               # Virtual environment
├── .gitignore          # Git ignore file
└── README.md           # Project documentation

Summary

You learned:

• What Kafka is (message broker)
• Why it’s useful (decoupling, persistence)
• How to produce messages
• How to consume messages
• Consumer groups concept

You built:

• Working producer that sends alerts
• Working consumer that receives alerts
• Everything runs locally with Docker

You can now:

• Explain Kafka to anyone
• Build event-driven systems
• Apply this to other problems

Resources

📦 Code: github.com/sprider/bus-stop-kafka
📚 Learn More: Kafka Docs
🎥 Watch: Nana’s Kafka Video