Kafka Demystified: Understanding Real-Time Streaming from the Ground Up

Kafka Demystified: Understanding Real-Time Streaming from the Ground Up

Introduction to Kafka

Kafka enables the processing of real-time data streams in a distributed and scalable manner. It's widely used in systems requiring reliable communication between components, especially for event-driven architectures and microservices.

Think of Kafka as a high-speed conveyor belt for data. Netflix uses it to process billions of events daily for recommendations, while Uber uses it to match riders with drivers in real time, and LinkedIn (Kafka's creator) uses it to track user activity across their platform.

However, these words sound like marketing jargon, and I prefer to deconstruct things to their basic first principles, so that is what I'm going to do. This guide outlines the essential components, explains the core concepts like brokers, producers, consumers, and topics, and dives into more complex abstractions like partitions, consumer groups, and Kafka's distributed nature.

Kafka Core Components

Let's explore the fundamental components of Kafka.

Topics and Messages

Messages in Kafka are organized into logical categories called topics. Each topic is append-only and immutable (you cannot go back and edit a message). New data is added sequentially at the end. Disks love that because this sequential write pattern is much faster than random writes, allowing Kafka to handle millions of messages per second.

There are two topics: one called "users" with two messages (John and Ed) and "Jobs" with two messages (Req1 and Req2).

Kafka Broker (Server)

The central server that receives and stores messages sent by producers and allows consumers to read the messages. Clients communicate with brokers via TCP, typically on port 9092.

Producers and Consumers

Producers connect to a broker and send messages to the Kafka broker by specifying the topic.

Consumers connect to a broker and poll messages from a topic for a given broker for processing. Consumers track their position (offset) in each partition, allowing them to resume from where they left off if they crash or restart.

• The Kafka broker hosts the topics and messages.

• Publishers push messages to the broker for a topic, and messages go to the end of the log.

• Consumers poll messages based on their current position (offset) in the partition.

Partitions

I always say the best way to work with billions of rows is to avoid working with billions of rows. Kafka is no different.

The topics in Kafka can be divided into partitions to allow data distribution and scalability. Each partition acts as a log that consumers read sequentially. Messages within partitions are identified by their offset, which is the message's position in the log.

Partitioning enables parallel processing. You can have two consumers, each processing a partition in parallel. While this allows scaling and improves reading throughput, it introduces complexity as producers and consumers must be aware of partitioning logic. A topic may have multiple partitions based on partition keys. For example, user IDs or geographic regions make good partition keys to ensure related messages stay together (e.g., users A-M in partition 0, N-Z in partition 1).

Pub/Sub vs Queue Model

Kafka supports both pub/sub and queue-based messaging patterns:

• Queue: The topic can be treated as a queue, meaning any message published to the queue will be consumed only once and removed from the queue. No two consumers can get the same message.

• Pub/Sub: The topic can be a publish-subscribe entity, meaning any message published to the topic can be consumed by different consumers.

Kafka achieves this dual functionality using consumer groups. Each consumer in a group is assigned specific partitions, ensuring messages are processed only once per group.

To simulate a queue, all consumers are placed in the same consumer group (they all have the same group name). Whereas to simulate pub/sub mode, you would place all consumers in unique group names.

Consumer Groups and Rebalancing

Each partition in a topic is assigned to one consumer in a group. When multiple consumers join the same group, Kafka rebalances partitions among them. Each partition is consumed by only one consumer in a group. This allows horizontal scaling while ensuring order within each partition is maintained. I'll demonstrate that in the code section.

Distributed Kafka Clusters and Zookeeper

Kafka operates as a distributed system with multiple brokers. Each broker can be a leader or follower for different partitions. The leader handles reads and writes, while followers replicate data for fault tolerance.

Kafka originally relied on Zookeeper to manage broker coordination, metadata, and leader election. Zookeeper tracks which broker is the leader of which partition. However, Zookeeper introduces complexity and is being phased out in newer Kafka versions. Kafka 4.0 no longer has Zookeeper.

How Kafka is distributed

When to Use Kafka

Kafka excels in scenarios requiring:

• High-throughput event streaming (processing millions of events per second)

• Log aggregation from multiple services

• Real-time analytics and monitoring

• Event-driven microservices that need reliable message delivery

However, Kafka may be overkill for simple request-response patterns, low-message volumes, or when you need complex message routing logic.

Conclusion

Now that you know the basics of Apache Kafka, you will understand its power and its limitations. That is the most important thing. Anything built on top of Kafka uses these fundamentals. Whether you're building real-time data pipelines, event-driven architectures, or distributed systems, understanding these core concepts will help you make informed decisions about when and how to use Kafka effectively.