Distributed Systems 6 min read

Kafka: Start Simple, Then Scale the Idea

Published on Mar 15, 2026 By Berkan Çetinkaya

Imagine a simple system.

A user performs an action, maybe they place an order, complete a payment, or sign up for an account.

One service processes that action and then calls another service to continue the workflow.

For small systems, this works perfectly well.

Something happens, and another service reacts.

When One Event Triggers Many Actions

As systems grow, a single event often needs to trigger multiple reactions.

Take a simple order event as an example.

When an order is created, several things might need to happen:

reserve inventory
charge the payment
send a confirmation email
update analytics dashboards
trigger shipment preparation

Now the architecture starts looking like this:

flowchart LR
    O["Order Service"]
    I["Inventory"]
    P["Payment"]
    E["Email"]
    A["Analytics"]
    S["Shipping"]

    O --> I
    O --> P
    O --> E
    O --> A
    O --> S

This introduces a new problem.

The service that produces the event now needs to know about every other service in the system.

That creates tight coupling, which makes systems harder to evolve.

Decoupling with a Queue

A common solution is to introduce a queue.

Instead of calling every service directly, the system publishes an event and other services consume it asynchronously.

This improves the architecture significantly.

Now the producer simply says:

"This event happened."

And other services decide whether they care about it.

flowchart LR
    O["Order Service"] --> Q["Queue"]
    Q --> I["Inventory Worker"]
    Q --> P["Payment Worker"]
    Q --> E["Email Worker"]

But traditional queues introduce another limitation.

The Problem with Disappearing Messages

In many queue systems, once a message is consumed, it disappears.

That works well for background jobs.

But it becomes problematic when multiple independent systems want to process the same event.

For example:

analytics pipelines
fraud detection
auditing systems
notifications

All of them may need the same event.

This is where a different idea becomes useful.

A Different Mental Model: The Event Log

Instead of deleting messages after consumption, imagine storing them in a log.

Every event is appended to the end of that log, and consumers can read it independently.

Consumers are no longer competing for messages.

They are simply reading the same history.

This idea is the foundation of Kafka.

Kafka is essentially a distributed append-only event log.

flowchart LR
    E1["Event 1"] --> E2["Event 2"] --> E3["Event 3"] --> E4["Event 4"]
    C1["Analytics Consumer"] --> E1
    C2["Audit Consumer"] --> E1
    C3["Notification Consumer"] --> E1

Organizing Events with Topics

As systems grow, events naturally fall into categories.

For example:

order events
payment events
shipment events
user activity events

Kafka organizes these streams using topics.

Topics help organize streams of events so that systems only consume what they need.

Scaling Throughput with Partitions

As event volume increases, a single log becomes a bottleneck.

Kafka solves this by splitting topics into partitions.

Each partition is an ordered sequence of events.

Different partitions can be processed in parallel, allowing Kafka to scale horizontally.

Ordering is guaranteed within a partition, but not across the entire topic.

flowchart TB
    T["orders topic"]
    T --> P0["partition 0"]
    T --> P1["partition 1"]
    T --> P2["partition 2"]

    P0 --> A0["events ordered"]
    P1 --> A1["events ordered"]
    P2 --> A2["events ordered"]

Scaling Consumers with Consumer Groups

Eventually, one consumer may not be enough to process all incoming events.

Kafka solves this with consumer groups.

Each partition is assigned to a single consumer in the group.

This allows systems to scale horizontally while avoiding duplicate processing.

flowchart LR
    P0["partition 0"] --> C1["consumer A"]
    P1["partition 1"] --> C2["consumer B"]
    P2["partition 2"] --> C3["consumer C"]

Offsets: How Consumers Track Progress

Kafka consumers do not remove messages from the log.

Instead, they keep track of their position using offsets.

Offsets allow consumers to resume processing exactly where they left off.

Event Replay

Because events remain in the log, consumers can replay past events.

This makes Kafka extremely useful for:

analytics pipelines
event sourcing
debugging production issues
rebuilding downstream systems

flowchart LR
    Log["Kafka Log"]
    Off1["Offset 120"]
    Off2["Replay from Offset 40"]
    Log --> Off1
    Log --> Off2

Not Everything Should Be Asynchronous

Not every operation in a system should be event-driven.

Consider user registration.

A user submits a registration form and expects an immediate response.

This flow is synchronous.

However, once the transaction succeeds, the system may publish an event.

Other services can then react independently.

A useful rule is:

Commands are synchronous.
Events are asynchronous.

flowchart LR
    U["User"] --> R["Register API"]
    R --> DB["User DB"]
    R --> OK["HTTP 200 / success"]
    R --> EV["user_registered event"]
    EV --> N["Notifications"]
    EV --> A["Analytics"]

Kafka vs Traditional Message Queues

Kafka is often compared to other messaging systems.

But they solve slightly different problems.

Feature	Kafka	RabbitMQ	NATS
Model	Event log	Message queue	Pub/sub
Message retention	Yes	Usually deleted	Often transient
Replay events	Yes	Limited	Limited
Throughput	Extremely high	Moderate	Very high
Best for	Event streaming	Background jobs	Low latency messaging

Kafka focuses on durable event streams, while many traditional queues focus on task distribution.

Consumer Rebalancing

Kafka consumer groups automatically rebalance when consumers join or leave.

Initially:

flowchart LR
    P0["partition 0"] --> C1["consumer A"]
    P1["partition 1"] --> C1
    P2["partition 2"] --> C1

If another worker joins:

flowchart LR
    P0["partition 0"] --> C1["consumer A"]
    P1["partition 1"] --> C2["consumer B"]
    P2["partition 2"] --> C1

Kafka redistributes partitions to keep the workload balanced.

The Operational Challenge

Despite its strengths, Kafka can be difficult to operate.

Running Kafka in production often means managing:

multiple brokers
replication
partition balancing
consumer rebalancing
storage management

In many organizations, Kafka is managed by a dedicated infrastructure team.

Simplifying the Model: Tansu

Because Kafka can be operationally complex, new systems are emerging to simplify its setup.

One example is Tansu.

Tansu aims to provide Kafka compatibility while simplifying operations.

Instead of tightly coupling storage and broker logic, it separates them and can rely on external storage systems such as:

object storage
databases
cloud infrastructure

The goal is to keep the Kafka programming model while reducing operational overhead.

Final Thoughts

Kafka may appear complicated at first.

But its core idea is actually very simple:

store events as a durable log and allow systems to process them independently.

Once systems grow large enough, this model solves problems that traditional queues struggle with:

multiple consumers
event replay
large-scale throughput
failure recovery

What begins as a simple idea, an event happens, eventually becomes the backbone of modern event-driven systems.