What Is a Kafka Topic?

Quick Answer

A Kafka topic is a named, append only log where producers publish messages and consumers read them. Think of it as a category or feed channel inside Apache Kafka.

Each topic is split into one or more partitions, which is how Kafka scales across multiple servers. Inside a partition every message gets a sequential offset (an integer ID), and messages are immutable once written.

Unlike a traditional message queue, Kafka topics retain messages even after they are consumed, based on a configurable retention policy (time or size). That is what makes them so useful for stream processing.

The big idea: why Kafka uses topics

Modern systems do not process data in nightly batches anymore. Credit card swipes, ride share GPS pings, IoT sensor readings, and click stream events all flow continuously, often at millions per second. Apache Kafka was built at LinkedIn to handle that scale.

To organize all that streaming data, Kafka groups related events into topics. An e commerce platform might use topics like:

• orders_placed
• inventory_updates
• payment_failures
• user_clicks

Applications that produce events (called producers) write to a specific topic. Applications that need those events (called consumers) subscribe to the topic and read them. Many independent consumers can read the same topic at the same time.

Topics are immutable append only logs

A Kafka topic is not a database table. It is a log file: new messages are appended to the end, old messages are never modified, and reads always happen in order.

This design has three big consequences:

• Writes are extremely fast: appending to the end of a file is one of the cheapest operations a computer can do.
• Reads are sequential: consumers stream forward through the log, which is also cheap and predictable.
• Replay is free: if a downstream service crashes, it can rewind to any earlier offset and reprocess from there.

Partitions: the secret to scale

A single topic can collect more data than any one machine can hold. Kafka solves that by splitting each topic into partitions, which are independent log files that can sit on different brokers (servers) in the cluster.

Two important effects:

• Parallelism: different consumers can read different partitions simultaneously, multiplying throughput.
• Horizontal scaling: adding more brokers lets you add more partitions and absorb more load.

When you create a topic, you choose how many partitions it has. A common starting point is 6 to 12 for moderate workloads, scaling up after you measure real throughput. Over allocating partitions has real costs: more file handles, more memory, more replication overhead. Do not jump to 1000 partitions without a reason.

Offsets: how consumers track progress

Every message written to a partition gets a sequential offset starting at 0. A consumer tracks the offset of the last message it processed for each partition it reads.

If a consumer crashes and restarts, it picks up from its last committed offset rather than reprocessing everything from the beginning (unless you explicitly ask it to). This is how Kafka guarantees at least once delivery semantics out of the box, and exactly once with extra configuration.

Partition keys and message ordering

When you publish a message, you can include a key. Kafka hashes the key and uses that hash to decide which partition to write to. All messages with the same key always land in the same partition.

This matters because Kafka guarantees order within a partition but not across the whole topic. If your application needs strict ordering for a particular entity (say, all events for user 12345), use the user ID as the partition key. That way every event for that user goes to the same partition and is read in order.

Without a key, Kafka uses round robin distribution, which spreads load evenly but provides no ordering guarantee across the topic.

Retention policies

Traditional message queues like RabbitMQ delete messages once they are acknowledged. Kafka does not. Each topic has a retention policy that controls how long data sticks around.

• Time based retention: for example, keep messages for 7 days. The default is 7 days but you can set it to hours or years.
• Size based retention: for example, keep up to 50 GB per partition.
• Log compaction: a special mode where Kafka keeps only the latest message for each key, useful for maintaining current state snapshots (like a user profile table).

Retention is what lets multiple independent systems consume the same data feed at different speeds without coordinating. A fraud detection service might read in near real time, while a nightly batch job reads the same topic 12 hours later.

Replication and durability

Each partition can be replicated across multiple brokers. The replication factor (commonly 3) means each partition has 3 copies on different machines. One copy is the leader (handles reads and writes), the others are followers (stay in sync).

If a broker dies, Kafka automatically promotes a follower to leader and the cluster keeps running. This is why Kafka is used as the durable backbone of critical financial, logistics, and telemetry systems.

Consumer groups

A consumer group is a set of consumer instances that share the work of reading a topic. Kafka automatically assigns each partition to one consumer in the group, so a topic with 12 partitions can be read by up to 12 consumers in parallel.

Different consumer groups read independently. Group A might be a fraud detection service, group B might be an analytics pipeline, and both read the same topic without affecting each other.

Common design mistakes

• Too few partitions: you cap your parallelism early and have to repartition later, which is painful.
• Too many partitions: high overhead, slower failover, and metadata bloat.
• No partition key when you need ordering: events for the same entity end up scattered across partitions and arrive out of order.
• Misconfigured retention: setting retention too low means consumers that lag behind lose data permanently.
• One giant topic for everything: mixing unrelated event types into one topic complicates consumers and access control. Use separate topics per logical event type.

Where Kafka topics fit in the stack

Kafka topics typically sit between operational systems and downstream consumers: databases, search indexes, real time dashboards, machine learning pipelines, and analytics warehouses. Combined with Kafka Connect (for integrations) and Kafka Streams or Apache Flink (for processing), they form the backbone of most modern event driven architectures.

The takeaway

A Kafka topic is a named, partitioned, append only log. Producers write, consumers read, retention lets multiple readers replay at their own pace, and partitions plus keys give you the right mix of scale and ordering. Get those four ideas right and Kafka stops feeling magical and starts feeling like just the right tool for streaming data.