Kafka, partitions, consumer groups, delivery semantics
Practise these as flashcards. One or two lines each; the ones you miss come back first.
Start freeA Kafka topic is a logical channel to which records are published. Data in a topic is split into partitions, and each partition is an ordered, immutable sequence of records identified by offsets.
If a key is provided with the message, Kafka uses a hash of the key to determine the partition, ensuring ordering for that key. If no key is provided, messages are distributed round-robin across partitions.
A consumer group allows multiple consumers to share the work of reading from a topic. Each partition is consumed by only one consumer in the group at a time, enabling parallelism and scalability.
An offset is a unique identifier for each record within a partition. It tracks a consumer's position and allows resuming consumption from a specific point.
Kafka automatically elects a new leader from the in-sync replicas for that partition, ensuring continued availability and durability.
At-most-once means a message is delivered zero or one time, so some messages may be lost but never duplicated. At-least-once ensures every message is delivered at least once, so no data is lost, but duplicates may occur if processing fails after offset commit.
Kafka replicates each partition across multiple brokers. One broker is the leader for each partition, and others are followers. If the leader fails, a follower is promoted to leader, ensuring data availability and durability.
A watermark is a mechanism to track the progress of event time in stream processing. It helps determine when a window can be closed, even if some late events may still arrive.
Kafka guarantees ordering only within a partition. By using a consistent key, all related messages go to the same partition, preserving their order.
A schema registry manages and enforces schemas for data in Kafka topics, enabling producers and consumers to agree on data structure and supporting schema evolution with compatibility checks.
I would choose batch processing for use cases like nightly ETL jobs or large-scale analytics where low latency is not required, and it's more efficient to process data in bulk for simplicity and throughput.
When consumers join or leave a group, Kafka triggers a rebalance to redistribute partitions among the available consumers. This ensures load balancing but can cause temporary unavailability or duplicate processing during the transition.
acks=all in a Kafka producer?