Why Consumption Patterns Matter
Reading time: 6 minutes | Series: Part 1 of 10
If you’ve worked with Apache Kafka in production, you’ve probably encountered this scenario: your consumer application works perfectly in testing with a few messages per second, but crumbles under real-world load. Messages pile up, consumer lag grows exponentially, and eventually your consumer group starts rebalancing repeatedly—grinding your entire processing pipeline to a halt.
The culprit? How you consume and process messages matters just as much as Kafka’s configuration itself.
Most developers start with the simplest approach: poll for messages, process them immediately in the same thread, then commit offsets. This works beautifully for demos and prototypes. But in production systems handling thousands of messages per second with complex processing logic, this naive pattern becomes your bottleneck.
This series explores five battle-tested consumption patterns that solve real production challenges. We’ll move from anti-patterns to sophisticated approaches, complete with code examples, performance benchmarks, and decision frameworks to help you choose the right pattern for your use case.
The Kafka Consumer Contract: Understanding the Basics
Before diving into patterns, let’s establish what Kafka expects from your consumer application.
The Three-Step Dance
Every Kafka consumer follows this fundamental cycle:
- Poll: Request a batch of records from Kafka brokers
- Process: Do something with those records (transform, store, forward, etc.)
- Commit: Tell Kafka which offsets you’ve successfully processed
Sounds simple, right? The complexity emerges from a critical constraint: Kafka expects regular heartbeats and timely poll intervals.
The Rebalancing Time Bomb
Your consumer participates in a consumer group, and Kafka needs to know it’s alive and making progress. Two key timeouts govern this:
max.poll.interval.ms(default: 5 minutes): Maximum time between poll() callssession.timeout.ms(default: 45 seconds): Maximum time without a heartbeat
If your processing takes too long and you miss these deadlines, Kafka assumes your consumer is dead and triggers a rebalancing—redistributing partitions across the consumer group. During rebalancing, no messages are processed. If your consumers repeatedly violate these timeouts, you enter a rebalancing loop where your application spends more time rebalancing than actually processing messages.
This is where consumption patterns become critical.
Offset Management: The Source of Truth
Offsets represent your position in each partition. Kafka offers three commit strategies:
- Auto-commit: Kafka automatically commits offsets at regular intervals (simple but risky)
- Synchronous commit: Block until Kafka acknowledges the commit (safe but slower)
- Asynchronous commit: Fire-and-forget commit (fast but requires careful error handling)
The pattern you choose determines how you manage offsets. Get this wrong, and you’ll either lose messages (committing before processing completes) or reprocess duplicates (processing without committing).
Why Simple Approaches Fail at Scale
The “poll and process in the same thread” approach fails for a fundamental reason: it couples consumption speed to processing speed.
The Blocking Problem
Consider this scenario:
- Your consumer polls 500 messages
- Each message requires a 100ms database query
- Total processing time: 50 seconds
- Your
max.poll.interval.ms: 300 seconds (5 minutes)
This works—barely. But what happens when:
- Database latency spikes to 200ms? Now you need 100 seconds.
- Message volume doubles? You’re processing 1,000 messages.
- You add additional processing logic?
You’re now at risk of violating max.poll.interval.ms, triggering rebalancing. Once rebalancing starts, you stop processing, consumer lag increases, and when you rejoin the group, you have even more messages to catch up on. This creates a death spiral.
The Throughput Ceiling
Even if you stay within timeout limits, you’re artificially capping throughput. If Kafka can deliver 10,000 messages per second but your single-threaded processing handles 1,000 per second, you’re wasting 90% of Kafka’s capacity.
Modern applications need to decouple consuming from processing—allowing the consumer to continuously poll while separate threads or async tasks handle the heavy lifting.
The Ordering Illusion
Many developers avoid concurrent processing because they believe it breaks ordering guarantees. This is a misunderstanding. Kafka guarantees ordering within a partition, not across partitions. As long as your pattern respects partition boundaries, you can process messages concurrently without violating ordering semantics.
The patterns we’ll explore show how to achieve high throughput while maintaining the ordering guarantees your application actually needs.
Overview: The Five Patterns
This series covers five consumption-processing patterns, each solving specific production challenges:
Pattern 1: Single-Threaded (The Anti-Pattern)
What it is: Poll and process in the same thread, blocking until processing completes.
When to use: Prototyping, extremely simple use cases with low volume (< 100 msg/sec), or when you literally have no alternative.
Why it fails: Couples consumption to processing speed, risks rebalancing, terrible throughput.
We’ll examine this in Part 2 to understand what NOT to do.
Pattern 2: Thread Pool with Blocking Queue
What it is: Consumer thread continuously polls and pushes to a queue; worker threads pull and process.
When to use: The default choice for most production systems. Handles high throughput while managing backpressure.
Key benefit: Decouples consumption from processing, preventing rebalancing while maximizing throughput.
Parts 3-4 deep dive into this pattern—the workhorse of Kafka consumption.
Pattern 3: Asynchronous CompletableFuture
What it is: Consumer submits messages to async processing pipelines using CompletableFuture chains.
When to use: I/O-bound processing (HTTP calls, database operations) where blocking threads is wasteful.
Key benefit: Non-blocking, efficient resource utilization, composable async operations.
Part 5 explores modern async approaches.
Pattern 4: Batch Accumulator
What it is: Collect messages into batches (by size or time window), process entire batches together.
When to use: Bulk operations like batch database inserts, analytics aggregations, or when per-message overhead is high.
Key benefit: Optimizes throughput at the cost of increased latency.
Part 6 shows when batching makes sense.
Pattern 5: Per-Partition Worker
What it is: Dedicated thread per assigned partition, each independently consuming and processing.
When to use: Stateful processing where partition ordering is critical, or when you need isolated failure handling per partition.
Key benefit: Maintains strict partition ordering, enables per-partition state management.
Part 7 covers advanced stateful scenarios.
Key Takeaways
Before moving to specific patterns, remember these fundamentals:
- Kafka expects regular polling: Violate
max.poll.interval.msand you trigger rebalancing - Decoupling is essential: Separate consumption from processing to avoid timeout issues
- Ordering is per-partition: You can process concurrently without breaking ordering guarantees
- Offset management is critical: Your pattern determines commit strategy
- One size doesn’t fit all: Different use cases need different patterns
The “simple” approach works for simple problems. Production systems handling real load need thoughtful consumption patterns.
References
- Diagrams and illustrations created using Claude AI
What’s Next?
In Part 2, we’ll implement Pattern 1 (Single-Threaded) and run benchmarks to see exactly why it fails under load. Understanding the anti-pattern helps us appreciate why the subsequent patterns exist.
Then in Part 3, we’ll build Pattern 2 (Thread Pool with Blocking Queue)—the pattern you’ll use in 80% of production scenarios.
📚 Series Navigation
- Part 1: Foundation ← You are here
- Part 2: Single-Threaded Anti-Pattern
- Part 3: Thread Pool Pattern (Basics)
- Part 4: Thread Pool Pattern (Advanced)
- Part 5-10: [Additional patterns and production topics]
Discussion: What consumption patterns are you currently using? Have you encountered rebalancing issues in production? Share your experiences in the comments below.