Mastering Kafka Client Patterns: Part 2 – Single-Threaded Anti-Pattern

The Simplest (and Most Dangerous) Approach

Reading time: 5 minutes | Series: Part 2 of 10

---

In Part 1, we established why consumption patterns matter. Now let’s examine the pattern most developers start with—and why it becomes a production nightmare under load.

The single-threaded pattern is deceptively simple: poll for messages, process them immediately in the same thread, then commit offsets. No queues, no thread pools, no complexity. Just a straightforward loop that seems perfectly reasonable until it isn’t.

Let’s build it, benchmark it, and understand exactly why this pattern fails at scale.

---

Implementation: The Basic Single-Threaded Consumer

Here’s what the single-threaded pattern looks like in practice:

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.*;

public class SingleThreadedConsumer {
    
    private final KafkaConsumer<String, String> consumer;
    private volatile boolean running = true;
    
    public SingleThreadedConsumer(String bootstrapServers, String groupId, String topic) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 300000); // 5 minutes
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); // Manual commits
        
        this.consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList(topic));
    }
    
    public void start() {
        try {
            while (running) {
                // Step 1: Poll for messages
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
                
                // Step 2: Process each message (BLOCKING!)
                for (ConsumerRecord<String, String> record : records) {
                    processMessage(record);
                }
                
                // Step 3: Commit offsets after processing
                consumer.commitSync();
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            consumer.close();
        }
    }
    
    private void processMessage(ConsumerRecord<String, String> record) {
        try {
            // Simulate processing: database query, HTTP call, business logic, etc.
            // This is where the blocking happens!
            Thread.sleep(100); // 100ms per message
            
            // Your actual business logic here
            System.out.println("Processed: " + record.value());
            
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
    
    public void shutdown() {
        running = false;
    }
}

What’s happening here:

1. We poll Kafka every second for new messages
2. For each message, we call processMessage() which blocks the thread
3. After processing all messages in the batch, we commit offsets
4. Repeat indefinitely

This code works. It’s readable, easy to understand, and handles offset management correctly. So what’s the problem?

---

Why This Fails at Scale: The Math of Failure

Let’s run some numbers to see where this breaks down.

Scenario: Moderate Load

Assume:

• Kafka delivers 500 messages per poll
• Each message takes 100ms to process (database query, API call, etc.)
• Processing time per batch: 500 × 100ms = 50 seconds
• max.poll.interval.ms: 300 seconds (5 minutes)

Result: You’re consuming ~50 seconds out of every ~51 seconds (50s processing + 1s poll). You stay within the 5-minute timeout, but barely. Your throughput is ~600 messages per minute (10 per second).

Scenario: Real Production Load

Now let’s increase to production-level traffic:

• Kafka delivers 2,000 messages per poll (not unusual)
• Processing time: 2,000 × 100ms = 200 seconds (3.3 minutes)
• Still under the 300-second limit, but you’re now at 67% of your timeout budget

What happens when:

• Database latency spikes to 150ms? You’re at 300 seconds—timeout!
• Network hiccup adds 50ms per request? You’re over the limit
• You add a new validation step that takes 20ms? Rebalancing triggered

The Rebalancing Death Spiral

Once you violate max.poll.interval.ms, here’s what happens:

1. Kafka detects your consumer hasn’t polled in time → triggers rebalancing
2. During rebalancing (10-30 seconds), no messages are processed
3. Consumer lag increases while rebalancing occurs
4. When you rejoin the group, you now have even more messages to catch up on
5. Processing these messages takes even longer → timeout again
6. Repeat steps 1-5 indefinitely

Your consumer spends more time rebalancing than processing. Consumer lag grows exponentially. Your application is effectively down, despite technically running.

---

The Hidden Problems

Beyond the obvious timeout issues, the single-threaded pattern has other fatal flaws:

1. Throughput Ceiling

Even if you stay within timeouts, you’re limited by single-thread performance. If your processing is I/O-bound (waiting on databases, APIs, etc.), your CPU sits idle while the thread blocks. Modern applications should use concurrent processing to maximize throughput.

2. Resource Underutilization

You’re running on a server with 16 CPU cores, but only using one. Meanwhile, Kafka partitions sit ready to deliver thousands of messages per second, but you can only handle hundreds.

3. No Backpressure Handling

When load spikes, you have no mechanism to push back. You either process everything (and timeout) or fail completely. There’s no graceful degradation.

4. Difficult Error Handling

If one message fails, what do you do? Skip it and lose data? Retry and delay the entire batch? The single-threaded model offers no elegant solution.

---

When Is This Pattern Acceptable?

Despite all these issues, there are rare scenarios where single-threaded consumption is appropriate:

Valid Use Cases:

• Prototyping and development: Quick proof-of-concept work where correctness matters more than performance
• Extremely low volume: < 100 messages per second with very fast processing (< 10ms per message)
• Strict ordering requirements with tiny scale: When you absolutely must process messages sequentially and volume is minimal
• Learning and experimentation: Understanding Kafka basics before adding complexity

The Rule:

If you can process your entire poll batch in under 30 seconds consistently, single-threaded might work. Anything beyond that, and you need a better pattern.

---

The Path Forward

The single-threaded pattern fails because it couples consumption speed to processing speed. The solution is to decouple these concerns:

• Let one thread focus on consuming messages from Kafka (keep polling fast)
• Let separate worker threads handle the heavy processing
• Use a queue to coordinate between them

This is exactly what Pattern 2: Thread Pool with Blocking Queue accomplishes—which we’ll build in Part 3.

---

Key Takeaways

• ❌ Single-threaded consumption blocks on processing, risking timeouts and rebalancing
• ❌ Throughput is limited to what one thread can handle, regardless of available resources
• ❌ No graceful handling of load spikes or slow processing
• ⚠️ Only acceptable for prototypes, learning, or extremely low-volume scenarios
• ✅ Solution: Decouple consumption from processing using worker threads

The simplest pattern is not the right pattern for production. In Part 3, we’ll build the thread pool pattern that solves all these problems.

---

References

• Diagrams and illustrations created using Claude AI

What’s Next?

In Part 3, we’ll implement the Thread Pool with Blocking Queue pattern—the workhorse of production Kafka consumers. We’ll see how decoupling consumption from processing eliminates timeouts while dramatically increasing throughput.

---

📚 Series Navigation

• Part 1: Foundation
• Part 2: Single-Threaded Anti-Pattern ← You are here
• Part 3: Thread Pool Pattern (Basics)
• Part 4: Thread Pool Pattern (Advanced)
• Part 5-10: [Additional patterns and production topics]

---

Discussion: Have you experienced rebalancing loops in production? What was your message processing time before you hit issues? Share your war stories below!