Rakshit Agarwal created CAMEL-25021:
---------------------------------------
Summary: Kafka consumer silently stalls indefinitely when record
exceeds max.partition.fetch.bytes
Key: CAMEL-25021
URL: https://issues.apache.org/jira/browse/CAMEL-25021
Project: Camel
Issue Type: Bug
Components: camel-kafka
Affects Versions: 4.14.5
Reporter: Rakshit Agarwal
Fix For: 4.14.5
The camel-kafka consumer enters an infinite silent loop when a Kafka topic
contains a record larger than the consumer's max.partition.fetch.bytes (default
1MB). The consumer calls poll() repeatedly, receives empty ConsumerRecords,
processes zero records, and loops back — with no error, no log, and no
progress. The consumer appears alive but is permanently stalled.
*Root Cause*
In KafkaFetchRecords.java, the main poll loop (line 376):
\{code:java}
while (isKafkaConsumerRunnableAndNotStopped() && isConnected() &&
pollExceptionStrategy.canContinue()) {
ConsumerRecords<Object, Object> allRecords = consumer.poll(pollDuration);
ProcessingResult result =
recordProcessorFacade.processPolledRecords(allRecords);
updateTaskState();
if (result != null && result.isBreakOnErrorHit() &&
!this.state.get().equals(State.PAUSED)) {
setReconnect(true);
setConnected(false);
}
}
\{code}
When the Kafka broker encounters a record larger than
max.partition.fetch.bytes, it returns an empty fetch response — this is correct
Kafka protocol behavior, the broker cannot fit the record into the fetch
buffer. The Camel consumer receives empty ConsumerRecords, processPolledRecords
processes zero records without error, and the while loop continues.
There is no detection for the "stuck on same offset with repeated empty polls"
condition. The pollExceptionStrategy is never triggered because no exception
occurs — the poll succeeds, it just returns nothing. No code path in
KafkaFetchRecords tracks whether the consumer offset is advancing or stalled.
The KafkaConfiguration class exposes maxPartitionFetchBytes as a configurable
property, but does not set a default — inheriting the Kafka client's default of
1,048,576 bytes (1MB). This means any record >1MB published to the topic by any
producer becomes a permanent poison pill that silently stalls the consumer.
On our side, we have added a 5MB MessageSizeLimitProcessor at the application
level to reject oversized messages after they are consumed. However, this
processor is never reached for messages >1MB because the consumer stalls before
fetching them — making the application-level size check unreachable.
The only current workaround is to increase maxPartitionFetchBytes to a value
larger than the largest possible message (e.g., 50MB). This trades the silent
stall for excessive memory
consumption — the entire oversized record is fetched into consumer heap,
deserialized into a Java String, and only then rejected by application-level
checks downstream. A 50MB message consumes ~100MB of heap (byte[] + String)
before rejection.
*Steps to Reproduce*
# Create a Kafka topic with max.message.bytes set to allow large messages
(e.g., 10MB)
# Produce a message larger than 1MB (e.g., 6MB) to the topic
# Produce several normal-sized messages after the oversized one
# Start a camel-kafka consumer route with default configuration (no
maxPartitionFetchBytes override)
# Observe: the consumer processes messages up to the offset before the
oversized record, then stalls indefinitely
# Check consumer group: CURRENT-OFFSET stops advancing. Check logs: no error,
no warning at any level
*Expected Behavior*
The consumer should detect that it is stuck at the same offset receiving
consecutive empty polls and surface an error — either by throwing an exception
routable to the configured errorHandler, or by logging a WARN that clearly
identifies the stuck offset, partition, and max.partition.fetch.bytes value.
The consumer should not silently loop forever with no indication of the problem.
*Actual Behavior*
The consumer loops indefinitely with no error, no warning, and no progress. The
poll loop has no tracking of whether the offset is advancing. The
pollExceptionStrategy is never invoked because no exception occurs. TRACE-level
"Polling..." messages continue but provide no indication of the stall.
*Impact / Severity*
* Any single oversized record in any partition permanently stalls the
consumer for that partition with no log output
* Restarting the consumer does not recover — it resumes from the same stuck
offset and stalls again
* The consumer thread remains alive and the consumer group shows the member
as active — standard monitoring (thread health, consumer group lag alerts) may
not detect the stall because lag appears constant rather than growing
* No exception is surfaced to the Camel route, error handler, or application
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
