[
https://issues.apache.org/jira/browse/CAMEL-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrea Cosentino reassigned CAMEL-25021:
----------------------------------------
Assignee: Andrea Cosentino
> Kafka consumer silently stalls indefinitely when record exceeds
> max.partition.fetch.bytes
> -----------------------------------------------------------------------------------------
>
> Key: CAMEL-25021
> URL: https://issues.apache.org/jira/browse/CAMEL-25021
> Project: Camel
> Issue Type: Bug
> Components: camel-kafka
> Affects Versions: 4.14.5
> Reporter: Rakshit Agarwal
> Assignee: Andrea Cosentino
> Priority: Major
> Fix For: 4.14.6
>
>
> The camel-kafka consumer enters an infinite silent loop when a Kafka topic
> contains a record larger than the consumer's max.partition.fetch.bytes
> (default 1MB). The consumer calls poll() repeatedly, receives empty
> ConsumerRecords, processes zero records, and loops back — with no error, no
> log, and no progress. The consumer appears alive but is permanently stalled.
> *Root Cause*
> In KafkaFetchRecords.java, the main poll loop (line 376):
> \{code:java}
> while (isKafkaConsumerRunnableAndNotStopped() && isConnected() &&
> pollExceptionStrategy.canContinue()) {
> ConsumerRecords<Object, Object> allRecords =
> consumer.poll(pollDuration);
> ProcessingResult result =
> recordProcessorFacade.processPolledRecords(allRecords);
> updateTaskState();
> if (result != null && result.isBreakOnErrorHit() &&
> !this.state.get().equals(State.PAUSED)) {
> setReconnect(true);
> setConnected(false);
> }
> }
> \{code}
> When the Kafka broker encounters a record larger than
> max.partition.fetch.bytes, it returns an empty fetch response — this is
> correct Kafka protocol behavior, the broker cannot fit the record into the
> fetch buffer. The Camel consumer receives empty ConsumerRecords,
> processPolledRecords processes zero records without error, and the while loop
> continues.
> There is no detection for the "stuck on same offset with repeated empty
> polls" condition. The pollExceptionStrategy is never triggered because no
> exception occurs — the poll succeeds, it just returns nothing. No code path
> in KafkaFetchRecords tracks whether the consumer offset is advancing or
> stalled.
> The KafkaConfiguration class exposes maxPartitionFetchBytes as a configurable
> property, but does not set a default — inheriting the Kafka client's default
> of 1,048,576 bytes (1MB). This means any record >1MB published to the topic
> by any producer becomes a permanent poison pill that silently stalls the
> consumer.
> On our side, we have added a 5MB MessageSizeLimitProcessor at the application
> level to reject oversized messages after they are consumed. However, this
> processor is never reached for messages >1MB because the consumer stalls
> before fetching them — making the application-level size check unreachable.
> The only current workaround is to increase maxPartitionFetchBytes to a value
> larger than the largest possible message (e.g., 50MB). This trades the silent
> stall for excessive memory
> consumption — the entire oversized record is fetched into consumer heap,
> deserialized into a Java String, and only then rejected by application-level
> checks downstream. A 50MB message consumes ~100MB of heap (byte[] + String)
> before rejection.
> *Steps to Reproduce*
> # Create a Kafka topic with max.message.bytes set to allow large messages
> (e.g., 10MB)
> # Produce a message larger than 1MB (e.g., 6MB) to the topic
> # Produce several normal-sized messages after the oversized one
> # Start a camel-kafka consumer route with default configuration (no
> maxPartitionFetchBytes override)
> # Observe: the consumer processes messages up to the offset before the
> oversized record, then stalls indefinitely
> # Check consumer group: CURRENT-OFFSET stops advancing. Check logs: no
> error, no warning at any level
> *Expected Behavior*
> The consumer should detect that it is stuck at the same offset receiving
> consecutive empty polls and surface an error — either by throwing an
> exception routable to the configured errorHandler, or by logging a WARN that
> clearly identifies the stuck offset, partition, and max.partition.fetch.bytes
> value. The consumer should not silently loop forever with no indication of
> the problem.
> *Actual Behavior*
> The consumer loops indefinitely with no error, no warning, and no progress.
> The poll loop has no tracking of whether the offset is advancing. The
> pollExceptionStrategy is never invoked because no exception occurs.
> TRACE-level "Polling..." messages continue but provide no indication of the
> stall.
> *Impact / Severity*
> * Any single oversized record in any partition permanently stalls the
> consumer for that partition with no log output
> * Restarting the consumer does not recover — it resumes from the same stuck
> offset and stalls again
> * The consumer thread remains alive and the consumer group shows the member
> as active — standard monitoring (thread health, consumer group lag alerts)
> may not detect the stall because lag appears constant rather than growing
> * No exception is surfaced to the Camel route, error handler, or application
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
