Aatirhassanpir commented on PR #8618: URL: https://github.com/apache/kafka/pull/8618#issuecomment-2608080365
# Title: Kafka Consumer Lag When Processing Large Messages ## Repository Version: 2.8.0 ## Key Words / Search Terms Kafka, consumer lag, large messages, message processing, performance ## Description In certain scenarios, Kafka consumers are experiencing significant lag when processing large messages. This results in delayed processing times and potential message backlog. The issue seems to be tied to specific message sizes, causing the consumers to slow down or even time out when trying to process messages that exceed a certain threshold. ## Expected Behavior Kafka consumers should be able to process messages efficiently without experiencing lag, regardless of message size. There should be no noticeable delays, and the system should be able to handle large messages seamlessly. ## Current Behavior When processing messages larger than 100MB, Kafka consumers experience an increasing lag. The lag grows with the size of the messages, resulting in delays in message consumption. This issue is particularly problematic in high-throughput systems where large messages are common. ## Steps to Reproduce (for bug reports) Set up a Kafka consumer and producer. Configure the producer to send messages larger than 100MB. Monitor the consumer's lag while it processes these messages. Observe that the consumer experiences increased lag for large messages. ##Use cases / motivating examples (for feature requests) Use case 1: A financial system that processes large transactions via Kafka. If a large transaction causes delays in message consumption, it could affect the overall system performance and client experience. Use case 2: A media company streaming large video files through Kafka. High lag during processing large video files could lead to buffering and poor user experience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
