Managing Kafka consumer buffer memory/size

2023-05-31 Thread Divya Pillai
Hi Team, We have a scenario where our application is processing messages at a slower rate. We want the consumer to stop fetching messages from the broker and re-fetch when an application is ready to process again. We have fetch.max.bytes but that doesn't manage the buffer memory. If my understand

Extra bytes in headers of kafka in case data is compressed

2023-07-06 Thread Divya Pillai
Hi Team, We have encountered a scenario where when data is compressed ( any compression snappy/gzip) and consumed by the consumer then we see that in a heap dump, there is a " byte" occupying the same amount of memory as in value. Somewhere it feels like the value is copied to headers? *When is t