Re: _consumer_offsets topic produce rate is dam high

2020-06-24 Thread Liam Clarke-Hutchinson
...the classpath is required to make the formatter available to the console producer consumer... On Thu, Jun 25, 2020 at 9:44 AM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > To investigate, you can read the messages on the topic to figure out how > consumer groups are committing.

Re: _consumer_offsets topic produce rate is dam high

2020-06-24 Thread Liam Clarke-Hutchinson
To investigate, you can read the messages on the topic to figure out how consumer groups are committing. They're stored in a binary format, but you can make them human readable using a formatter. Here's what I did last time to do this - the classpath is required to make the formatter available to t

Re: _consumer_offsets topic produce rate is dam high

2020-06-24 Thread Karolis Pocius
Check if any of your consumers have auto commit turned off and instead commit after processing each message. Also, even if all consumers are using auto commit, maybe some of them have the interval set to something crazy low like 1 ms. On Sat, Jun 20, 2020 at 8:31 PM Ashutosh singh wrote: > Hi G

_consumer_offsets topic produce rate is dam high

2020-06-20 Thread Ashutosh singh
Hi Guys, Hope you all are doing well. All of sudden I see very high throughput for _consumer_offsets topic. it is around 30-40K per second. What could be the reason for such a high rate ? Do I need to be concerned around this ? [image: image.png] This happens for an hour and after that it