Re: Kafka work stealing question

Hey Gagan, If there are more partitions than consumers in a group, some consumers are allocated multiple partitions. If there are no more messages for the partitions that have been allocated to a consumer, that consumer will continue to poll until messages show up in those partitions. It won't st

Re: [EXTERNAL] Re: Health Check

Not sure if this is helpful, Tim. I recently recorded a video that shows the gist of monitoring Kafka with Prometheus: https://www.youtube.com/watch?v=nk3sk1LO7Bo Cheers, Alex On Sat, Dec 21, 2019 at 12:13 PM Liam Clarke wrote: > Nope, it's a third party project: > https://github.com/promethe

Re: Confusingly unbalanced broker

what's causing this behavior. HTH, Alex Woolford On Fri, Feb 7, 2020 at 2:38 PM Dylan Martin wrote: > Hi all! > > I have a cluster of about 20 brokers and one of them is transmitting about > 4 times as much data as the others (80mB/sec vs 20mB/sec). It has the > roughly the sam

Re: Why would all consumers pull from the same partition?

As Chandrajeet said, the default behavior is to hash on the key to assign the message to a partition. You may be able to distribute the data more evenly across the brokers by changing the partition strategy in the producer. It *is* possible to change the default behavior to, say, round-robin. That

Re: Kafka Quota help

Hey Ravi, There's a KIP to add quota management to the admin API: KIP-546 My best guess (based on searching through Slack, Google, etc...) is that this will be part of Apache Kafka

Re: high CPU usage after Kafka upgrade

It might be worth attaching a profiler to see what's eating up all the cycles, Peter. I used this recently, and it turned out that my Prometheus monitoring was the culprit: https://github.com/jvm-profiling-tools/async-profiler >From my terminal history: cd /tmp wget https://github.com/jvm-profil