Re: Rebalancing stuck, never finishes

2021-02-26 Thread Péter Sinóros-Szabó
Hey Sophie, thanks for the link, I was checking that ticket, but I was not sure if it is relevant for our case. Eventually we "fixed" our problem with reducing the session.timeout.ms (it was set to a high value for other reasons). But today, in another service, we faced the same problem when upgr

Re: Rebalancing stuck, never finishes

2021-02-26 Thread Murilo Tavares
Hi I got the same behaviour yesterday while trying to upgrade my KafkaStreams app from 2.4.1 to 2.7.0. Our brokers are on 2.2.1. Looking at KAFKA-9752 it mentions the cause being two other tickets: https://issues.apache.org/jira/browse/KAFKA-7610 https://issues.apache.org/jira/browse/KAFKA-9232 A

Re: Rebalancing stuck, never finishes

2021-02-26 Thread Murilo Tavares
Just to provide a bit more detail, I noticed Peter's pattern: "Rebalance failed. org.apache.kafka.common.errors.DisconnectException: null" "(Re-)joining group" But I also get a different pattern, interchangeably: Group coordinator broker-1:9092 (id: 2147483646 rack: null) is unavailable or invalid

high CPU usage after Kafka upgrade

2021-02-26 Thread Péter Sinóros-Szabó
Hi, I just upgraded from Kafka 2.4.1 to 2.6.1 and I see huge CPU usage on the broker after the upgrade. Upgrade in this case means that I only bumped the broker version on 1 of the brokers out of the 6 and didn't change the protocol or message format versions. Before the upgrade, it used about 35%

Re: high CPU usage after Kafka upgrade

2021-02-26 Thread Alex Woolford
It might be worth attaching a profiler to see what's eating up all the cycles, Peter. I used this recently, and it turned out that my Prometheus monitoring was the culprit: https://github.com/jvm-profiling-tools/async-profiler >From my terminal history: cd /tmp wget https://github.com/jvm-profil

Re: high CPU usage after Kafka upgrade

2021-02-26 Thread Jhanssen Fávaro
Hi Peter, I am on the same, a lot of questions about the Kafka's upgrade process. But looks like tha this CPU Increase is expected, at least while you don't finish every broker upgrade. In this case, when you say you didn't change the version, you say that for any brokers right ? Basically, you s

Re: high CPU usage after Kafka upgrade

2021-02-26 Thread Péter Sinóros-Szabó
Hi, thanks, yes I planned to run a profiler on it, Opsian to be exact, to see what's going on, but the async profiles is a good option as well. I just wanted to ask if anyone experienced this before. I will get back here if I find something useful. Peter On Fri, 26 Feb 2021 at 18:34, Alex Woolf

Re: high CPU usage after Kafka upgrade

2021-02-26 Thread Péter Sinóros-Szabó
Hi, No, CPU increase shouldn't be there. Upgrades usually bring lower CPU usage. And yes, I followed the upgrade protocol as it is described in the documentation, I got the CPU increase when I upgraded the 1st instance as the first step. Cheers, Peter On Fri, 26 Feb 2021 at 18:35, Jhanssen Fáva

Re: high CPU usage after Kafka upgrade

2021-02-26 Thread Jhanssen Fávaro
Not sure if this applies only to this 0.10 version. [image: image.png] Regards. On Fri, Feb 26, 2021 at 2:54 PM Péter Sinóros-Szabó wrote: > Hi, > > No, CPU increase shouldn't be there. Upgrades usually bring lower CPU > usage. > > And yes, I followed the upgrade protocol as it is described in

Re: Rebalancing stuck, never finishes

2021-02-26 Thread Sophie Blee-Goldman
Peter, It does seem like KAFKA-9752 is the most likely suspect, although if your clients were upgraded to 2.6.1 then I don't believe they would be on an early enough version of the JoinGroup to run into this. I'm not 100% sure though, it may be a good idea to leave a comment on that ticket and pin