It might be worth attaching a profiler to see what's eating up all the cycles, Peter.
I used this recently, and it turned out that my Prometheus monitoring was the culprit: https://github.com/jvm-profiling-tools/async-profiler >From my terminal history: cd /tmp wget https://github.com/jvm-profiling-tools/async-profiler/releases/download/v1.8.3/async-profiler-1.8.3-linux-x64.tar.gz tar xvf async-profiler-1.8.3-linux-x64.tar.gz cd async-profiler-1.8.3-linux-x64 ./profiler.sh -d 30 -f /tmp/flamegraph.svg 8983 ... where 8983 is the pid of the Kafka process. ... and then it spat out a beautiful interactive flame chart. On Fri, Feb 26, 2021 at 10:26 AM Péter Sinóros-Szabó <peter.sinoros-sz...@transferwise.com.invalid> wrote: > Hi, > > I just upgraded from Kafka 2.4.1 to 2.6.1 and I see huge CPU usage on the > broker after the upgrade. Upgrade in this case means that I only bumped the > broker version on 1 of the brokers out of the 6 and didn't change the > protocol or message format versions. Before the upgrade, it used about 35% > CPUs. After the upgrade it uses 200% but if I add two more CPUs to the > host, it is happy to use about 350%. > > I tried 2.5.1 and 2.7.0 versions too. All of those versions show the same. > > Any idea what may be wrong? > > Thanks, > Peter >
