Hi Ismael, Gotcha, will do. Okay, in reading to docs you linked, that may explain what we're seeing. When we upgraded to 0.10.0, we did not upgrade the clients from 0.9.0.1, so while the message format is the default--in this case, 0.10.0--the message format expected by the consumers is pre-0.10.0. While I am not seeing increased CPU utilization, it appears that the memory requirements for the brokers have changed with the upgrade given that I had to increase the broker memory heap size from 6 to 10GB to prevent out-of-memory errors from occurring.
Would the message format difference result in consumed and/or produced messages piling up in a buffer, and, consequently, increase the broker memory heap size requirement due to the format mismatch? That would be awesome because that means we just need to update the log.message.format.version to 0.9.0 until we upgrade the clients. --John --John On Sun, Jul 9, 2017 at 10:46 AM, Ismael Juma <ism...@juma.me.uk> wrote: > Hi John, > > Please read the upgrade documentation for the relevant versions: > > http://kafka.apache.org/documentation.html#upgrade > > Also, let's try to keep the discussion in one thread. I asked some > questions in the related "0.10.1 memory and garbage collection issues" > thread that you started. > > Ismael > > On Sun, Jul 9, 2017 at 3:30 PM, John Yost <hokiege...@gmail.com> wrote: > > > Hi Everyone, > > > > Ever since we've upgraded from 0.9.0.1 to 0.10.0 our five-node Kafka > > cluster is unstable. Specifically, whereas before a 6GB memory heap > worked > > fine, following the upgrade all five brokers crashed with out of memory > > errors within an hour of the upgrade. I boosted the memory heap to 10GB, > > which fixed the OOM error problem, but now it appears the GC pauses are > > preventing the cluster from maintaining more than one ISR. I realize I > > could up the replica lag settings to improve the ISR numbers, but that's > > treating the symptom and not the root problem. > > > > There appears to be a change in the memory requirements somewhere in the > > Kafka stack, which could be on the producer side as well, but I want to > > rule out any configuration issues on the broker side. Are there any 0.9 > > defaults in particular anyone is aware of that I should change for 0.10.x > > to resolve the root problem(s) of these observations? > > > > --John > > >
