Hi John, Yes, down conversion when consuming messages does increase JVM heap usage as we have to load the data into the JVM heap to convert it. If down conversion is not needed, we are able to send the data without copying it to the JVM heap.
Ismael On Sun, Jul 9, 2017 at 4:23 PM, John Yost <hokiege...@gmail.com> wrote: > Hi Ismael, > > Gotcha, will do. Okay, in reading to docs you linked, that may explain what > we're seeing. When we upgraded to 0.10.0, we did not upgrade the clients > from 0.9.0.1, so while the message format is the default--in this case, > 0.10.0--the message format expected by the consumers is pre-0.10.0. While I > am not seeing increased CPU utilization, it appears that the memory > requirements for the brokers have changed with the upgrade given that I had > to increase the broker memory heap size from 6 to 10GB to prevent > out-of-memory errors from occurring. > > Would the message format difference result in consumed and/or produced > messages piling up in a buffer, and, consequently, increase the broker > memory heap size requirement due to the format mismatch? That would be > awesome because that means we just need to update the > log.message.format.version to 0.9.0 until we upgrade the clients. > > --John > > --John > > On Sun, Jul 9, 2017 at 10:46 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi John, > > > > Please read the upgrade documentation for the relevant versions: > > > > http://kafka.apache.org/documentation.html#upgrade > > > > Also, let's try to keep the discussion in one thread. I asked some > > questions in the related "0.10.1 memory and garbage collection issues" > > thread that you started. > > > > Ismael > > > > On Sun, Jul 9, 2017 at 3:30 PM, John Yost <hokiege...@gmail.com> wrote: > > > > > Hi Everyone, > > > > > > Ever since we've upgraded from 0.9.0.1 to 0.10.0 our five-node Kafka > > > cluster is unstable. Specifically, whereas before a 6GB memory heap > > worked > > > fine, following the upgrade all five brokers crashed with out of memory > > > errors within an hour of the upgrade. I boosted the memory heap to > 10GB, > > > which fixed the OOM error problem, but now it appears the GC pauses are > > > preventing the cluster from maintaining more than one ISR. I realize I > > > could up the replica lag settings to improve the ISR numbers, but > that's > > > treating the symptom and not the root problem. > > > > > > There appears to be a change in the memory requirements somewhere in > the > > > Kafka stack, which could be on the producer side as well, but I want to > > > rule out any configuration issues on the broker side. Are there any 0.9 > > > defaults in particular anyone is aware of that I should change for > 0.10.x > > > to resolve the root problem(s) of these observations? > > > > > > --John > > > > > >
