Ah, cool, thanks Ismael! --John
On Tue, Sep 19, 2017 at 10:20 AM, Ismael Juma <ism...@juma.me.uk> wrote: > 0.10.0.1 consumers understand the older formats. So, the conversion only > happens when the message format is newer than what the consumer > understands. For the producer side, the conversion is not particularly > costly since the data is in the heap and, if you use compression, 0.9.0.x > would do recompression either way. > > Ismael > > On Tue, Sep 19, 2017 at 2:41 PM, John Yost <hokiege...@gmail.com> wrote: > > > Hi Everyone, > > > > We recently upgraded our cluster from 0.9.0.1 to 0.10.0.1 but had to keep > > our Kafka clients at 0.9.0.1. We now want to upgrade our clients and, > > concurrently, the message version to 0.10.0.1. > > > > When we did the 0.9.0.1 -> 0.10.0.1 broker upgrade we were not able to > > upgrade the kafka clients to 0.10.0.1, and we did not set the message > > format. As a result the 0.9.0.1 -> 0.10.0.1 format conversion for both > > incoming out outgoing messages caused our memory heap requirements to go > > from 6 GB up to 12 GB, resulting in long GC pauses that caused our > brokers > > to crash. > > > > Once I explicitly set the message format to 0.9.0.1 everything was fine. > > However, now that I want to convert to the 0.10.0.1 message format, I am > > looking for guidance as to the best way to do this. I can switch our > Kafka > > clients to the 0.10.0.1 release, no problem. > > > > But...our challenge is that we occasionally have to replay topics, and I > am > > concerned that we will be back to the issue of converting outgoing > messages > > from 0.9.0.1 to 0.10.0.1 in these replay scenarios as well as in > > inter-broker replication. > > > > Please confirm the best way to upgrade our clients and message format to > > 0.10.0.1 while preserving the stability of our cluster as well as the > > ability to replay topics that will have both 0.9.0.1 and > 0.10.0.1-formatted > > messages. > > > > Thanks > > > > --John > > >
