0.10.0.1 consumers understand the older formats. So, the conversion only happens when the message format is newer than what the consumer understands. For the producer side, the conversion is not particularly costly since the data is in the heap and, if you use compression, 0.9.0.x would do recompression either way.
Ismael On Tue, Sep 19, 2017 at 2:41 PM, John Yost <hokiege...@gmail.com> wrote: > Hi Everyone, > > We recently upgraded our cluster from 0.9.0.1 to 0.10.0.1 but had to keep > our Kafka clients at 0.9.0.1. We now want to upgrade our clients and, > concurrently, the message version to 0.10.0.1. > > When we did the 0.9.0.1 -> 0.10.0.1 broker upgrade we were not able to > upgrade the kafka clients to 0.10.0.1, and we did not set the message > format. As a result the 0.9.0.1 -> 0.10.0.1 format conversion for both > incoming out outgoing messages caused our memory heap requirements to go > from 6 GB up to 12 GB, resulting in long GC pauses that caused our brokers > to crash. > > Once I explicitly set the message format to 0.9.0.1 everything was fine. > However, now that I want to convert to the 0.10.0.1 message format, I am > looking for guidance as to the best way to do this. I can switch our Kafka > clients to the 0.10.0.1 release, no problem. > > But...our challenge is that we occasionally have to replay topics, and I am > concerned that we will be back to the issue of converting outgoing messages > from 0.9.0.1 to 0.10.0.1 in these replay scenarios as well as in > inter-broker replication. > > Please confirm the best way to upgrade our clients and message format to > 0.10.0.1 while preserving the stability of our cluster as well as the > ability to replay topics that will have both 0.9.0.1 and 0.10.0.1-formatted > messages. > > Thanks > > --John >
