On Fri, Apr 29, 2016 at 6:29 PM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Two questions: > > 1. My understanding based on KIP-35 is that this won't be a problem for > clients that want to support older broker versions since they will use v0 > produce requests with broken checksum to send to those, and any broker > advertising support for v1 produce requests will also support valid > checksums? In other words, the KIP is structured in terms of Java client > versions, but I'd like to make sure we have the compatibility path for > non-Java clients cleanly mapped out. (And think we do, especially given > that Dana is proposing, but still would like an ack on that.)
Yes, I'm treating these as the same: broker/client <= 0.9 messages == v0 Fetch api version <= 1 Produce api version <= 1 broker/client >= 0.10 messages >= v1 Fetch api version >= 2 Produce api version >= 2 I dont think there will be any problem for clients that want to support both encodings. > 2. We're completely disabling checksumming of the compressed payload on > consumption. Normally you'd want to validate each level of framing for > correct end-to-end validation. You could still do this (albeit more weakly) > by validating the checksum is one of the two potentially valid values > (correct checksum or old, incorrect checksum). This obviously has > computational cost. Are we sure the tradeoff we're going with makes sense? Yes, to be honest, not validating on consumption is mostly because I just haven't dug into the bowels of the java client compressor / memory records call chains. It seems non-trivial to switch validation based on the message version in the consumer code. I did not opt for the weak validation that you suggest because I think the broker should always validate v1 messages on produce, and that piece shares the same code path within the lz4 java classes. I suppose we could make the default to raise an error on checksums that fail weak validation, and then switch to strong validation in the broker. Alternately, if you have suggestions on how to wire up the consumer code to switch lz4 behavior based on message version, I would be happy to run with that. -Dana
