The magic byte is used to version message format so we'll need to make sure that check is in place--I actually don't see it in the current consumer code which I think is a bug we should fix for the next release (filed KAFKA-2523). The purpose of that field is so there is a clear check on the format rather than the scrambled scenarios Becket describes.
Also, Becket, I don't think just fixing the java client is sufficient as that would break other clients--i.e. if anyone writes a v1 messages, even by accident, any non-v1-capable consumer will break. I think we probably need a way to have the server ensure a particular message format either at read or write time. -Jay On Thu, Sep 3, 2015 at 3:47 PM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > Hi Guozhang, > > I checked the code again. Actually CRC check probably won't fail. The newly > added timestamp field might be treated as keyLength instead, so we are > likely to receive an IllegalArgumentException when try to read the key. > I'll update the KIP. > > Thanks, > > Jiangjie (Becket) Qin > > On Thu, Sep 3, 2015 at 12:48 PM, Jiangjie Qin <j...@linkedin.com> wrote: > > > Hi, Guozhang, > > > > Thanks for reading the KIP. By "old consumer", I meant the > > ZookeeperConsumerConnector in trunk now, i.e. without this bug fixed. If > we > > fix the ZookeeperConsumerConnector then it will throw exception > complaining > > about the unsupported version when it sees message format V1. What I was > > trying to say is that if we have some ZookeeperConsumerConnector running > > without the fix, the consumer will complain about CRC mismatch instead of > > unsupported version. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Thu, Sep 3, 2015 at 12:15 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > >> Thanks for the write-up Jiangjie. > >> > >> One comment about migration plan: "For old consumers, if they see the > new > >> protocol the CRC check will fail".. > >> > >> Do you mean this bug in the old consumer cannot be fixed in a > >> backward-compatible way? > >> > >> Guozhang > >> > >> > >> On Thu, Sep 3, 2015 at 8:35 AM, Jiangjie Qin <j...@linkedin.com.invalid > > > >> wrote: > >> > >> > Hi, > >> > > >> > We just created KIP-31 to propose a message format change in Kafka. > >> > > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-31+-+Message+format+change+proposal > >> > > >> > As a summary, the motivations are: > >> > 1. Avoid server side message re-compression > >> > 2. Honor time-based log roll and retention > >> > 3. Enable offset search by timestamp at a finer granularity. > >> > > >> > Feedback and comments are welcome! > >> > > >> > Thanks, > >> > > >> > Jiangjie (Becket) Qin > >> > > >> > >> > >> > >> -- > >> -- Guozhang > >> > > > > >
