Hi All, As you may have noticed, the message format patch for KIP-98 was merged last Friday. This patch also included the message format changes for KIP-101 and KIP-82. I wanted to mention the notable differences from our initial proposal:
1. We decided not to implement an unsigned varint type. Representing unsigned types in Java is awkward and the benefit seemed marginal. We don't really have any other instances of unsigned types in the protocol, so we thought it would be simpler and more consistent to only use signed types. 2. KIP-101 adds the partition leader epoch as a new field to the message batch header. The broker always overrides the value set by the client before appending to the log. We were concerned about the cost to recompute the CRC when this field changed, so we relocated the field and excluded it from the CRC computation. This makes it similar to the offset, which is also set by the broker and is not covered by the CRC. 3. This is not yet merged, but thanks to some work from Ismael, we intend to change the checksum algorithm to CRC32C, which will have a native interface in Java 9. Ismael has done the performance benchmarking and the results look extremely promising. See the KIP for the full format: https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging . We also wanted to share some performance results that Apurva has put together: https://docs.google.com/spreadsheets/d/1dHY6M7qCiX-NFvsgvaE0YoVdNq26uA8608XIh_DUpI4/edit#gid=61107630. Overall, it's looking good, but it's worth mentioning that this depends on disabling the record-level CRC, which we currently compute only to retain backwards compatibility with the client API: https://issues.apache.org/jira/browse/KAFKA-4935. Let us know if you have any feedback. Thanks, Jason
