Hello Kafka devs 👋 Colin encouraged me in the 3.9.0 RC2 thread to contribute ideas around how the protocol documentation can be improved. While I have concrete ideas on this, the current biggest issue as I see it is that new changes are not making it into documentation, and there seems to be a bit of a general lack of process with regards to this issue.
KIP-893 was a very poignant example of this. It introduces a new concept in the protocol's byte serialization format, none of which made it into documentation. This was extremely subtle and very time consuming to debug for me as an author of a third-party protocol implementation in Python that must remain compatible with Apache Kafka. Based on the view of the existing documentation, this flat out looked like a bug. The Python ecosystem solves this issue by requiring PEPs to have a "How to teach this section", forcing documentation to not be an afterthought. I am proposing to introduce the exact same concept for KIPs. I believe this will be useful for all KIPs, not just those of the sort mentioned above. For changes to the protocol, I will also suggest that it should be required for specification to be updated _before_ implementation changes are merged, but this should perhaps be a separate discussion. Forcing us to include a plan for documentation in all future KIPs is also a solid strategy to incrementally improve documentation over time. Kafka's docs are lacking also in other regards not related to the protocol. The proposed change will make sure we have a net positive development over time, towards a greater state of Kafka docs. Without first ensuring that there is a mechanism like this that makes sure documentation does not rot over time, it doesn't seem like the best investment of time to improve documentation of the current state. For the continued success of Kafka this is highly important because it is what enables a thriving ecosystem. - Have there been something similar discussed previously? - What do you all think of this proposal? BR, Anton -- [image: Aiven] <https://www.aiven.io/> *Anton Agestam* (he/him or they/them) Software Engineer, *Aiven* anton.ages...@aiven.io | +46 704 486 289 aiven.io <https://www.aiven.io/> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
