I would like to start a discussion around the work that has started in regards to KIP-30 https://cwiki.apache.org/confluence/display/KAFKA/KIP-30+-+Allow+for+brokers+to+have+plug-able+consensus+and+meta+data+storage+sub+systems
The impetus for working on this came a lot from the community. For the last year(~+) it has been the most asked question at any talk I have given (personally speaking). It has come up a bit also on the mailing list talking about zkclient vs currator. A lot of folks want to use Kafka but introducing dependencies are hard for the enterprise so the goals behind this is making it so that using Kafka can be done as easy as possible for the operations teams to-do when they do. If they are already supporting ZooKeeper they can keep doing that but if not they want (users) to use something else they are already supporting that can plug-in to-do the same things. For the core project I think we should leave in upstream what we have. This gives a great baseline regression for folks and makes the work for "making what we have plug-able work" a good defined task (carve out, layer in API impl, push back tests pass). From there then when folks want their implementation to be something besides ZooKeeper they can develop, test and support that if they choose. We would like to suggest that we have the plugin interface be Java based for minimizing depends for JVM impl. This could be in another directory something TBD /<name>. If you have a server you want to try to get it working but you aren't on the JVM don't be afraid just think about a REST impl and if you can work inside of that you have some light RPC layers (this was the first pass prototype we did to flush-out the public api presented on the KIP). There are a lot of parts to working on this and the more implementations we have the better we can flush out the public interface. I will leave the technical details and design to JIRA tickets that are linked through the confluence page as these decisions come about and code starts for reviews and we can target the specific modules having the context separate is helpful especially if multiple folks are working on it. https://issues.apache.org/jira/browse/KAFKA-2916 Do other folks want to build implementations? Maybe we should start a confluence page for those or use an existing one and add to it so we can coordinate some there to. Thanks! ~ Joe Stein - - - - - - - - - - - - - - - - - - - [image: Logo-Black.jpg] http://www.elodina.net http://www.stealth.ly - - - - - - - - - - - - - - - - - - -
