Sorry if this mail is not send properly, but I have no idea how to send in gmail response to mailing list without having original message in mailbox.
> How does it compare to Sarama? There are several important differences. It's been about two months since I looked at sarama API and I know that things changed there, but here are the main reasons that made us decide to write our own library: Sarama is providing API build on channels. If you want to read or write message from kafka, at some point you will get two or more channels that you will have to read from. This was very problematic for us, because it's non trivial to make sure your writes are successful or not. You cannot also choose to make write of certain group of messages atomic. Kafka on the other hand provides blocking API, which is very easy to work with and understand what is the logic flow. Sarama's code is split into many files, approximately 150 lines (comments and whitespaces included) of code per file. This made it really hard for me to read the code. I believe that kafka library, having only few files that you can read top-down requires less grep and is a bit easier to read and understand. Testing is another topic that in my opinion makes great difference. This library was written from the very beginning with unit tests in mind. It is easy to write code that is using kafka connection and test it without real server running. This is possible because of public API defined as set of tiny interfaces and kafkatest package that mocks all common interfaces. As side effect of having API build from small interfaces, it is possible to wrap producer and consumer with custom functionality without need to adjust any code using it. We are using this pattern to for example transparently gather statistics about kafka usage across clients or filter messages. I have no idea if sarama is supporting this already, but kafka can write consumed messages offset into kafka which is functionality introduced in latest release. We don't really care about this functionality though, so it's not tested with real applications. Last worth mentioning thing is that if for some reason you don't like connection abstraction layer, you can use only the wire protocol implementation over any reader/writer. Hope you will find this fair and helpful comparison.
