Hi I'm trialling Kafka Streaming for a large stream processing job, however I'm seeing message loss even in the simplest scenarios.
I've tried to boil it down to the simplest scenario where I see loss which is the following: 1. Ingest messages from an input stream (String, String) 2. Decode message into a type from JSON 3. If succesful, send to a second stream and update an atomic counter. (String, CustomType) 4. A foreach on the second stream that updates an AtomicCounter each time a message arrives. I would expect that since we have at least once guarantees that the second stream would see at least as many messages as were sent to it from the first, however I consistently see message loss. I've tested multiple times sending around 200k messages. I don't see losses every time, maybe around 1 in 5 runs with the same data. The losses are small, around 100 messages, but I would expect none. I'm running version 0.10.1.0 with Zookeeper, Kafka and the Stream Consumer all running on the same machine in order to mitigate packet loss. I'm running Ubuntu 16.04 with OpenJDK. Any advice would be greatly appreciated as I can't move forward with Kafka Streams as a solution if messages are consistently lost between stream on the same machine. Thanks Ryan
