Re: Kafka-streams calling subtractor with null aggregator value in KGroupedTable.reduce() and other weirdness

2018-07-13 Thread Vasily Sulatskov
Hi John, Thanks for your reply. I am not sure if this behavior I've observed is a bug or not, as I've not been resetting my application properly. On the other hand if the subtractor or adder in the reduce operation are never supposed to be called with null aggregator value, perhaps it would make s

Re: [VOTE] 2.0.0 RC2

2018-07-13 Thread Ismael Juma
Hi Rajini, Thanks for generating the RC. It seems like the kafka-streams-scala 2.12 artifact is missing from the Maven repository: https://repository.apache.org/content/groups/staging/org/apache/kafka/ Since this is the first time we are publishing this artifact, it is possible that this never w

Re: Kafka-streams calling subtractor with null aggregator value in KGroupedTable.reduce() and other weirdness

2018-07-13 Thread John Roesler
Hi again Vasily, Ok, it looks to me like this behavior is the result of the un-clean topology change. Just in case you're interested, here's what I think happened. 1. Your reduce node in subtopology1 (KSTREAM-REDUCE-02 / "table" ) internally emits pairs of "oldValue"/"newValue" . (side-n

Re: Kafka-streams calling subtractor with null aggregator value in KGroupedTable.reduce() and other weirdness

2018-07-13 Thread Vasily Sulatskov
Hi John, Thanks for your explanation. I have an answer to the practical question, i.e. a null aggregator value should be interpreted as a fatal application error. On the other hand, looking at the app topology, I see that a message from KSTREAM-REDUCE-02 / "table" goes goes to KTABLE-SEL

Re: Kafka-streams calling subtractor with null aggregator value in KGroupedTable.reduce() and other weirdness

2018-07-13 Thread John Roesler
Hi Vasily, I'm glad you're making me look at this; it's good homework for me! This is very non-obvious, but here's what happens: KStreamsReduce is a Processor of (K, V) => (K, Change) . I.e., it emits new/old Change pairs as the value. Next is the Select (aka GroupBy). In the DSL code, this is