Hi John, Thanks for explanation.
Yes, indeed, this is happening in my app. I also checked my state store behind the output topic and there is always the latest value. Is it possible that TestTopologyDriver can fully mimic Kafka Streams? Maybe somehow records in the output topic can be ordered by offsets? Regards, Adam. On Wed, 11 Sep 2019 at 18:09, John Roesler <j...@confluent.io> wrote: > Hi Adam, > > I haven't looked into your project yet, but just wanted to mention > this to see if it explains your observations. > > TopologyTestDriver processes every input record fully and > synchronously. So, when you pipe an input record in, it traverses all > internal processing, including intermediate topics, until all results > are written to sink topics (or the end of the topology is reached). > > This may indeed result in a different order of output records than > you'd observe with Kafka Streams, which would instead write to the > intermediate topic in one thread and then process it in a separate > thread. > > Does that add up with what you're seeing? > > Thanks, > -John > > On Wed, Sep 11, 2019 at 6:40 AM Adam Domanski <adoman...@gmail.com> wrote: > > > > Hi, > > > > I probably found a bug in TopologyTestDriver for quite non-trivial Kafka > > Streams topology. > > > > The streaming logic is the following: There is a concept of children and > > parents. Children are aggregated under parent. Some children of master > > parent can send poison pills to other parents. Such parents die but > already > > aggregated kids should go then to the master. Application remembers > killed > > parents in a local store. If a new kid comes, the store is checked and > then > > kid is rerouted to the master. > > > > I'm using intermediate topics in my topology as well as sometimes > > tombstones. > > > > Here is Gradle (Java 11 + Kotlin) based project which demonstrate the > > issue: https://github.com/czterocyty/kafka_streams_test_case > > > > You can see that few tests fail there as I expect that latest output > Parent > > records have the latest generation. > > > > So seems that TopologyTestDriver does not care about the order of output > > records. > > > > Best regards, > > Adam Domański >
