Re: Kafka streams dropping events in join when reading from earliest message on topic

2019-06-20 Thread giselle . vandongen
It seems like there were multiple issues: 1. The two streams were read in separately: val stream1: KStream[String, String] = builder.stream[String, String](Set("topic1")) val stream2: KStream[String, String] = builder.stream[String, String](Set("topic2")) instead of together: val raw

Re: Kafka streams dropping events in join when reading from earliest message on topic

2019-06-17 Thread giselle . vandongen
I verified keys and timestamps and they match. If I start the publisher and processor at the same time, the join has entirely correct output with 6000 messages coming in and 3000 coming out. Putting the grace period to a higher value has no effect. When is the watermark for the grace period ad

Kafka streams dropping events in join when reading from earliest message on topic

2019-06-14 Thread giselle . vandongen
I have two streams of data flowing into a Kafka cluster. I want to process this data with Kafka streams. The stream producers are started at some time t. I start up the Kafka Streams job 5 minutes later and start reading from earliest from both topics (about 900 000 messages already on each topi