Hi Miguel, Yes, the grace period is the solution to fix the problem.
Alternatively, you can try to set a higher value for "max.task.idle.ms" configuration, because this is some kind of out-of-order data. Let's say, A topic has 1 record per second (fast), B topic has 1 record per minute (slow). You can set the "max.task.idle.ms" as 60 seconds or higher, to force the stream to wait for 1 minute for the empty topic B, before processing the records. >From the document: https://kafka.apache.org/30/documentation/streams/developer-guide/config-streams#max-task-idle-ms *Any config value greater than zero indicates the number of extra milliseconds that Streams will wait if it has a caught-up but empty partition. In other words, this is the amount of time to wait for new data to be produced to the input partitions to ensure in-order processing of data in the event of a slow producer. * Hope it helps. Thank you. Luke On Thu, Dec 30, 2021 at 2:52 AM Miguel González <miguel.gonza...@klar.mx> wrote: > Hi team > > So I ran into a complicated issue, something which I believe Kafka Streams > is not prepared for. > > Basically my app is reading from two topics and joining them. > > But when testing the in my staging environment I found, one topic moves > faster than the other one basically pushing stream time forward > > Some partitions are even months apart. I found a question on SO detailing > something similar: > > https://stackoverflow.com/questions/69126351/bulk-processing-data-through-a-join-with-kafka-streams-results-in-skipping-reco > > The problem for me is that joins are no longer working. Setting a huge > grace period has somehow alleviated the problem for now but I don't that > that's the right approach and not all events join at the end anyway? Have > other users faced something similar, if so, how can it be resolved? Can we > somehow delay the processing to make them aligned some how > > thanks > - Miguel >
