Hi, I'd like some help with understanding how exactly max.task.idle.ms
works. I have a topology that consumes from an input topic A, and a join
operator that enriches the topic A messages with a KTable from a compacted
topic B. The enriched messages are output to topic C.

If I set max.task.idle.ms to 5000, what should be the expected behaviour in
the following scenario (assuming all messages have the same key and topics
have same partition count):

   1. Message published to A with timestamp t_0
   2. Three seconds of wall clock time elapses
   3. Message published to B with timestamp t_0

My understanding is that a message should be output to topic C containing
the enriched result after max.task.idle.ms elapses since it should account
for the late producer of topic B. The join operator should only be invoked
after max.task.idle.ms has elapsed. However, what actually happens is that
nothing is published in topic C. I suspect my understanding of
max.task.idle.ms is not entirely correct so would appreciate any insight
here!

Here is a small project demonstrating the above
https://github.com/ThousandEyes-Derek/max-task-idle-ms

MaxTaskIdleStreamsApp contains the streams application that describes the
topology.
MessageProducer produces messages into the input and compacted topics based
on the scenario I described above.

Thanks!
Derek

Reply via email to