Hi Derek,
What max.task.idle.ms does is set a wait time for the stream application to
wait for new messages when one or more
input topics have no messages after a poll. In your case, the application
polls for the first time and finds a message on
topic A ("input") and no messages on topic B ("table"). Since you have
max.task.idle.ms set to 5000, it waits up to five
seconds for messages to arrive on topic B. When you produce your second
message three seconds later, the app will
process the messages.
Kafka Streams will process the messages from both topics in timestamp
order, but since both of your messages have
the same timestamp, my guess is that it's processing the message from the
stream ("input") first and it finds nothing to
join to on "table". My guess is that if you tweak the second message
timestamp to be lower than the timestamp of the
second message you'll get the result you want.
Regards,
Alexandre
On Mon, Aug 29, 2022 at 11:55 AM Derek Mok <derek.mok9...@gmail.com> wrote:
> Hi, I'd like some help with understanding how exactly max.task.idle.ms
> works. I have a topology that consumes from an input topic A, and a join
> operator that enriches the topic A messages with a KTable from a compacted
> topic B. The enriched messages are output to topic C.
>
> If I set max.task.idle.ms to 5000, what should be the expected behaviour
> in
> the following scenario (assuming all messages have the same key and topics
> have same partition count):
>
> 1. Message published to A with timestamp t_0
> 2. Three seconds of wall clock time elapses
> 3. Message published to B with timestamp t_0
>
> My understanding is that a message should be output to topic C containing
> the enriched result after max.task.idle.ms elapses since it should account
> for the late producer of topic B. The join operator should only be invoked
> after max.task.idle.ms has elapsed. However, what actually happens is that
> nothing is published in topic C. I suspect my understanding of
> max.task.idle.ms is not entirely correct so would appreciate any insight
> here!
>
> Here is a small project demonstrating the above
> https://github.com/ThousandEyes-Derek/max-task-idle-ms
>
> MaxTaskIdleStreamsApp contains the streams application that describes the
> topology.
> MessageProducer produces messages into the input and compacted topics based
> on the scenario I described above.
>
> Thanks!
> Derek
>
