Re: KTable in Compact Topic takes too long to be updated

2020-02-20 Thread John Roesler
Aha! Thanks, Renato, that's very clear. I think there's a couple of ways you can model this, but one thing that applies to all of them is that you should consider the `max.task.idle.ms` configuration option. If you set it higher than `max.poll.interval.ms`, then Streams will be able to ensure that

Re: KTable in Compact Topic takes too long to be updated

2020-02-19 Thread Renato Melo
Hi John, Thank you for your reply. Let me clarify. I used the word aggregate, but we are using aggregate functions. Our case is a relationship whole-part between messageA and message1, 2, n. Like order and order items. So translating our case, messageA is the order and message1 and 2 are item

Re: KTable in Compact Topic takes too long to be updated

2020-02-19 Thread John Roesler
Hi Renato, Can you describe a little more about the nature of the join+aggregation logic? It sounds a little like the KTable represents the result of aggregating messages from the KStream? If that's the case, the operation you probably wanted was like: > KStream.groupBy().aggregate() which prod