Re: kafka streams re-partitioning on incoming events

2023-07-24 Thread Pushkar Deole
Thanks a lot Bruno! I am just trying the Processor API as you mentioned above, so the processor will write record to another kafka topic with new key. I am just having difficulty to read in another processor from that kafka topic and wondering if I need to create another stream with source as inte

Re: kafka streams re-partitioning on incoming events

2023-07-14 Thread Bruno Cadonna
Hi Pushkar, The events after repartitioning are processed by a different task than the task that read the events from the source topic. The task assignor assigns those tasks to stream threads. So events with the same key will be processed by the same task. As far as I understood from your earl

Re: kafka streams re-partitioning on incoming events

2023-07-14 Thread Pushkar Deole
Thanks Bruno.. What do you mean exactly with "...and then process them in that order"? By this, I mean to say if the order of events in partition will be processed after repartition. Probably I don't need to go through internal details but does the partitions of topic are again assigned to stream

Re: kafka streams re-partitioning on incoming events

2023-07-14 Thread Bruno Cadonna
Hi Pushkar, you can use repartition() for repartition your data. Method through() is actually deprecated in favor of repartition(). Before you repartition you need to specify the new key with selectKey(). What do you mean exactly with "...and then process them in that order"? The order of th

Re: kafka streams re-partitioning on incoming events

2023-07-13 Thread Pushkar Deole
Hello, *Kafka dev community, @matthiasJsax* Can you comment on below question? It is very important for us since we are getting inconsistencies due to current design On Sun, Jul 9, 2023 at 6:15 PM Pushkar Deole wrote: > Hi, > > We have a kafka streams application that consumes from multiple top