Re: State store and input topics with different number of partition

2017-09-01 Thread Matthias J. Sax
Your observation is correct. Kafka Streams creates a task per partition. As you have a shared state store over two operator, the tasks of both input streams need to be merged to ensure co-partitioning. Thus, task0 reads topic1 partition0 and topic2 partion0, and all other task[123] only topic1 par

Re: Writing streams to kafka topic

2017-09-01 Thread Matthias J. Sax
Hi, this is not supported by the DSL layer. What you would need to do, is to add a custom stateful transform() operator after there window (`stream.groupByKey().aggregate().toStream().transform().to()`), that buffers the output and remembers the latest result. Second, you would schedule a punctuat

State store and input topics with different number of partition

2017-09-01 Thread Hugues . Deslandes
Hi, I'd like to have your comments on the problem I met while testing my app with kafka streams (0.10.2.1) Roughly, my stream app has 2 input topics : . the first one has 4 partitions (main data) . the second one has only one partition and receives messages from time to time At first, I suppo