Re: Consuming from N topics "co-partitioned"

2020-01-08 Thread Matthias J. Sax
> I assume most of the work must be done at PartitionAssignor level, Correct. > there is >> any implementation of it that is able to assign partitions like this? Well, you can look into `StreamsPartitionAssignor` (but it does much more than just co-partitioning, hence, it might be hard to extra

Consuming from N topics "co-partitioned"

2020-01-08 Thread Francesco Guardiani
Hi everybody, I wish to implement a KafkaConsumer that consumes messages from N co-partitioned topics, in a similar way to KafkaStreams for join semantic. e.g. for two consumers and two topics with two partitions: C0 = {T0P0, T1P0} C1 = {T0P1, T1P1} ... Any ideas/tips? I assume most of the work