Hello Jan, Kafka Streams does implement the repartitioning for certain join / aggregate cases, such that the joining streams and tables will be co-partitioned by the specific keys. This is abstracted from the users that code with the high-level Streams DSL.
Since Kafka topic partitions are replicated themselves, usually users do not need to send a record to multiple partitions for replication (admit that currently only the partition leader can take writes AND reads, but we are also considering to let followers also handle reads in the future). Guozhang On Sat, Mar 12, 2016 at 3:59 PM, Jan Filipiak <jan.filip...@trivago.com> wrote: > Hi, > > I am very exited about all of this in general. Sadly I haven’t had the > time to really take a deep look. One thing that is/was always a difficult > topic to resolve many to many relationships in table x table x table joins > is the repartitioning that has to happen at some point. > > From the documentation I saw this: > > "The *keys* of data records determine the partitioning of data in both > Kafka and Kafka Streams, i.e. how data is routed to specific partitions > within topics." > > This feels unnecessarily restrictive as i can't currently imagin how to > resolve many to many relationships with this. One can also emmit every > record to many partitions to make up for no read replicas in kafka aswell > as partitioning schemes that don't work like this (Shards processing > overlapping key spaces). > > I would really love to hear your thoughts on these topics. Great work! > Google grade technologies for everyone! > I <3 logs > > > > > On 10.03.2016 22:26, Jay Kreps wrote: > >> Hey all, >> >> Lot's of people have probably seen the ongoing work on Kafka Streams >> happening. There is no real way to design a system like this in a vacuum, >> so we put up a blog, some snapshot docs, and something you can download >> and >> use easily to get feedback: >> >> >> http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple >> >> We'd love comments or thoughts from anyone... >> >> -Jay >> >> > -- -- Guozhang
