Snehal beat me to it, as my suggestion would have also been to take a look at Kafka Streams. :-) Kafka Streams should be the easiest way to achieve what you're describing. Snehal's links are good starting points.
Further pointers are: https://github.com/confluentinc/examples/blob/master/kafka-streams/src/main/java/io/confluent/examples/streams/PageViewRegionLambdaExample.java https://github.com/confluentinc/examples/blob/master/kafka-streams/src/test/java/io/confluent/examples/streams/JoinLambdaIntegrationTest.java Both of these examples demonstrate how to work on topics that have the same key (here: a user id). -Michael On Wed, Jul 6, 2016 at 8:44 AM, Snehal Nagmote <nagmote.sne...@gmail.com> wrote: > Just an update, as I was reading about Kafka Streams, this functionality is > by default supported with Kafka Streams Library. > Following links are really helpful > > > http://docs.confluent.io/3.0.0/streams/developer-guide.html#partition-grouper > > https://github.com/apache/kafka/blob/0.10.0/streams/src/main/java/org/apache/kafka/streams/processor/DefaultPartitionGrouper.java > > (Kafka Stream is supported with 0.10.0) > > Thanks, > Snehal > > On 5 July 2016 at 11:47, Snehal Nagmote <nagmote.sne...@gmail.com> wrote: > > > Hello Yardena , > > > > You may want to take a look at manual assignment for partitions section > > mentioned here , > > > > > http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client > > . > > > > However I have not tried using this for multiple topics , but looking at > > api , it should be doable. > > > > You have to use same partitioning method which was used by producer to > > determine the correct partition for consumer process for multiple topics. > > > > Note that , you would lose the ordering guarantee with this approach > since > > Kafka guarantees ordering within partition for a single topic , > > > > Thanks, > > Snehal > > > > > > On 4 July 2016 at 07:50, Yardena Meymann <yarde...@feature.fm> wrote: > > > >> Hi, > >> > >> We have several topics, same number of partitions for each, same key > used > >> for all topics. > >> We also have several processes consuming the topics (one consumer > group). > >> What we wish would happen is that messages with the same key would end > up > >> consumed by the same process, regardless of the topic. > >> Can it be achieved with Kafka? What is needed for that? > >> > >> Thanks in advance, > >> Yardena > >> > > > > >
