Many thanks Andras!!! That's exactly what I am looking for. Regards, Brilly
Sent from TypeApp On Jan 22, 2018, 21:37, at 21:37, Andras Beni <andrasb...@cloudera.com> wrote: >Hi Brilly, > >Do I understand right, that you want your consumers to consume only >from a >given partition / partitions based on a preconfigured value? >You need to copy the mechanism the default partitioner (if this is used >on >producer side) uses to determine partition. >( >https://github.com/apache/kafka/blob/d9b784e1470714c8b04e7c3d74f626a96ca1591e/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L68 > ) >You can do this the following way > >import org.apache.kafka.common.utils.Utils; > >String topicToConsume = ...; > >Object theKeyYouWantToConsume = ...; > >byte[] theKeyAsByteArray = yourKeySerializer.serialize(topicToConsume, >theKeyYouWantToConsume); > >int partition = Utils.toPositive(Utils.murmur2(theKeyAsByteArray)) % >partitionCount; > >consumer.assign(Collections.singletonList(new >TopicPartition(topicToConsume, >partition)))); > >However, still this way you will receive messages with different keys >that >map to the same partition. >I recommend you create separate topics for separate kinds of messages >and >rely on default partition assignment instead of producing all messages >to >the same topic and assigning partitions manually. And if you still need >to >consume all messages as if they were in one stream, you still can >subscribe >to multiple topics with a pattern. > >HTH, >Andras > > > >On Mon, Jan 22, 2018 at 3:08 AM, TSANG, Brilly ><brilly.ts...@hk.daiwacm.com> >wrote: > >> Hi folks, >> >> >> I'm working with a topic that have many messages. Kafka is scale >> horizontally. As a result, when they spread out to multiple >processes, only >> 1 will work with the specific key. The rest are not related and >should stop >> the processing. >> >> Is there a way from the client API to hash the partition number from >the >> key? With the partition number, we can stop the other processor from >> wasting cycle and start processing other events >> Regards, >> Brilly >> >> >> >> ________________________________ >> >> ************************************************************ >> ******************** >> DISCLAIMER: >> This email and any attachment(s) are intended solely for the >person(s) >> named above, and are or may contain information of a proprietary or >> confidential nature. If you are not the intended recipient(s), you >should >> delete this message immediately. Any use, disclosure or distribution >of >> this message without our prior consent is strictly prohibited. >> This message may be subject to errors, incomplete or late delivery, >> interruption, interception, modification, or may contain viruses. >Neither >> Daiwa Capital Markets Hong Kong Limited, its subsidiaries, affiliates >nor >> their officers or employees represent or warrant the accuracy or >> completeness, nor accept any responsibility or liability whatsoever >for any >> use of or reliance upon, this email or any of the contents hereof. >The >> contents of this message are for information purposes only, and >subject to >> change without notice. >> This message is not and is not intended to be an offer or >solicitation to >> buy or sell any securities or financial products, nor does any >> recommendation, opinion or advice necessarily reflect those of Daiwa >> Capital Markets Hong Kong Limited, its subsidiaries or affiliates. >> ************************************************************ >> ******************** >>
