The mapping of ProducerRecord keys to TopicPartitions is configurable within the producer. As such, you wouldn't be able to ask the Cluster for a mapping since it doesn't know how the mapping was generated.
To determine the TopicPartition a given key would map to, you will need to run it through the Serializer the Producer is configured with to get the serialized byte array for the key. The DefaultPartitioner's implementation is located here <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java> and relies heavily on a Utils class located here <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L381>. The partition id for Key based partitioning is the result of Utils. toPositive(Utils.murmur2(keyBytes)) % numPartitions If you are using a custom Partitioner, it's possible you'll need additional information. The interface requires both the actual and serialized Key and Value of the ProducerRecord, as well as the topic name and internal Cluster object. It's unlikely the topic name and cluster information are used for anything other than finding the number of partitions in a topic, but it's possible for the custom Partitioner to use it in other ways. Likewise, the Value of the ProducerRecord is not likely to be used, but is available to the custom implementation. On Mon, Jan 14, 2019 at 5:54 AM Sven Ludwig <s_lud...@gmx.de> wrote: > One more question: > > Is there a way to ask Kafka which ProducerRecord.key is mapped to which > TopicPartition (for debugging purposes etc.)? > > > > Gesendet: Montag, 14. Januar 2019 um 13:49 Uhr > Von: "Sven Ludwig" <s_lud...@gmx.de> > An: users@kafka.apache.org > Betreff: Aw: Re: Re: Doubts in Kafka > Hi, > > >> ...AND deactivate the key-based log-cleaner on the > >> broker so that it does not delete older records > >> that have the same key? > > > How old records are cleaned is independent of what you do with > > processed records. You usually retain them for enough time so > > you don't loose them before processing them > > + some safety time... > > Yes, I got that. > > My wording was not sharp enough. I realized now that what I really meant > here was log compaction. But log compaction would only ever get activated > if one would set cleanup.policy=compact for a topic or perhaps as a default > for topics. So I do not have to be worried about log compaction when giving > each ProducerRecord the device UUID as key, as long as it does not get > activated. > > Glad to see that the approach seems valid. > > Thank you and Cheers! > Sven > > > Gesendet: Freitag, 11. Januar 2019 um 12:43 Uhr > Von: "Peter Levart" <peter.lev...@gmail.com> > An: users@kafka.apache.org, "Sven Ludwig" <s_lud...@gmx.de> > Betreff: Re: Aw: Re: Doubts in Kafka > > On 1/10/19 2:26 PM, Sven Ludwig wrote: > > Okay, but > > > > what if one also needs to preserve the order of messages coming from a > particular device? > > > > With Kafka, this is perhaps possible if all messages from a particular > device go into the same partition. > > > > Would it be a good and efficient solution for this approach to set the > key of each Kafka ProducerRecord to the unique ID of the Device > > Exactly! > > > AND deactivate the key-based log-cleaner on the broker so that it does > not delete older records that have the same key? > > How old records are cleaned is independent of what you do with processed > records. You usually retain them for enough time so you don't loose them > before processing them + some safety time... > > Regards, Peter > > > > > Sven > > > > > > Gesendet: Donnerstag, 10. Januar 2019 um 08:35 Uhr > > Von: "Peter Levart" <peter.lev...@gmail.com> > > An: users@kafka.apache.org, "aruna ramachandran" <arunaeie...@gmail.com> > > Betreff: Re: Doubts in Kafka > > Hi Aruna, > > > > On 1/10/19 8:19 AM, aruna ramachandran wrote: > >> I am using keyed partitions with 1000 partitions, so I need to create > 1000 > >> consumers because consumers groups and re balancing concepts is not > worked > >> in the case of manually assigned consumers.Is there any replacement for > the > >> above problem. > >> > > What API are you using in the KafkaConsumer? Are you using > > subscribe(Collection<String> topics) or are you using > > assign(Collection<TopicPartition> partitions) ? > > > > The 1st one (subscribe) is the one you should be using for your usecase. > > With that call, when you subscribe to a multi-partition topic and you > > have multiple KafkaConsumer(s) configured with the same consumer group > > id, then partitions of the topic are dynamically assigned (and possibly > > reassigned when consumers come or go) to a set of live consumers. Will > > this work for you (and why not)? > > > > Regards, Peter > >
