That is puzzling to me, too. Could it be that you have `null` keys for the "new topic" you mentioned in your original email? For `null` keys, the fallback would be round-robin.
Or you just got lucky and the keys you write get distributed evenly "by chance" -- in general, if the data is not skewed, hash partitioning should result in a fairly even distribution, too. -Matthias On 11/15/19 1:21 AM, Mikkel Gadegaard wrote: > Well it definitely gives me something to move ahead with. > > I am however puzzled how I could observe a really even distribution over > the partitions when specifying `PARTITIONER_CLASS_CONFIG`, whereas when I > remove it the same set of test messages are written to only one partition. > > Thanks > Mikkel > > -- > > > On Fri, Nov 15, 2019 at 12:22 AM Matthias J. Sax <matth...@confluent.io> > wrote: > >> In Kafka Streams the producer config `PARTITIONER_CLASS_CONFIG` does not >> take effect, because Kafka Streams computes and set partition numbers >> explicitly and thus the producer does never use the partitioner to >> compute a partition, but accepts whatever Kafka Streams specifies on >> each `ProducerRecord`. >> >> If you want to change the partitioning strategy, you need to implement a >> custom `StreamPartitioner` and pass it into the corresponding methods. >> For example, `to()` or `through()`. >> >> Hope this helps. >> >> >> -Matthias >> >> On 11/14/19 9:51 AM, Mikkel Gadegaard wrote: >>> I've set up a POC using KafkaStreams with microservices consuming and >>> producing from/to topics. In the beginning I hadn't thought about >>> partition strategy, and so I was using the DefaultPartitioner for >> producer >>> partition assignments. My messages have keys (I use these for >>> forking/joining), and the keys are time based UUIDs, this causes some >>> rather uneven distribution on my topics. I looked around google and >>> stumbled on KIP-369 (Alternative Partitioner to Support "Always >>> Round-Robin" Selection) and figured that would be what I needed, so since >>> 2.4 isn't out yet I borrowed the class from the PR on github, added it to >>> my project and added the property to my config, like so: >>> >>> streamProperties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, >>> RoundRobinPartitioner.class.getCanonicalName()); >>> >>> >>> And the round robin strategy works on a newly added topic, spreading >>> messages evenly over 4 partitions. But, and I'm finally getting to my >>> question, it doesn't seem to have any effect on existing topics, in other >>> words, it seems to be continuing to use the DefaultPartitioner for topics >>> created before I added the RoundRobinPartioner class to my >>> project/properties. >>> >>> Is it me that just hasn't understood that it is impossible to change >>> strategy for an existing partition or do I have to do something specific >>> apart from re-deploying the Microservice containing the producer? >>> >>> Thanks >>> Mikkel >>> >>> -- >>> >> >> >
signature.asc
Description: OpenPGP digital signature
