Hi Gyula, Yes, I think the semantics of the Partitioner interface is a bit off. The `numPartitions` value ideally should be the number of partitions of the `targetTopic`.
Here’s a JIRA I just filed to track the issue: https://issues.apache.org/jira/browse/FLINK-6288. Cheers, Gordon On April 10, 2017 at 1:16:18 AM, Gyula Fóra (gyula.f...@gmail.com) wrote: Hi all, We had some problems with custom partitioning for the 0.8 Kafka producer and now that I checked the code it seems there might be a problem with the logic. The producer determines the number of partitions in the open method and seems to be using that as a value passed to the custom partitioner for producing the records. This will however only work if the defaultTopicId (topic) has the same number of partitions as all other topics in the kafka cluster when producing to multiple topics. In our case the default topic had 16 and new ones have 3 as default so it gives an out of range partition error. Is my understanding correct or am I overlooking something? Thank you! Gyula
