No worries, we have already pushed the fix for our jobs on a custom build :)
Have a safe trip! Gyula Tzu-Li (Gordon) Tai <tzuli...@apache.org> ezt írta (időpont: 2017. ápr. 11., K, 23:11): > That workaround should work, yes. > The proper fix would also be something similar I guess, only just exposing > extra APIs to properly provide different partitioners for different topics. > > Btw, sorry for the slow responses, as I’m also currently traveling for the > Flink Forward conference in San Francisco. > > Cheers, > Gordon > > On April 10, 2017 at 7:05:10 AM, Gyula Fóra (gyula.f...@gmail.com) wrote: > > In the worst case scenario we will have a custom build that will just cache > the different partition numbers in a map. (But still call partitioner.open > only once) > I think this simple intermediate fix would actually be good enough for most > people who get blocked by this in the short run. > > Gyula > > Gyula Fóra <gyula.f...@gmail.com> ezt írta (időpont: 2017. ápr. 10., H, > 16:01): > > > I understand the reasoning, on the other hand this creates a problem that > > is very hard to work around. :/ > > > > Do you have any suggestions how to get around this? > > > > Gyula > > > > Tzu-Li (Gordon) Tai <tzuli...@apache.org> ezt írta (időpont: 2017. ápr. > > 10., H, 15:57): > > > > I would prefer to make this a blocker for a future bugfix actually, and > > not 1.2.1. > > > > The reason is that to fix this properly we might need to look again into > > (and possibly change) how partitioners are provided. > > The main problem is that the `open` method can only possibly be called > > once with the partitions of one topic. > > So, we might need the user to provide multiple partitioners, one for each > > of all the possible topics that will be written to. > > > > One way or another, my gut feeling is that this would need somewhat > slight > > change to the Kafka producer APIs. > > And I’m not so sure of rushing API changes into releases. > > > > > > On April 10, 2017 at 6:46:29 AM, Gyula Fóra (gyula.f...@gmail.com) > wrote: > > > > Thanks for checking this out. > > > > I would say this is definitely a blocking issue for the bugfix release, > > what do you think? > > > > Gyula > > > > Tzu-Li (Gordon) Tai <tzuli...@apache.org> ezt írta (időpont: 2017. ápr. > > 10., H, 15:39): > > > > Hi Gyula, > > > > Yes, I think the semantics of the Partitioner interface is a bit off. > > The `numPartitions` value ideally should be the number of partitions of > the > > `targetTopic`. > > > > Here’s a JIRA I just filed to track the issue: > > https://issues.apache.org/jira/browse/FLINK-6288. > > > > Cheers, > > Gordon > > > > On April 10, 2017 at 1:16:18 AM, Gyula Fóra (gyula.f...@gmail.com) > wrote: > > > > Hi all, > > > > We had some problems with custom partitioning for the 0.8 Kafka producer > > and now that I checked the code it seems there might be a problem with > the > > logic. > > > > The producer determines the number of partitions in the open method and > > seems to be using that as a value passed to the custom partitioner for > > producing the records. > > This will however only work if the defaultTopicId (topic) has the same > > number of partitions as all other topics in the kafka cluster when > > producing to multiple topics. > > > > In our case the default topic had 16 and new ones have 3 as default so it > > gives an out of range partition error. > > > > Is my understanding correct or am I overlooking something? > > > > Thank you! > > Gyula > > > > >
