Hey Bhavesh, I think it is useful to notify the user about the partition change.
The problem of having a listener in producer is that it is hard to guarantee the synchronization. For example, consider the following sequence: 1. producer sender thread refreshes the metadata with partition change. 2. user thread called send with customized partitioner, the partitioner decided the partition with new metadata refreshed in step 1. 3. producer sender thread calls onParitionChange() At that point, the message sent in step 2 was sent using the new metadata. If we update the metadata after invoking onParttitionChange(), it is a little strange because the metadata has not changed yet. Also the metadata refresh can happen in caller thread as well, not sure how it would work with multiple caller thread. I am thinking it seems the user can actually get the idea of whether the cluster has changed or not because the partition() method actually takes a cluster parameter. So if user cares about the partition number change, they can do the following: 1. store a copy of cluster as cache in the partitioner. 2. when partition() is called, check if the hash of this cluster is the same as the cached cluster. 3. If the hash of the passed in cluster is different from the hash of cached cluster, that means a metadata refresh occurred, people can check if there is partition change or not before do the partitioning. Thanks, Jiangjie (Becket) Qin On Wed, Sep 16, 2015 at 12:08 AM, Bhavesh Mistry <mistry.p.bhav...@gmail.com > wrote: > Hi Kafka Dev Team, > > I would like you get your feedback about adding yet another method or API > call to onPartitionsChange( ) to Partitioner Interface to get notify about > partition changes upon metadata refresh. > > This will allow custom logic (implementor of Partitioner) to be notified if > partition ownership or partition online vs offline, or partition > increase/decrease event happen and when changes were propagated to > individual producer instance. > > Please note this is my first KIP and if process is not followed correctly, > please do let me know. I will be more than happy to follow or correct > something that I may have missed. > > Thanks in advance and looking forward to your feedback. > > Thanks, > > Bhavesh >
