Hi Steven, I think it would be useful to be able to subscribe yourself on updates of > reassignment changes.
I agree this would be really useful, but, to the extent I understand the networking underpinnings of the admin client, it might be difficult to do well in practice. Part of the problem is that you might "set a watch" (to borrow the ZK terminology) via one broker (or the controller), only for that broker to fail (or the controller be re-elected). Obviously you can detect the loss of connection and set a new watch via a different broker (or the new controller), but that couldn't be transparent to the user, because the AdminClient doesn't know what changed while it was disconnected/not watching. Another issue is that to avoid races you really need to combine fetching the current state with setting the watch (as is done in the native ZooKeeper API). I think there are lots of subtle issues of this sort which would need to be addressed to make something reliable. In the mean time, ZooKeeper already has a (proven and mature) API for watches, so there is, in principle, a good workaround. I say "in principle" because in the KIP-236 proposal right now the /admin/reassign_partitions znode is legacy and the reassignment is represented by /admin/reassigments/$topic/$partition. That naming scheme for the znode would make it harder for ZooKeeper clients like yours because such clients would need to set a child watch per topic. The original proposal for the naming scheme was /admin/reassigments/$topic-$partition, which would mean clients like yours would need only 1 child watch. The advantage of /admin/reassigments/$topic/$partition is it scales better. I don't currently know how well ZooKeeper copes with nodes with many children, so it's difficult for me weigh those two options, but I would be happy to switch back to /admin/reassigments/$topic-$partition if we could reassure ourselves it would scale OK to the reassignment sizes would people need in practice. Overall I would prefer not to tackle something like this in *this* KIP, though it could be something for a future KIP. Of course I'm happy to hear more discussion about this too! Cheers, Tom On 15 December 2017 at 18:51, Steven Aerts <steven.ae...@gmail.com> wrote: > Tom, > > > I think it would be useful to be able to subscribe yourself on updates of > reassignment changes. > Our internal kafka supervisor and monitoring tools are currently subscribed > to these changes in zookeeper so they can babysit our clusters. > > I think it would be nice if we could receive these events through the > adminclient. > In the api proposal, you can only poll for changes. > > No clue how difficult it would be to implement, maybe you can piggyback on > some version number in the repartition messages or on zookeeper. > > This is just an idea, not a must have feature for me. We can always poll > over > the proposed api. > > > Regards, > > > Steven > > > Op vr 15 dec. 2017 om 19:16 schreef Tom Bentley <t.j.bent...@gmail.com>: > > > Hi, > > > > KIP-236 lays the foundations for AdminClient APIs to do with partition > > reassignment. I'd now like to start discussing KIP-240, which adds APIs > to > > the AdminClient to list and describe the current reassignments. > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 240%3A+AdminClient.listReassignments+AdminClient.describeReassignments > > > > Aside: I have fairly developed ideas for the API for starting a > > reassignment, but I intend to put that in a third KIP. > > > > Cheers, > > > > Tom > > >
