Thinking about this some more, I release that the proposed API for tracking progress is a bit specific to this reassignment use case. A more generally useful API would be to be able to find out, for a partition on a broker:
* When the broker became a follower * When the broker was last in the ISR * If the broker is not in the ISR, how many messages behind it is That's not enough to be able to calculate a percentage completion of a reassignment though (we don't know how far behind it was when it became a follower). But maybe we could maintain a maximum of how far behind it has fallen behind the ISR, since it became a follower/dropped out of the ISR. btw, we need the middle bullet to cope with following sequence: 1. Start syncing 2. Finish syncing 3. Fall behind (drop out of ISR for some reason) 4. User queries for if reassignment has finished (it has, but just looking at the ISR would give the impression that it has not). On 21 July 2017 at 11:09, Tom Bentley <t.j.bent...@gmail.com> wrote: > Aside: I've started this new DISCUSS thread for KIP-179 since the original > one had the incorrect KIP number 178. The original thread can be found > here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707. > mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=VZWWX+ > 5dccpxry...@mail.gmail.com%3e > > I've just updated KIP-179 to support Ismael's request for the command to > be able to support progress reporting of an ongoing partition reassignment. > > I'll call out two things which I'm not sure about since I don't yet have > much experience of Kafka being used operationally: > > 1. As currently constructed the --progress option could report an overall > progress percentage, per-partition percentages and errors. It cannot > provide any kind of time-to-completion estimate. Part of me is loath to do > this, as I'm sure we all remember file transfer dialogs that provide > amusing/baffling time-to-completion estimates. So it might be hard to do > _well_. On the other hand I expect the thing people will be interested in > will often be "when will it be finished?" > > 2. There is no option for the tool to wait for reassignment completion. I > can imagine users might want to script something to happen after the > reassignment is complete, and without some kind of --wait option they will > have to poll for completion "manually". Having a --wait optin and putting > this polling in the tool means we have a lot more control over how often > such polling happens. > > The KIP is available here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+ > ReassignPartitionsCommand+to+use+AdminClient > > Thanks, > > Tom >
