Hi Ismael, I've been working on the progress reporting assuming that it would be acceptable for the ReassignPartitionsCommand to poll the AdminClient API (and in turn the AdminClient API to poll the broker) in order to report progress in an interactive way.
The alternative would, of course, be for the broker to push notify the AdminClient when it became aware of changes in progress. I didn't think this is what you meant as it would be a departure from the Kafka norm of request-response, but thought it worthwhile to check before I spend any more time on a polling-based approach. Thanks, Tom On 19 July 2017 at 16:08, Tom Bentley <t.j.bent...@gmail.com> wrote: > Ah, thank you! I took the number from the "Next KIP Number: 178" on the > KIP index and didn't check the tables. So this is now KIP-179. The old link > will point you to the right place. > > On 19 July 2017 at 15:55, Ismael Juma <ism...@juma.me.uk> wrote: > >> One more thing, it looks like there is already a KIP-178: >> >> KIP-178: Size-based log directory selection strategy >> >> Ismael >> >> On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley <t.j.bent...@gmail.com> >> wrote: >> >> > OK, I will work on adding support for this to the KIP, with the >> intention >> > of a two part implementation. >> > >> > On 19 July 2017 at 14:59, Ismael Juma <ism...@juma.me.uk> wrote: >> > >> > > Hi Tom, >> > > >> > > It's fine for the tool not to have this functionality from the start. >> > > However, since we're adding new Kafka protocol APIs, we need to >> consider >> > > some of these details to ensure we're building towards the end state, >> if >> > > that makes sense. Protocol APIs are used by multiple clients, so >> there is >> > > value in thinking ahead a bit when it comes to the design. The >> > > implementation can often be done in stages. >> > > >> > > Does that make sense? >> > > >> > > Ismael >> > > >> > > On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley <t.j.bent...@gmail.com> >> > > wrote: >> > > >> > > > Hi Ismael, >> > > > >> > > > Answers in-line: >> > > > >> > > > 1. Have you considered how progress would be reported? Partition >> > > > > reassignment can take a long time and it would be good to have a >> > > > mechanism >> > > > > for progress reporting. >> > > > > >> > > > >> > > > The ReassignPartitionsCommand doesn't currently have a mechanism to >> > track >> > > > progress. All you can do at the moment is initiate a reassignment >> (with >> > > > --execute), and later check whether the assignment is in the state >> you >> > > > asked for (with --verify). I agree it would be nice to be able to >> track >> > > > progress. >> > > > >> > > > This will be the first 'big' bit of work I've done on Kafka, so I >> would >> > > > prefer to limit the scope of this KIP where possible. That said, I >> > > suppose >> > > > it could be done by having receiving controllers publish their >> progress >> > > to >> > > > ZooKeeper, and adding Protocol and AdminClient API for getting this >> > > > information. If you're keen on this I can certainly modify the KIP >> to >> > add >> > > > this. >> > > > >> > > > Alternatively I could write a second KIP to add this ability. What >> > other >> > > > long running tasks are there for which we'd like the ability to >> report >> > > > progress? If there are others it might be possible to come up with a >> > > common >> > > > mechanism. >> > > > >> > > > >> > > > > 2. Removals can only happen in major releases. In your example, >> the >> > > > removal >> > > > > could only happen in 2.0.0. >> > > > > >> > > > >> > > > OK, I'll update the KIP. >> > > > >> > > >> > >> > >
