Hi Tom,
I think it makes sense to keep progress reporting simple in the initial
version. As you say, time to completion is tricky to compute and it seems
like it should be tackled as its own KIP.
Regarding waiting for reassignment completion, it's easy enough for people
to do that via a script, so I don't think we need to add it to the tool.
One question that occurred to me, the tool allows one to add partitions to
existing topics:
val partitionsOpt = parser.accepts("partitions", "The number of partitions
for the topic being created or " +
"altered (WARNING: If partitions are increased for a topic that has a
key, the partition logic or ordering of the messages will be affected")
It seems like it may make sense to have that as an AdminClient API as well.
If we agree to do that, then we need to decide if it should be implemented
client-side or by adding a protocol API. The former is simpler, but the
latter would allow non Java clients to use it without duplicating the logic
of assigning replicas to the new partitions. What are your thoughts?
Ismael
On Sat, Jul 22, 2017 at 10:14 AM, Tom Bentley <t.j.bent...@gmail.com> wrote:
> Thinking about this some more, I release that the proposed API for tracking
> progress is a bit specific to this reassignment use case. A more generally
> useful API would be to be able to find out, for a partition on a broker:
>
> * When the broker became a follower
> * When the broker was last in the ISR
> * If the broker is not in the ISR, how many messages behind it is
>
> That's not enough to be able to calculate a percentage completion of a
> reassignment though (we don't know how far behind it was when it became a
> follower). But maybe we could maintain a maximum of how far behind it has
> fallen behind the ISR, since it became a follower/dropped out of the ISR.
>
> btw, we need the middle bullet to cope with following sequence:
>
> 1. Start syncing
> 2. Finish syncing
> 3. Fall behind (drop out of ISR for some reason)
> 4. User queries for if reassignment has finished (it has, but just looking
> at the ISR would give the impression that it has not).
>
>
>
>
> On 21 July 2017 at 11:09, Tom Bentley <t.j.bent...@gmail.com> wrote:
>
> > Aside: I've started this new DISCUSS thread for KIP-179 since the
> original
> > one had the incorrect KIP number 178. The original thread can be found
> > here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.
> > mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=VZWWX+
> > 5dccpxry...@mail.gmail.com%3e
> >
> > I've just updated KIP-179 to support Ismael's request for the command to
> > be able to support progress reporting of an ongoing partition
> reassignment.
> >
> > I'll call out two things which I'm not sure about since I don't yet have
> > much experience of Kafka being used operationally:
> >
> > 1. As currently constructed the --progress option could report an overall
> > progress percentage, per-partition percentages and errors. It cannot
> > provide any kind of time-to-completion estimate. Part of me is loath to
> do
> > this, as I'm sure we all remember file transfer dialogs that provide
> > amusing/baffling time-to-completion estimates. So it might be hard to do
> > _well_. On the other hand I expect the thing people will be interested in
> > will often be "when will it be finished?"
> >
> > 2. There is no option for the tool to wait for reassignment completion. I
> > can imagine users might want to script something to happen after the
> > reassignment is complete, and without some kind of --wait option they
> will
> > have to poll for completion "manually". Having a --wait optin and putting
> > this polling in the tool means we have a lot more control over how often
> > such polling happens.
> >
> > The KIP is available here:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+
> > ReassignPartitionsCommand+to+use+AdminClient
> >
> > Thanks,
> >
> > Tom
> >
>
