Hi Ismael,
I assume that's the --partitions for kafka-topics.sh? I must admit I hadn't
considered that tool, only kafka-reassign-partitions.sh. Thanks for
pointing it out, because obviously the AdminClient API needs to be suitable
for reuse in kafka-topics.sh too. Since AdminClient doesn't currently
support an alterTopics() I guess the API should mirror the API for
newTopics(), so something like:
public AlterTopicsResult alterTopics(Set<AlteredTopic> topics);
// where:
public class AlteredTopic {
// constructor etc
public String name();
public int numPartitions();
public int replicationFactor();
Map<Integer,List<Integer>> replicasAssignment();
}
Note that although NewTopic contains a `Map<String,String> configs`, I
think the API for changing a topic's config already exists: alterConfigs().
This API is better than having separate methods to set the number of
partitions/replicas and assigning them to brokers, since sometimes people
will want to set the assignment at the same time as changing the
partitions/replicas.
An API like this could then be used by both tools.
On 24 July 2017 at 16:23, Ismael Juma <ism...@juma.me.uk> wrote:
> Hi Tom,
>
> I think it makes sense to keep progress reporting simple in the initial
> version. As you say, time to completion is tricky to compute and it seems
> like it should be tackled as its own KIP.
>
> Regarding waiting for reassignment completion, it's easy enough for people
> to do that via a script, so I don't think we need to add it to the tool.
>
> One question that occurred to me, the tool allows one to add partitions to
> existing topics:
>
> val partitionsOpt = parser.accepts("partitions", "The number of partitions
> for the topic being created or " +
> "altered (WARNING: If partitions are increased for a topic that has a
> key, the partition logic or ordering of the messages will be affected")
>
> It seems like it may make sense to have that as an AdminClient API as well.
> If we agree to do that, then we need to decide if it should be implemented
> client-side or by adding a protocol API. The former is simpler, but the
> latter would allow non Java clients to use it without duplicating the logic
> of assigning replicas to the new partitions. What are your thoughts?
>
> Ismael
>
> On Sat, Jul 22, 2017 at 10:14 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > Thinking about this some more, I release that the proposed API for
> tracking
> > progress is a bit specific to this reassignment use case. A more
> generally
> > useful API would be to be able to find out, for a partition on a broker:
> >
> > * When the broker became a follower
> > * When the broker was last in the ISR
> > * If the broker is not in the ISR, how many messages behind it is
> >
> > That's not enough to be able to calculate a percentage completion of a
> > reassignment though (we don't know how far behind it was when it became a
> > follower). But maybe we could maintain a maximum of how far behind it has
> > fallen behind the ISR, since it became a follower/dropped out of the ISR.
> >
> > btw, we need the middle bullet to cope with following sequence:
> >
> > 1. Start syncing
> > 2. Finish syncing
> > 3. Fall behind (drop out of ISR for some reason)
> > 4. User queries for if reassignment has finished (it has, but just
> looking
> > at the ISR would give the impression that it has not).
> >
> >
> >
> >
> > On 21 July 2017 at 11:09, Tom Bentley <t.j.bent...@gmail.com> wrote:
> >
> > > Aside: I've started this new DISCUSS thread for KIP-179 since the
> > original
> > > one had the incorrect KIP number 178. The original thread can be found
> > > here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.
> > > mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=VZWWX+
> > > 5dccpxry...@mail.gmail.com%3e
> > >
> > > I've just updated KIP-179 to support Ismael's request for the command
> to
> > > be able to support progress reporting of an ongoing partition
> > reassignment.
> > >
> > > I'll call out two things which I'm not sure about since I don't yet
> have
> > > much experience of Kafka being used operationally:
> > >
> > > 1. As currently constructed the --progress option could report an
> overall
> > > progress percentage, per-partition percentages and errors. It cannot
> > > provide any kind of time-to-completion estimate. Part of me is loath to
> > do
> > > this, as I'm sure we all remember file transfer dialogs that provide
> > > amusing/baffling time-to-completion estimates. So it might be hard to
> do
> > > _well_. On the other hand I expect the thing people will be interested
> in
> > > will often be "when will it be finished?"
> > >
> > > 2. There is no option for the tool to wait for reassignment
> completion. I
> > > can imagine users might want to script something to happen after the
> > > reassignment is complete, and without some kind of --wait option they
> > will
> > > have to poll for completion "manually". Having a --wait optin and
> putting
> > > this polling in the tool means we have a lot more control over how
> often
> > > such polling happens.
> > >
> > > The KIP is available here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+
> > > ReassignPartitionsCommand+to+use+AdminClient
> > >
> > > Thanks,
> > >
> > > Tom
> > >
> >
>
