Hi Tom and Paolo, It's true that increasing the number of partitions is done via the kafka-topics tool, which is also being converted to use the AdminClient (but via a different JIRA). I also agree that it would be good to consider if alterTopics would be a sensible way to support all the use cases or if it's better to have separate APIs. I think it makes sense to have a single KIP though as they are related and it will be easier to evaluate as a whole.
Does that make sense? Ismael On Tue, Jul 25, 2017 at 10:16 AM, Paolo Patierno <ppatie...@live.com> wrote: > Hi, > > > I was digging into it because I need something like an Admin Client alter > API for my work on rewriting the TopicCommand tool using them. > > The AlterConfigs API is used for changing topic level configuration (i.e. > retention.ms, retention.bytes and so on). > > A new AlterTopic API could be better in order to change topic "high level" > structure so number of partitions, replication factors and so on. > > My opinion is that we need separate API because from my point of view they > are different settings. > > > Thanks, > > Paolo. > > > Paolo Patierno > Senior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Embedded & IoT > Microsoft Azure Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > ________________________________ > From: Tom Bentley <t.j.bent...@gmail.com> > Sent: Tuesday, July 25, 2017 9:02 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use > AdminClient > > Hi Ismael, > > I assume that's the --partitions for kafka-topics.sh? I must admit I hadn't > considered that tool, only kafka-reassign-partitions.sh. Thanks for > pointing it out, because obviously the AdminClient API needs to be suitable > for reuse in kafka-topics.sh too. Since AdminClient doesn't currently > support an alterTopics() I guess the API should mirror the API for > newTopics(), so something like: > > public AlterTopicsResult alterTopics(Set<AlteredTopic> topics); > // where: > public class AlteredTopic { > // constructor etc > public String name(); > public int numPartitions(); > public int replicationFactor(); > Map<Integer,List<Integer>> replicasAssignment(); > } > > Note that although NewTopic contains a `Map<String,String> configs`, I > think the API for changing a topic's config already exists: alterConfigs(). > > This API is better than having separate methods to set the number of > partitions/replicas and assigning them to brokers, since sometimes people > will want to set the assignment at the same time as changing the > partitions/replicas. > > An API like this could then be used by both tools. > > > > On 24 July 2017 at 16:23, Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi Tom, > > > > I think it makes sense to keep progress reporting simple in the initial > > version. As you say, time to completion is tricky to compute and it seems > > like it should be tackled as its own KIP. > > > > Regarding waiting for reassignment completion, it's easy enough for > people > > to do that via a script, so I don't think we need to add it to the tool. > > > > One question that occurred to me, the tool allows one to add partitions > to > > existing topics: > > > > val partitionsOpt = parser.accepts("partitions", "The number of > partitions > > for the topic being created or " + > > "altered (WARNING: If partitions are increased for a topic that > has a > > key, the partition logic or ordering of the messages will be affected") > > > > It seems like it may make sense to have that as an AdminClient API as > well. > > If we agree to do that, then we need to decide if it should be > implemented > > client-side or by adding a protocol API. The former is simpler, but the > > latter would allow non Java clients to use it without duplicating the > logic > > of assigning replicas to the new partitions. What are your thoughts? > > > > Ismael > > > > On Sat, Jul 22, 2017 at 10:14 AM, Tom Bentley <t.j.bent...@gmail.com> > > wrote: > > > > > Thinking about this some more, I release that the proposed API for > > tracking > > > progress is a bit specific to this reassignment use case. A more > > generally > > > useful API would be to be able to find out, for a partition on a > broker: > > > > > > * When the broker became a follower > > > * When the broker was last in the ISR > > > * If the broker is not in the ISR, how many messages behind it is > > > > > > That's not enough to be able to calculate a percentage completion of a > > > reassignment though (we don't know how far behind it was when it > became a > > > follower). But maybe we could maintain a maximum of how far behind it > has > > > fallen behind the ISR, since it became a follower/dropped out of the > ISR. > > > > > > btw, we need the middle bullet to cope with following sequence: > > > > > > 1. Start syncing > > > 2. Finish syncing > > > 3. Fall behind (drop out of ISR for some reason) > > > 4. User queries for if reassignment has finished (it has, but just > > looking > > > at the ISR would give the impression that it has not). > > > > > > > > > > > > > > > On 21 July 2017 at 11:09, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > > > > > Aside: I've started this new DISCUSS thread for KIP-179 since the > > > original > > > > one had the incorrect KIP number 178. The original thread can be > found > > > > here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707. > > > > mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=VZWWX+ > > > > 5dccpxry...@mail.gmail.com%3e > > > > > > > > I've just updated KIP-179 to support Ismael's request for the command > > to > > > > be able to support progress reporting of an ongoing partition > > > reassignment. > > > > > > > > I'll call out two things which I'm not sure about since I don't yet > > have > > > > much experience of Kafka being used operationally: > > > > > > > > 1. As currently constructed the --progress option could report an > > overall > > > > progress percentage, per-partition percentages and errors. It cannot > > > > provide any kind of time-to-completion estimate. Part of me is loath > to > > > do > > > > this, as I'm sure we all remember file transfer dialogs that provide > > > > amusing/baffling time-to-completion estimates. So it might be hard to > > do > > > > _well_. On the other hand I expect the thing people will be > interested > > in > > > > will often be "when will it be finished?" > > > > > > > > 2. There is no option for the tool to wait for reassignment > > completion. I > > > > can imagine users might want to script something to happen after the > > > > reassignment is complete, and without some kind of --wait option they > > > will > > > > have to poll for completion "manually". Having a --wait optin and > > putting > > > > this polling in the tool means we have a lot more control over how > > often > > > > such polling happens. > > > > > > > > The KIP is available here: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+ > > > > ReassignPartitionsCommand+to+use+AdminClient > > > > > > > > Thanks, > > > > > > > > Tom > > > > > > > > > >
