Hi all,
I've been thinking about the proposed changes in KIP-179 and, on reflection,
I don't think the API presented is really ideal. Some of the limitations it
has include:
1. It sticks to the current, batch oriented (i.e. a single set of
reassignments at a time), model.
2. It still doesn't real
This thread has been very quiet for a while now. It's unclear whether this
is because no one has anything more to say, or whether no one has taken a
look at it in its current form. I suspect the latter, so I'm not calling
the vote today, but instead asking for more review.
What's currently propose
If there are no further comments, I will start a vote on this next week.
Thanks,
Tom
On 20 October 2017 at 08:33, Tom Bentley wrote:
> Hi,
>
> I've made a fairly major update to KIP-179 to propose APIs for setting
> throttled rates and throttled replicas with the ability to remove these
> auto
Hi,
I've made a fairly major update to KIP-179 to propose APIs for setting
throttled rates and throttled replicas with the ability to remove these
automatically at the end of reassignment.
I'd be grateful for your feedback:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+Reass
One question I have is about whether/how to scope throttling to a
reassignment. Currently throttles are only loosely associated with
reassignment: You can start a reassignment without any throttling, add
throttling to an in-flight reassignment, and remember/forget to remove
throttling after the rea
I'm starting to think about KIP-179 again. In order to have more
manageably-scoped KIPs and PRs I think it might be worth factoring-out the
throttling part into a separate KIP. Wdyt?
Keeping the throttling discussion in this thread for the moment...
The throttling behaviour is currently spread ac
Hi Ismael,
OK, KIP-195 has been factored out.
Regarding the dynamic configs, I personally still think we should have a
> specific protocol API for that
Can you explain a little more why?
With regards to throttling, it would be
> worth thinking about a way where the throttling configs can be
>
Hi Tom,
It won't be used within Kafka, but it's a public API that can be used by
other projects. And the protocol can be used by non-Java clients. So, there
is still value in including it.
Regarding the dynamic configs, I personally still think we should have a
specific protocol API for that. Wit
Hi Ismael,
It would be good to get at least some of this into 1.0.0.
We could put the increasePartitions() work into another KIP, but it would
be an unused AdminClient API in that release. The consumer of this API will
be the TopicsCommand when that get refactored to use the AdminClient.
That's s
Hi Tom,
What do you think of moving `increasePartitionsCount` (or
`increaseNumPartitions`) to a separate KIP? That is simple enough that we
could potentially include it in 1.0.0. I'd be happy to review it.
ReassignPartitions is more complex and we can probably aim to include that
in the January re
On Wed, Sep 6, 2017, at 00:20, Tom Bentley wrote:
> Hi Ted and Colin,
>
> Thanks for the comments.
>
> It seems you're both happier with reassign rather than assign, so I'm
> happy
> to stick with that.
>
>
> On 5 September 2017 at 18:46, Colin McCabe wrote:
>
> > ...
>
>
> > Do we expect t
Hi Ted and Colin,
Thanks for the comments.
It seems you're both happier with reassign rather than assign, so I'm happy
to stick with that.
On 5 September 2017 at 18:46, Colin McCabe wrote:
> ...
> Do we expect that reducing the number of partitions will ever be
> supported by this API? It
On Tue, Sep 5, 2017, at 09:39, Tom Bentley wrote:
> I've revised this KIP again:
>
> * Change the alterPartitionCounts() API to support passing an optional
> assignment for the new partitions (which is already supported by
> kafka-topics.sh). At the same time I didn't want the API to suggest it
>
bq. What about startPartitionAssignment() ?
Make sense.
startPartitionReassignment() seems to be better since the API deals with
reassignment.
Cheers
On Tue, Sep 5, 2017 at 9:39 AM, Tom Bentley wrote:
> I've revised this KIP again:
>
> * Change the alterPartitionCounts() API to support passing
I've revised this KIP again:
* Change the alterPartitionCounts() API to support passing an optional
assignment for the new partitions (which is already supported by
kafka-topics.sh). At the same time I didn't want the API to suggest it was
possible to change the existing assignments in the same ca
Hi all,
I've updated the KIP as follows:
* remove the APIs supporting progress reporting in favour of the APIs being
implemented in KIP-113.
* added some APIs to cover the existing functionality around throttling
inter-broker transfers, which was previously a TODO.
To respond to Colin's suggesti
On Wed, Aug 9, 2017, at 07:17, Tom Bentley wrote:
> Hi Dong and Jun,
>
> Thanks again for your input in this discussion and on KIP-113. It's
> difficult that discussion is split between this thread and the one for
> KIP-113, but I'll try to respond on this thread to questions asked on
> this
> thr
Hi Dong and Jun,
Thanks again for your input in this discussion and on KIP-113. It's
difficult that discussion is split between this thread and the one for
KIP-113, but I'll try to respond on this thread to questions asked on this
thread.
It seems there is some consensus that the alterTopic() API
Hi, Tom,
Thanks for the KIP. A few minor comments below.
1. Implementation wise, the broker handles adding partitions differently
from changing replica assignment. For the former, we directly update the
topic path in ZK with the new partitions. For the latter, we write the new
partition reassignm
Hey Tom,
Thanks for your reply. Here are my thoughts:
1) I think the DescribeDirsResponse can be used by AdminClient to query the
lag of follower replica as well. Here is how it works:
- AdminClient sends DescribeDirsRequest to both the leader and the follower
of the partition.
- DescribeDirsRes
Hi Dong,
Thanks for your reply.
You're right that your DescribeDirsResponse contains appropriate data. The
comment about the log_end_offset in the KIP says "Enable user to track
movement progress by comparing LEO of the *.log and *.move". That makes me
wonder whether this would only work for repl
Hey Tom,
Thanks for the KIP. It seems that the prior discussion in this thread has
focused on reassigning partitions (or AlterTopics). I haven't looked into
this yet. I have two comments on the replicaStatus() API and the
ReplicaStatusRequest:
- It seems that the use-case for ReplicaStatusReques
Hi again Ismael,
1. It's worth emphasising that reassigning partitions is a different
>> process than what happens when a topic is created, so not sure trying to
>> make it symmetric is beneficial. In addition to what was already
>> discussed,
>> one should also enable replication throttling befo
Hi Ismael,
1. It's worth emphasising that reassigning partitions is a different
> process than what happens when a topic is created, so not sure trying to
> make it symmetric is beneficial. In addition to what was already discussed,
> one should also enable replication throttling before moving th
Hi both,
Thanks for the replies. A few points:
1. It's worth emphasising that reassigning partitions is a different
process than what happens when a topic is created, so not sure trying to
make it symmetric is beneficial. In addition to what was already discussed,
one should also enable replicati
> Regarding adding the possibility to alter the topic config through the
> AlterTopic API, the current TopicCommand implementation provides a warning
> on doing this suggesting to use the ConfigCommand tool. So it would be a
> step back allowing to do the configs change with the alter topic as well
22 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use
AdminClient
Hi Ismael,
Thanks for taking the time to look at this.
Currently the proposal around the ReplicaStatusRequest/Response just allows
you to see the lag for the given replicas. It
Hi Ismael,
Thanks for taking the time to look at this.
Currently the proposal around the ReplicaStatusRequest/Response just allows
you to see the lag for the given replicas. It's not something that's tied
to a prior request to alter the topic at all, though obviously you can use
it to monitor how
Hi Tom,
A high-level point for discussion before going into the details. The
proposed protocol API `alterTopics` has 2 types of operations:
1. Operations that cause data movement (or deletion): increase/decrease of
replication factor and partition reassignment. These are currently done by
`kafka-
I have added the timeout I mentioned before, because it makes the
implementation of topic alteration more symmetric with the topic creation
APIs.
I have also added a section ("Policy") on retrofitting the
CreateTopicPolicy's rules to topic alteration requests, and made a few
other minor fixes.
I'
I've updated the KIP to fix those niggles, but I've not factored out the
topic name from the ReplicaStatusRequest, yet.
Looking at the topic creation APIs in more detail, the CreateTopicsOptions
has
* `shouldValidateOnly()`, which would make a lot of sense for the alter
topic APIs
* `timeoutMs()`
Thanks Paolo,
* in the "Public Interfaces" section you wrote
> alterTopics(Set) but then a collection is used (instead of a
> set) in the Proposed Changes section. I'm ok with collection.
>
Agree it should be Collection.
> * in the summary of the alterTopics method you say "The request
t.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>
From: Tom Bentley
Sent: Tuesday, July 25, 2017 11:07 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use
AdminClient
Hi Ismae
gt; Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
e<http://paolopatierno.wordpress.com/>
>
>
> ________________
> From: Tom Bentley
> Sent: Tuesday, July 25, 2017 9:02 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use
> AdminClient
>
> H
Bentley
Sent: Tuesday, July 25, 2017 9:02 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use
AdminClient
Hi Ismael,
I assume that's the --partitions for kafka-topics.sh? I must admit I hadn't
considered that tool, only kafka-reassign
Hi Ismael,
I assume that's the --partitions for kafka-topics.sh? I must admit I hadn't
considered that tool, only kafka-reassign-partitions.sh. Thanks for
pointing it out, because obviously the AdminClient API needs to be suitable
for reuse in kafka-topics.sh too. Since AdminClient doesn't current
Hi Tom,
I think it makes sense to keep progress reporting simple in the initial
version. As you say, time to completion is tricky to compute and it seems
like it should be tackled as its own KIP.
Regarding waiting for reassignment completion, it's easy enough for people
to do that via a script, s
Thinking about this some more, I release that the proposed API for tracking
progress is a bit specific to this reassignment use case. A more generally
useful API would be to be able to find out, for a partition on a broker:
* When the broker became a follower
* When the broker was last in the ISR
Aside: I've started this new DISCUSS thread for KIP-179 since the original
one had the incorrect KIP number 178. The original thread can be found
here:
http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=vzwwx+5dccpxry...@mail.gmail.com%3e
I've just
40 matches
Mail list logo
