Re: [DISCUSS] KIP-73: Replication Quotas

Thanks Ben, this makes sense to me. As discussed in the PR[1] and the JIRA you filed[2], it probably makes sense to also split the throttle limit replication.quota.throttled.rate into separate leader and follower limits. Ismael [1] *https://github.com/apache/kafka/pull/1906#issuecomment-249733772

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Ben, Just for my understanding : When you said : What we really want to do is apply: LeaderThrottle [104,107,109] FollowerThrottle [105,113] ---> This means that we want to apply leader throttle to only replica that is the leader out of 104, 107, 109 right? Also when you said : [104,107,10

Re: [DISCUSS] KIP-73: Replication Quotas

Makes sense to me. Thanks Ben! On Mon, Sep 26, 2016 at 9:39 AM Jun Rao wrote: > Yes, this change makes sense to me since it gives the admin better control. > > Thanks, > > Jun > > On Sun, Sep 25, 2016 at 7:35 AM, Ben Stopford wrote: > > > Hi All > > > > We’ve made an adjustment to KIP-73: Repli

Re: [DISCUSS] KIP-73: Replication Quotas

Yes, this change makes sense to me since it gives the admin better control. Thanks, Jun On Sun, Sep 25, 2016 at 7:35 AM, Ben Stopford wrote: > Hi All > > We’ve made an adjustment to KIP-73: Replication Quotas which I’d like to > open up to the community for approval. > > Previously the admin p

[DISCUSS] KIP-73: Replication Quotas

Hi All We’ve made an adjustment to KIP-73: Replication Quotas which I’d like to open up to the community for approval. Previously the admin passed a list of replicas to be throttled: quota.replication.throttled.replicas = [partId]:[replica],[partId]:[replica],[partId]:[replica] etc T

Re: [DISCUSS] KIP-73: Replication Quotas

Yes, I think its a great discussion to have. There are definitely pros and cons to both approaches and worth thinking about the right way forward. On Thu, Aug 18, 2016 at 11:03 AM, Todd Palino wrote: > This all makes a lot of sense, and mirrors what I’m thinking as I finally > took some time to

Re: [DISCUSS] KIP-73: Replication Quotas

Yeah, I’m good with where we are right now on this KIP. It’s a workable solution that we can add tooling to support. I would prefer to have soft quotas, but given that it is more complex to implement, we can go with hard quotas for the time being and consider it as an improvement later. -Todd On

Re: [DISCUSS] KIP-73: Replication Quotas

Todd, Thanks for the detailed reply. So, it sounds like that you are ok with the current proposal in the KIP for now and we can brainstorm on more automated stuff separately? Are you comfortable with starting the vote on the current proposal? Jun On Thu, Aug 18, 2016 at 11:00 AM, Todd Palino wr

Re: [DISCUSS] KIP-73: Replication Quotas

This all makes a lot of sense, and mirrors what I’m thinking as I finally took some time to really walk through scenarios around why we move partitions around. What I’m wondering if it makes sense to have a conversation around breaking out the controller entirely, separating it from the brokers, a

Re: [DISCUSS] KIP-73: Replication Quotas

Joel just reminded me to take another look at this one :) So first off, this is great. It’s something that we definitely need to have, especially as we get into the realm of moving partitions around more often. I do prefer to have the cluster handle this automatically. What I envision is a single

Re: [DISCUSS] KIP-73: Replication Quotas

Just my take, since Jun and Ben originally wanted to solve a more general approach and I talked them out of it :) When we first add the feature, safety is probably most important in getting people to adopt it - I wanted to make the feature very safe by never throttling something admins don't want

Re: [DISCUSS] KIP-73: Replication Quotas

Joel, Yes, for your second comment. The tricky thing is still to figure out which replicas to throttle and by how much since in general, admins probably don't want already in-sync or close to in-sync replicas to be throttled. It would be great to get Todd's opinion on this. Could you ping him? Ye

Re: [DISCUSS] KIP-73: Replication Quotas

> For your first comment. We thought about determining "effect" replicas > automatically as well. First, there are some tricky stuff that one has to > Auto-detection of effect traffic: i'm fairly certain it's doable but definitely tricky. I'm also not sure it is something worth tackling at the out

Re: [DISCUSS] KIP-73: Replication Quotas

Joel, For your first comment. We thought about determining "effect" replicas automatically as well. First, there are some tricky stuff that one has to figure out as Ben pointed out. For example, what's the definition of "effect" replicas. If a replica falls out of ISR temporarily, does it become a

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Joel Ha! yes we had some similar thoughts, on both counts. Both are actually good approaches, but come with some extra complexity. Segregating the replication type is tempting as it creates a more general solution. One issue is you need to draw a line between lagging and not lagging. The I

Re: [DISCUSS] KIP-73: Replication Quotas

On Wed, Aug 17, 2016 at 9:13 PM, Ben Stopford wrote: > > Let's us know if you have any further thoughts on KIP-73, else we'll kick > off a vote. > I think the mechanism for throttling replicas looks good. Just had a few more thoughts on the configuration section. What you have looks reasonable,

Re: [DISCUSS] KIP-73: Replication Quotas

Hi everyone Let's us know if you have any further thoughts on KIP-73, else we'll kick off a vote. Thanks B On Friday, 12 August 2016, Jun Rao wrote: > Mayuresh, > > I was thinking of the following. > > If P1 has data and P2 is throttled, we will return empty data for P2 and > send the respons

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Mayuresh So, we’re not actually delaying the FetchResponse, on the leader, unless it is empty i.e. less min.bytes, ready to be sent, for both throttled and non-thorttled partitions. So there shouldn’t be any need for the leader to ask the follower to delay. If there are bytes to be transmit

Re: [DISCUSS] KIP-73: Replication Quotas

Mayuresh, I was thinking of the following. If P1 has data and P2 is throttled, we will return empty data for P2 and send the response back immediately. The follower will issue the next fetch request immediately, but the leader won't return any data in P2 until the quota is not exceeded. We are no

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Jun, Correct me if I am wrong. If the response size includes throttled and unthrottled replicas, I am thinking if this is possible : The leader broker B1 receives a fetch request partition P1 and P2 for a topic from replica broker B2. In this case lets say that only P2 is throttled on the leade

Re: [DISCUSS] KIP-73: Replication Quotas

Hi, Joel, Yes, the response size includes both throttled and unthrottled replicas. However, the response is only delayed up to max.wait if the response size is less than min.bytes, which matches the current behavior. So, there is no extra delay to due throttling, right? For replica fetchers, the d

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Jun, I'm not sure that would work unless we have separate replica fetchers, since this would cause all replicas (including ones that are not throttled) to get delayed. Instead, we could just have the leader populate the throttle-time field of the response as a hint to the follower as to how lon

Re: [DISCUSS] KIP-73: Replication Quotas

Mayuresh, That's a good question. I think if the response size (after leader throttling) is smaller than min.bytes, we will just delay the sending of the response up to max.wait as we do now. This should prevent frequent empty responses to the follower. Thanks, Jun On Wed, Aug 10, 2016 at 9:17

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Mayureesh That’s a good question and something that should be covered. I think if the leader throttles partitions so the response gets too small, it should be automatically delayed in Purgatory. Likewise, on the follower, if the request is contains no partitions the request will be again be

Re: [DISCUSS] KIP-73: Replication Quotas

This might have been answered before. I was wondering when the leader quota is reached and it sends empty response ( If the inclusion of a partition, listed in the leader's throttled-replicas list, causes the LeaderQuotaRate to be exceeded, that partition is omitted from the response (aka returns 0

Re: [DISCUSS] KIP-73: Replication Quotas

Thanks again for the responses everyone. I’ve removed the the extra fetcher threads from the proposal, switching to the inclusion-based approach. The relevant section is: The follower makes a requests, using the fixed size of replica.fetch.response.max.bytes as per KIP-74

Re: [DISCUSS] KIP-73: Replication Quotas

When there are several unthrottled replicas, we could also just do what's suggested in KIP-74. The client is responsible for reordering the partitions and the leader fills in the bytes to those partitions in order, up to the quota limit. We could also do what you suggested. If quota is exceeded, i

Re: [DISCUSS] KIP-73: Replication Quotas

> > > > On the leader side, one challenge is related to the fairness issue that Ben > brought up. The question is what if the fetch response limit is filled up > by the throttled replicas? If this happens constantly, we will delay the > progress of those un-throttled replicas. However, I think we c

Re: [DISCUSS] KIP-73: Replication Quotas

Joel, Mayuresh, Thanks for the suggestion. Yes, I think we could use a single replica fetch thread to handle both throttled and unthrottled replicas. On the follower side, this is easy. If the quota is exceeded, we remember the amount of the delay needed, take those throttled replicas off the fet

Re: [DISCUSS] KIP-73: Replication Quotas

Nice write up Ben. I agree with Joel for keeping this simple by excluding the partitions from the fetch request/response when the quota is violated at the follower or leader instead of having a separate set of threads for handling the quota and non quota cases. Even though its different from the c

Re: [DISCUSS] KIP-73: Replication Quotas

On the need for both leader/follower throttling: that makes sense - thanks for clarifying. For completeness, can we add this detail to the doc - say, after the quote that I pasted earlier? >From an implementation perspective though: I’m still interested in the simplicity of not having to add separ

Re: [DISCUSS] KIP-73: Replication Quotas

Just to elaborate on what Ben said why we need throttling on both the leader and the follower side. If we only have throttling on the follower side, consider a case that we add 5 more new brokers and want to move some replicas from existing brokers over to those 5 brokers. Each of those broker is

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Joel Thanks for taking the time to look at this. Appreciated. Regarding throttling on both leader and follower, this proposal covers a more general solution which can guarantee a quota, even when a rebalance operation produces an asymmetric profile of load. This means administrators don’t n

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Ben, Thanks for the detailed write-up. So the proposal involves self-throttling on the fetcher side and throttling at the leader. Can you elaborate on the reasoning that is given on the wiki: *“The throttle is applied to both leaders and followers. This allows the admin to exert strong guarante

[DISCUSS] KIP-73: Replication Quotas

We’ve created KIP-73: Replication Quotas The idea is to allow an admin to throttle moving replicas. Full details are here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas Please take a loo