Hi, David,
Thanks for the explanation. The KIP looks good to me now.
Jun
On Tue, Jun 9, 2020 at 4:27 AM David Jacot wrote:
> Hi Jun,
>
> 40. Yes, ThrottleTimeMs is set when the error code is set to QuotaViolated.
> This
> is required to let the client know how long it must wait. This is explai
Hi Jun,
40. Yes, ThrottleTimeMs is set when the error code is set to QuotaViolated.
This
is required to let the client know how long it must wait. This is explained
in the
"Handling of new/old clients".
Best,
David
On Mon, Jun 8, 2020 at 9:29 PM Jun Rao wrote:
> Hi, David,
>
> Thanks for the u
Hi, David,
Thanks for the updated KIP. Another minor comment below.
40. For the new `QUOTA_VIOLATED` error in the response to
CreateTopics/CreatePartitions/DeleteTopics, could you clarify
whether ThrottleTimeMs is set when the error code is set to QUOTA_VIOLATED?
Jun
On Mon, Jun 8, 2020 at 9:32
Hi Jun,
30. The rate is accumulated at the partition level. Let me clarify this in
the KIP.
Best,
David
On Sat, Jun 6, 2020 at 2:37 AM Anna Povzner wrote:
> Hi David,
>
> The KIP looks good to me. I am going to the voting thread...
>
> Hi Jun,
>
> Yes, exactly. That's a separate thing from thi
Hi David,
The KIP looks good to me. I am going to the voting thread...
Hi Jun,
Yes, exactly. That's a separate thing from this KIP, so working on the fix.
Thanks,
Anna
On Fri, Jun 5, 2020 at 4:36 PM Jun Rao wrote:
> Hi, Anna,
>
> Thanks for the comment. For the problem that you described, pe
Hi, Anna,
Thanks for the comment. For the problem that you described, perhaps we need
to make the quota checking and recording more atomic?
Hi, David,
Thanks for the updated KIP. Looks good to me now. Just one minor comment
below.
30. controller_mutations_rate: For topic creation and deletion,
Hi Anna and Jun,
You are right. We should allocate up to the quota for each old sample.
I have revamped the Throttling Algorithm section to better explain our
thought process and the token bucket inspiration.
I have also added a chapter with few guidelines about how to define
the quota. There is
Hi David and Jun,
I dug a bit deeper into the Rate implementation, and wanted to confirm that
I do believe that the token bucket behavior is better for the reasons we
already discussed but wanted to summarize. The main difference between Rate
and token bucket is that the Rate implementation allows
Hi, David, Anna,
Thanks for the discussion and the updated wiki.
11. If we believe the token bucket behavior is better in terms of handling
the burst behavior, we probably don't need a separate KIP since it's just
an implementation detail.
Regarding "So instead of having one sample equal to 560
Hi all,
I just published an updated version of the KIP which includes:
* Using a slightly modified version of our Rate. I have tried to formalize
it based on our discussion. As Anna suggested, we may find a better way to
implement it.
* Handling of ValidateOnly as pointed out by Tom.
Please, chec
Hi David,
As a user I might expect the validateOnly option to do everything except
actually make the changes. That interpretation would imply the quota should
be checked, but the check should obviously be side-effect free. I think
this interpretation could be useful because it gives the caller eit
Hi Tom,
That's a good question. As the validation does not create any load on the
controller, I was planning to do it without checking the quota at all. Does
that
sound reasonable?
Best,
David
On Thu, Jun 4, 2020 at 4:23 PM David Jacot wrote:
> Hi Jun and Anna,
>
> Thank you both for your repl
Hi Jun and Anna,
Thank you both for your replies.
Based on our recent discussion, I agree that using a rate is better to
remain
consistent with other quotas. As you both suggested, it seems that changing
the way we compute the rate to better handle spiky workloads and behave a
bit more similarly
Hi Jun and David,
Regarding token bucket vs, Rate behavior. We recently observed a couple of
cases where a bursty workload behavior would result in long-ish pauses in
between, resulting in lower overall bandwidth than the quota. I will need
to debug this a bit more to be 100% sure, but it does loo
Hi, David,
Thanks for the reply.
11. To match the behavior in the Token bucket approach, I was thinking that
requests that don't fit in the previous time windows will be accumulated in
the current time window. So, the 60 extra requests will be accumulated in
the latest window. Then, the client al
Hi David,
One quick question about the implementation (I don't think it's spelled out
in the KIP): Presumably if you make, for example, a create topics request
with validate only it will check for quota violation, but not count towards
quota violation, right?
Many thanks,
Tom
On Wed, Jun 3, 202
Hi David,
2) sorry, that was my mistake.
Regards,
Rajini
On Wed, Jun 3, 2020 at 3:08 PM David Jacot wrote:
> Hi Rajini,
>
> Thanks for your prompt response.
> 1) Good catch, fixed.
> 2) The retry mechanism will be in the client so a new field is not
> required in the requests.
>
> Regards,
>
Hi Rajini,
Thanks for your prompt response.
1) Good catch, fixed.
2) The retry mechanism will be in the client so a new field is not
required in the requests.
Regards,
David
On Wed, Jun 3, 2020 at 2:43 PM Rajini Sivaram
wrote:
> Hi David,
>
> Thanks for the updates, looks good. Just a couple o
Hi David,
Thanks for the updates, looks good. Just a couple of minor comments:
1) There is a typo in "*The channel will be mutated as well when
`throttle_time_ms > 0`." * Should be *muted*?
2) Since the three requests will need a new field for `
*retryQuotaViolatedException*`, we should perhaps ad
Hi all,
I have updated the KIP based on our recent discussions. I have mainly
changed the
following points:
* I have renamed the quota as suggested by Jun.
* I have changed the metrics to be "token bucket" agnostic. The idea is to
report the
burst and the rate per principal/clientid.
* I have remo
Hi Rajini,
Thanks for your feedback. Please find my answers below:
1) Our main goal is to protect the controller from the extreme users
(DDoS). We want
to protect it from large requests or repetitive requests coming from a
single user.
That user could be used by multiple apps as you pointed out w
Hi David,
Thanks for the KIP. A few questions below:
1) The KIP says: *`Typically, applications tend to send one request to
create all the topics that they need`*. What would the point of throttling
be in this case? If there was a user quota for the principal used by that
application, wouldn't we
Hi Jun,
Thanks for your reply.
10. I think that both options are likely equivalent from an accuracy point
of
view. If we put the implementation aside, conceptually, I am not convinced
by the used based approach because resources don't have a clear owner
in AK at the moment. A topic can be created
Hi, David, Anna,
Thanks for the response. Sorry for the late reply.
10. Regarding exposing rate or usage as quota. Your argument is that usage
is not very accurate anyway and is harder to implement. So, let's just be a
bit loose and expose rate. I am sort of neutral on that. (1) It seems to me
th
Hi folks,
I have updated the KIP. As mentioned by Jun, I have made the
quota per principal/clientid similarly to the other quotas. I have
also explained how this will work in conjunction with the auto
topics creation.
Please, take a look at it. I plan to call a vote for it in the next few
days if
Hi David,
Thanks for the explanation and confirmation that evolving the APIs is not
off the table in the longer term.
Kind regards,
Tom
Hi Tom,
>> What exactly is the problem with having a huge backlog of pending
>> operations? I can see that the backlog would need persisting so that the
>> controller could change without losing track of the topics to be mutated,
>> and the mutations would need to be submitted in batches to the co
Hi Jun,
Coming back to your question regarding the differences between the token
bucket algorithm and our current quota mechanism. I did some tests and
they confirmed my first intuition that our current mechanism does not work
well with a bursty workload. Let me try to illustrate the difference wi
Hi David,
Thanks for the reply.
>> If I understand the proposed throttling algorithm, an initial request
> would
> >> be allowed (possibly making K negative) and only subsequent requests
> >> (before K became positive) would receive the QUOTA_VIOLATED. That would
> >> mean it was still possible t
Hi Tom,
Thanks for the feedback.
>> If I understand the proposed throttling algorithm, an initial request
would
>> be allowed (possibly making K negative) and only subsequent requests
>> (before K became positive) would receive the QUOTA_VIOLATED. That would
>> mean it was still possible to block
Hi Anna and Jun,
Anna, thanks for your thoughtful feedback. Overall, I agree with what you
said. If I summarize, you said that using time on server threads is not
easier
to tune than a rate based approach and it does not really capture all the
load neither as the control requests are not taken int
Hi David and Jun,
I wanted to add to the discussion about using requests/sec vs. time on
server threads (similar to request quota) for expressing quota for topic
ops.
I think request quota does not protect the brokers from overload by itself
-- it still requires tuning and sometimes re-tuning, be
Hi David,
Thanks for the KIP.
If I understand the proposed throttling algorithm, an initial request would
be allowed (possibly making K negative) and only subsequent requests
(before K became positive) would receive the QUOTA_VIOLATED. That would
mean it was still possible to block the controller
Hi, David,
Thanks for the reply. A few more comments.
1. I am actually not sure if a quota based on request rate is easier for
the users. For context, in KIP-124, we started with a request rate quota,
but ended up not choosing it. The main issues are (a) requests are not
equal; some are more expe
Hi Jun,
Thank you for the feedback.
1. You are right. At the end, we do care about the percentage of time that
an operation ties up the controller thread. I thought about this but I was
not entirely convinced by it for following reasons:
1.1. While I do agree that setting up a rate and a burst i
Hi, David,
Thanks for the KIP. A few quick comments.
1. About quota.partition.mutations.rate. I am not sure if it's very easy
for the user to set the quota as a rate. For example, each partition
mutation could take a different number of ZK operations (depending on
things like retry). The time to
36 matches
Mail list logo
