Jay Kreps [jay.kr...@gmail.com]
> > Sent: Friday, April 24, 2015 3:20 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> >
> > Hey Jun/Joel,
> >
> > Yeah we will definitely want to quota non-produce/consume requests.
> >
https://reviews.apache.org/r/33378/
Thanks,
Aditya
From: Joel Koshy [jjkosh...@gmail.com]
Sent: Friday, April 24, 2015 5:34 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
I think Jay meant a catch-all request/sec limit on all request
ontrol be disabled/enabled without affect
> > anything
> > > else? From the design wiki page, it seems to me that each request will at
> > > least pay a penalty of checking quota enablement.
> > >
> > > Thanks.
> > >
> > &
_
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Friday, April 24, 2015 3:20 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Hey Jun/Joel,
>
> Yeah we will definitely want to quota non-produce/consume reque
metric per-client and a good default quota.
Thanks,
Aditya
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Friday, April 24, 2015 3:20 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Hey Jun/Joel,
Yeah we will definitely want to quota n
n and this avoids any potential
> > performance
> > > > penalties we were concerned about earlier.
> > > >
> > > > 2. Delayed Fetch Requests - Similarly, the proposal here is to reuse
> > the
> > > > DelayedFetch objects and insert them into the purgat
is to reuse
> > the
> > > > DelayedFetch objects and insert them into the purgatory with no
> watcher
> > > > keys if the request is throttled. Timeout used is the
> > Max(quota_delay_time,
> > > > max_wait_timeout). Having no watch keys provides the same bene
t; > proceeds normally i.e. perform a readFromLocalLog and return a
> response.
> > > The caveat here is that if the request is throttled but the throttle
> time
> > > is less than the max_wait timeout on the fetch request, the request
> will be
> > > delayed to
..@us.ibm.com
>
>
>
> From: Joel Koshy
> To: dev@kafka.apache.org
> Date: 04/21/2015 01:22 PM
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
>
>
> Given the caveats, it may be worth doing further investigation on the
> alternate approach which is
fetch response) is still under discussion, and
> > may not be got it in time with KAFKA-1927.
> >
> > On Thu, Apr 9, 2015 at 4:49 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > I think it's reasonable to batch the protocol chan
oughts?
>
> Thanks,
> Aditya
>
>
> From: Guozhang Wang [wangg...@gmail.com]
> Sent: Monday, April 13, 2015 7:27 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> I think KAFKA-2063 (bounding fetc
.com]
Sent: Thursday, April 16, 2015 9:06 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Hi Adi,
2. I assume you were saying "than strictly needed for replications" here?
Also the concern I have is around error code: today if the replication is
not finished withi
ions with no watcher keys. By inserting
> elements
> > with no watch keys, the purgatory simply becomes a delay queue. It may
> also
> > make sense to add a new API to the purgatory called
> > delayFor() that basically accepts an operation without any watch keys
> > (Thanks for th
ments have been received.
Thanks,
Aditya
From: Guozhang Wang [wangg...@gmail.com]
Sent: Wednesday, April 15, 2015 3:42 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Thanks for the summary. A few comments below:
1. Say a produce r
API to the purgatory called
> delayFor() that basically accepts an operation without any watch keys
> (Thanks for the suggestion Joel).
>
> Thoughts?
>
> Thanks,
> Aditya
>
>
> From: Guozhang Wang [wangg...@gmail.com]
> Sent: Monday, April 13, 2015 7:27 PM
> To: dev@kafka
wangg...@gmail.com]
Sent: Monday, April 13, 2015 7:27 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
I think KAFKA-2063 (bounding fetch response) is still under discussion, and
may not be got it in time with KAFKA-1927.
On Thu, Apr 9, 2015 at 4:49 PM, Aditya Auradkar <
t; > because we choose client based quotas where there is no notion of
> > partitions. I've explained in a bit more detail in that section.
> >
> > Aditya
> >
> > ________________________
> > From: Joel Koshy [jjkosh...@gmail.com]
> >
t more detail in that section.
>
> Aditya
>
>
> From: Joel Koshy [jjkosh...@gmail.com]
> Sent: Wednesday, April 08, 2015 6:30 AM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Thanks for updating the wiki. Looks great
ection.
Aditya
From: Joel Koshy [jjkosh...@gmail.com]
Sent: Wednesday, April 08, 2015 6:30 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Thanks for updating the wiki. Looks great overall. Just a couple
more comments:
Client status code:
- v0 requests -&g
__
> From: Gwen Shapira [gshap...@cloudera.com]
> Sent: Tuesday, April 07, 2015 11:31 AM
> To: Sriharsha Chintalapani
> Cc: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Yeah, I was not suggesting adding auth to metrics - I think this needlessly
> c
oing to start a voting
thread.
Thanks,
Aditya
From: Gwen Shapira [gshap...@cloudera.com]
Sent: Tuesday, April 07, 2015 11:31 AM
To: Sriharsha Chintalapani
Cc: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Yeah, I was not suggesting adding auth
quest latency metrics. Basically, we want
>>an
>> > easy
>> > > > way for clients to be aware if they are being throttled.
>> > > >
>> > > > 4. For purgatory v delay queue, I think we are on the same page. I
>> feel
>> > &
> > > > know how much to back off. So perhaps this and config-management
> > > > > > should be moved to a separate discussion, but it would be good
> to
> > > have
> > > > > > this discussion going and incorporated into the first quota
>
> feel
> > > it
> > > > is nicer to use the purgatory but I'm happy to use a DelayQueue if
> > there
> > > > are performance implications. I don't know enough about the current
> and
> > > > Yasuhiro's new implementation t
gh about the current and
> > > > Yasuhiro's new implementation to be sure one way or the other.
> > > >
> > > > Stepping back, I think these two things are the only remaining point of
> > > > discussion within the current proposal. Any con
; > > > are performance implications. I don't know enough about the current
> and
> > > > Yasuhiro's new implementation to be sure one way or the other.
> > > >
> > > > Stepping back, I think these two things are the only remaining point
>
Jay Kreps [jay.kr...@gmail.com]
Sent: Tuesday, April 07, 2015 7:41 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Totally. But is that the only use? What I wanted to flesh out was whether
the goal was:
1. Expose throttling in the client metrics
2. Enable programmatic response
ow.
> Tong Li
> OpenStack & Kafka Community Development
> Building 501/B205
> liton...@us.ibm.com
>
> Jay Kreps wrote on 04/07/2015 10:41:19 AM:
>
> > From: Jay Kreps
> > To: "dev@kafka.apache.org"
> > Date: 04/07/2015 10:44 AM
> > Subject: Re:
see some response inline below.
Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com
Jay Kreps wrote on 04/07/2015 10:41:19 AM:
> From: Jay Kreps
> To: "dev@kafka.apache.org"
> Date: 04/07/2015 10:44 AM
> Subject: Re: [KIP-DISCUSSION]
7;m inclined to agree with Joel that this is
> > good
> > > > to have in the initial implementation.
> > > >
> > > > 3. Config - +1. I'll remove it from the KIP. We can discuss this in
> > > > parallel.
> > > >
> > > > 4. Purg
point of
> > discussion within the current proposal. Any concerns if I started a
> voting
> > thread on the proposal after the KIP discussion tomorrow? (assuming we
> > reach consensus on these items)
> >
> > Thanks,
> > Aditya
> >
> > From: Jay
___
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Saturday, April 04, 2015 1:36 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Hey Aditya,
>
> 2. For the return flag I'm not terribly particular. If we
here is already a max wait. The completion
> > criteria can check if minWait time has elapsed before declaring the
> > operation complete. For this to impact performance, a significant number
> of
> > clients may need to exceed their quota at the same time and even then I'm
>
act. Two layers of delays might add
> complexity to the implementation which I'm hoping to avoid.
>
> Aditya
>
> ____________
> From: Joel Koshy [jjkosh...@gmail.com]
> Sent: Friday, April 03, 2015 12:48 PM
> To: dev@kafka.apache.org
> Subje
complexity to the implementation which I'm hoping to avoid.
>
> Aditya
>
> ________________________
> From: Joel Koshy [jjkosh...@gmail.com]
> Sent: Friday, April 03, 2015 12:48 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
rs of delays might add
> complexity to the implementation which I'm hoping to avoid.
>
> Aditya
>
>
> From: Joel Koshy [jjkosh...@gmail.com]
> Sent: Friday, April 03, 2015 12:48 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
I'm hoping to
avoid.
Aditya
From: Joel Koshy [jjkosh...@gmail.com]
Sent: Friday, April 03, 2015 12:48 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Aditya, thanks for the updated KIP and Jay/Jun thanks for the
comments. Couple
gt; https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
>>
>> Please take a look and let me know if there are any concerns.
>>
>> Thanks,
>> Aditya
>> ________________
>> From: Aditya Auradkar
>> Sent: Friday, April
Aditya
>
> From: Aditya Auradkar
> Sent: Friday, April 03, 2015 10:10 AM
> To: dev@kafka.apache.org
> Subject: RE: [KIP-DISCUSSION] KIP-13 Quotas
>
> Thanks Jun.
>
> Some thoughts:
>
> 10) I think it is better we throttle re
: Aditya Auradkar
Sent: Friday, April 03, 2015 10:10 AM
To: dev@kafka.apache.org
Subject: RE: [KIP-DISCUSSION] KIP-13 Quotas
Thanks Jun.
Some thoughts:
10) I think it is better we throttle regardless of the produce/fetch version.
This is a nice feature where clients can tell if they are being throttled
...@confluent.io]
Sent: Friday, April 03, 2015 9:16 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Thanks for the update.
10. About whether to return a new field in the response to indicate
throttling. Earlier, the plan was to not change the response format and
just have a metric on
ance to read/comment.
>
> Thanks,
> Aditya
>
>
> From: Steven Wu [stevenz...@gmail.com]
> Sent: Friday, March 20, 2015 9:14 AM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> +1 on Jun's suggestion of maintaining one set/style of metrics
ks,
Aditya
From: Steven Wu [stevenz...@gmail.com]
Sent: Friday, March 20, 2015 9:14 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
+1 on Jun's suggestion of maintaining one set/style of metrics at broker.
In Netflix, we have to convert the yammer metrics to s
gt; > > > Hey everyone,
> > >> > > > >
> > >> > > > > Thanks for the great discussion. There are currently a few
> > points
> > >> on
> > >> > > this
> > >> > > > > KIP that need addressi
and I want to make sure we are on the
> >> same
> >> > > page
> >> > > > > about those.
> >> > > > >
> >> > > > > 1. Append and delay response vs delay and return error
> >> > > > > - I think we've discussed the pros and cons of each app
t; > > > - I think we've discussed the pros and cons of each approach but
>> > > haven't
>> > > > > chosen an approach yet. Where does everyone stand on this issue?
>> > > > >
>> > > > > 2. Metrics Migration and usage in quotas
>> > > > > - The metrics l
reuse. For that to happen, we need to migrate the server to the new
> > > metrics
> > > > > package.
> > > > > - Need more clarification on how to compute throttling time and
> > > windowing
> > > > > for quotas.
> > > > >
> >
ed to migrate the server to the new
> > metrics
> > > > package.
> > > > - Need more clarification on how to compute throttling time and
> > windowing
> > > > for quotas.
> > > >
> > > > I'm going to start a new KIP to discuss metrics migr
ing time and
> windowing
> > > for quotas.
> > >
> > > I'm going to start a new KIP to discuss metrics migration separately.
> That
> > > will also contain a section on quotas.
> > >
> > > 3. Dynamic Configuration management
ow per-client
> > overrides.
> >
> > Is there something else that I'm missing?
> >
> > Thanks,
> > Aditya
> >
> > From: Jay Kreps [jay.kr...@gmail.com]
> > Sent: Wednesday, March 18, 2015 2:10 PM
&g
re time (in 1 second increments) and finding the first time unit when
the check doesn't fail.
Thanks,
Aditya
From: Jun Rao [j...@confluent.io]
Sent: Thursday, March 19, 2015 9:24 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Yeah originally our plan was errors when the quota was exceeded, enforced
at a per-topic level. But after some discussion this doesn't work that
well. The problem is the consumer. Popular topics have many consumers, any
one of which can either misbehave or do some kind of "catch-up" or full
reload.
hing that will model default quotas and allow per-client
> overrides.
>
> Is there something else that I'm missing?
>
> Thanks,
> Aditya
> ____
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Wednesday, March 18, 2015 2:10 PM
> To
> the quota is against all of those instances added up across all their
topics
ah. I miss this point earlier. so you intend to enforce quota at client-id
level cross all topics. I thought quota is at per-topic level. when we have
sudden surge of traffic, it is usually just a topic (not all topics
- Being discussed in KIP-5. Basically
> we need something that will model default quotas and allow per-client
> overrides.
>
> Is there something else that I'm missing?
>
> Thanks,
> Aditya
> ________
> From: Jay Kreps [jay.kr...@gm
l model default quotas and allow per-client overrides.
Is there something else that I'm missing?
Thanks,
Aditya
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Wednesday, March 18, 2015 2:10 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quot
Hey Steven,
The current proposal is actually to enforce quotas at the
client/application level, NOT the topic level. So if you have a service
with a few dozen instances the quota is against all of those instances
added up across all their topics. So actually the effect would be the same
either way
Is it possible these 3 options during the next KIP hangout?
Aditya
From: Steven Wu [stevenz...@gmail.com]
Sent: Tuesday, March 17, 2015 10:08 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Jay,
let's say an app produces
Jay,
let's say an app produces to 10 different topics. one of the topic is sent
from a library. due to whatever condition/bug, this lib starts to send
messages over the quota. if we go with the delayed response approach, it
will cause the whole shared RecordAccumulator buffer to be filled up. that
Hey Guozhang,
Cool, I think we are mostly on the same page. I think I agree with what you
are saying, the error code thing is basically a matter of taste. It would
be possible to put all kinds of things as error codes and it would be
possible to devise a scheme for clients to deal with it--for exa
Hey Steven,
It is true that hitting the quota will cause back-pressure on the producer.
But the solution is simple, a producer that wants to avoid this should stay
under its quota. In other words this is a contract between the cluster and
the client, with each side having something to uphold. Quit
Yes I think there are two policy issues but I don't think they are
separate/mutually exclusive for the purposes of this discussion - the
first being what should the broker do if quota is violated and second,
what should it return to the client (error code or status code). (The
separate discussion u
Ewen, I see your point regarding the shared buffer. yes, a bad/slow broker
could potentially consume up all buffer. On the other hand, I do like the
batching behavior of shared RecordAccumulator buffer.
On Tue, Mar 17, 2015 at 8:25 AM, Guozhang Wang wrote:
> Ewen,
>
> 1. I think we are on the sa
Ewen,
1. I think we are on the same page as per "malicious clients", that it
should not be the target of either approach. I was just trying to separate
the discussion from "what if user just keep retrying" and maybe I was not
clear.
2. I was not advocating option A on the wiki, in my previous ema
Steven - that's a reasonable concern. I think I've mentioned the same sort
of issue in the issues about the new producer's RecordAccumulator not
timing out sends, e.g. in https://issues.apache.org/jira/browse/KAFKA-1788.
The shared buffer causes problems if one broker isn't available for awhile
sin
wait. we create one kafka producer for each cluster. each cluster can have
many topics. if producer buffer got filled up due to delayed response for
one throttled topic, won't that penalize other topics unfairly? it seems to
me that broker should just return error without delay.
sorry that I am ch
I think I can answer my own question. delayed response will cause the
producer buffer to be full, which then result in either thread blocking or
message drop.
On Mon, Mar 16, 2015 at 11:24 PM, Steven Wu wrote:
> please correct me if I am missing sth here. I am not understanding how
> would throt
please correct me if I am missing sth here. I am not understanding how
would throttle work without cooperation/back-off from producer. new Java
producer supports non-blocking API. why would delayed response be able to
slow down producer? producer will continue to fire async sends.
On Mon, Mar 16,
I think we are really discussing two separate issues here:
1. Whether we should a) append-then-block-then-returnOKButThrottled or b)
block-then-returnFailDuetoThrottled for quota actions on produce requests.
Both these approaches assume some kind of well-behaveness of the clients:
option a) assum
You are right, shoe-horning status into an error field is a bad idea.
While many projects use a single "status" field to indicate different
error and non-error states, it doesn't seem like a good fit for the
current Kafka implementation.
Do you think that adding a "status" field to our protocol is
Agreed that trying to shoehorn non-error codes into the error field is a
bad idea. It makes it *way* too easy to write code that looks (and should
be) correct but is actually incorrect. If necessary, I think it's much
better to to spend a couple of extra bytes to encode that information
separately
We discussed an error code for rate-limiting (which I think made
sense), isn't it a similar case?
On Mon, Mar 16, 2015 at 10:10 PM, Jay Kreps wrote:
> My concern is that as soon as you start encoding non-error response
> information into error codes the next question is what to do if two such
> c
My concern is that as soon as you start encoding non-error response
information into error codes the next question is what to do if two such
codes apply (i.e. you have a replica down and the response is quota'd). I
think I am trying to argue that error should mean "why we failed your
request", for
I think its not too late to reserve a set of error codes (200-299?)
for "non-error" codes.
It won't be backward compatible (i.e. clients that currently do "else
throw" will throw on non-errors), but perhaps its worthwhile.
On Mon, Mar 16, 2015 at 9:42 PM, Jay Kreps wrote:
> Hey Jun,
>
> I'd real
Hey Jun,
I'd really really really like to avoid that. Having just spent a bunch of
time on the clients, using the error codes to encode other information
about the response is super dangerous. The error handling is one of the
hardest parts of the client (Guozhang chime in here).
Generally the err
16, 2015 4:27 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
It's probably useful for a client to know whether its requests are
throttled or not (e.g., for monitoring and alerting). From that
perspective, option B (delay the requests and return an error) seems better.
Thanks,
Ju
It's probably useful for a client to know whether its requests are
throttled or not (e.g., for monitoring and alerting). From that
perspective, option B (delay the requests and return an error) seems better.
Thanks,
Jun
On Wed, Mar 4, 2015 at 3:51 PM, Aditya Auradkar <
aaurad...@linkedin.com.inv
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Thursday, March 12, 2015 10:35 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Actually I think we are making the same argument. request.timeout.ms seems
like it is a timeout after which your request will be sent back as
Actually I think we are making the same argument. request.timeout.ms seems
like it is a timeout after which your request will be sent back as an
error. Actually this is not true. There are many many things that can cause
a request to be slow--sitting in queue on the server, waiting on slow I/O,
GC
We're getting off the KIP a little bit here, but while I understand the
idea of not honoring the request timeout, I think that the broker ignoring
it without telling the client is not a good implementation. It gets back to
a larger discussion, which I was mentioning, of just negotiating with (or
at
Hi Aditya,
I just wanted to give you use case of rate limiting that we have
implemented with producer which is a work around:
Use Case 1:
1) topic based rate limiting per producer instance (not across multiple
instance of producers yet, we have producer which we send Heartbeat and
regular messag
Hey Todd,
Yeah it is kind of weird to do the quota check after taking a request, but
since the penalty is applied during that request and it just delays you to
the right rate, I think it isn't exactly wrong. I admit it is weird, though.
What you say about closing the connection makes sense. The i
will only be delayed for another 3 seconds. Is there a reason
> this isn't good enough? I'm also going to dig into the existing quota
> package and see what we can leverage.
>
> 7. Exposing quota usage: We do plan to expose the quota metrics via JMX.
> Adding an API to
Thanks, Jay. On the interface, I agree with Aditya (and you, I believe)
that we don't need to expose the public API contract at this time, but
structuring the internal logic to allow for it later with low cost is a
good idea.
Glad you explained the thoughts on where to hold requests. While my gut
what we
can leverage.
7. Exposing quota usage: We do plan to expose the quota metrics via JMX. Adding
an API to query quota status is absolutely feasible and also a good idea.
Thanks,
Aditya
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Monday, March 09, 2015 5
Hey Todd,
Nice points, let me try to respond:
Plugins
Yeah let me explain what I mean about plugins. The contracts that matter
for us are public contracts, i.e. the protocol, the binary format, stuff in
zk, and the various plug-in apis we expose. Adding an internal interface
later will not be ha
First, a couple notes on this...
3 - I generally agree with the direction of not pre-optimizing. However, in
this case I'm concerned about the calculation of the cost of doing plugins
now vs. trying to refactor the code to do it later. It would seem to me
that doing it up front will have less fric
Hey Adi,
Great write-up. Here are some comments:
1. I don't think you need a way to disable quotas on a per-client basis,
that is just the equivalent of setting the quota to be infinite, right?
2. I agree that the configuration problem is a general part of doing
dynamic configuration, and it is
88 matches
Mail list logo
