The quota check for the fetch request is a bit different from the produce
request. I assume that for the fetch request, we will first get an
estimated fetch response size to do the quota check. There are two things
to think about. First, when we actually send the response, we probably
don't want to record the metric again since it will double count. Second,
the bytes that the fetch response actually sends could be more than the
estimate. This means that the metric may not be 100% accurate. We may be
able to limit the fetch size of each partition to what's in the original
estimate.
For the produce request, I was thinking that another way to do this is to
first figure out the quota_timeout. Then wait in Purgatory for
quota_timeout with no key. If the request is not satisfied in quota_timeout
and (request_timeout > quota_timeout), wait in Purgatory for
(request_timeout - quota_timeout) with the original keys.
Thanks,
Jun
On Tue, Apr 14, 2015 at 5:01 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:
> This is an implementation proposal for delaying requests in quotas using
> the current purgatory. I'll discuss the usage for produce and fetch
> requests separately.
>
> 1. Delayed Produce Requests - Here, the proposal is basically to reuse
> DelayedProduce objects and insert them into the purgatory with no watcher
> keys if the request is being throttled. The timeout used in the request
> should be the Max(quota_delay_time, replication_timeout).
> In most cases, the quota timeout should be greater than the existing
> timeout but in order to be safe, we can use the maximum of these values.
> Having no watch keys will allow the operation to be enqueued directly into
> the timer and will not add any overhead in terms of watching keys (which
> was a concern). In this case, having watch keys is not beneficial since the
> operation must be delayed for a fixed amount of time and there is no
> possibility for the operation to complete before the timeout i.e.
> tryComplete() can never return true before the timeout. On timeout, since
> the operation is a TimerTask, the timer will call run() which calls
> onComplete().
> In onComplete, the DelayedProduce can repeat the check in tryComplete()
> (only if acks=-1 whether all replicas fetched upto a certain offset) and
> return the response immediately.
>
> Code will be structured as follows in ReplicaManager:appendMessages()
>
> if(isThrottled) {
> fetch = new DelayedProduce(timeout)
> purgatory.tryCompleteElseWatch(fetch, Seq())
> }
> else if(delayedRequestRequired()) {
> // Insert into purgatory with watched keys for unthrottled requests
> }
>
> In this proposal, we avoid adding unnecessary watches because there is no
> possibility of early completion and this avoids any potential performance
> penalties we were concerned about earlier.
>
> 2. Delayed Fetch Requests - Similarly, the proposal here is to reuse the
> DelayedFetch objects and insert them into the purgatory with no watcher
> keys if the request is throttled. Timeout used is the Max(quota_delay_time,
> max_wait_timeout). Having no watch keys provides the same benefits as
> described above. Upon timeout, the onComplete() is called and the operation
> proceeds normally i.e. perform a readFromLocalLog and return a response.
> The caveat here is that if the request is throttled but the throttle time
> is less than the max_wait timeout on the fetch request, the request will be
> delayed to a Max(quota_delay_time, max_wait_timeout). This may be more than
> strictly necessary (since we are not watching for
> satisfaction on any keys).
>
> I added some testcases to DelayedOperationTest to verify that it is
> possible to schedule operations with no watcher keys. By inserting elements
> with no watch keys, the purgatory simply becomes a delay queue. It may also
> make sense to add a new API to the purgatory called
> delayFor() that basically accepts an operation without any watch keys
> (Thanks for the suggestion Joel).
>
> Thoughts?
>
> Thanks,
> Aditya
>
> ________________________________________
> From: Guozhang Wang [wangg...@gmail.com]
> Sent: Monday, April 13, 2015 7:27 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> I think KAFKA-2063 (bounding fetch response) is still under discussion, and
> may not be got it in time with KAFKA-1927.
>
> On Thu, Apr 9, 2015 at 4:49 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > I think it's reasonable to batch the protocol changes together. In
> > addition to the protocol changes, is someone actively driving the server
> > side changes/KIP process for KAFKA-2063?
> >
> > Thanks,
> > Aditya
> >
> > ________________________________________
> > From: Jun Rao [j...@confluent.io]
> > Sent: Thursday, April 09, 2015 8:59 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> >
> > Since we are also thinking about evolving the fetch request protocol in
> > KAFKA-2063 (bound fetch response size), perhaps it's worth thinking
> through
> > if we can just evolve the protocol once.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Apr 8, 2015 at 10:43 AM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Thanks for the detailed review. I've addressed your comments.
> > >
> > > For rejected alternatives, we've rejected per-partition distribution
> > > because we choose client based quotas where there is no notion of
> > > partitions. I've explained in a bit more detail in that section.
> > >
> > > Aditya
> > >
> > > ________________________________________
> > > From: Joel Koshy [jjkosh...@gmail.com]
> > > Sent: Wednesday, April 08, 2015 6:30 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > >
> > > Thanks for updating the wiki. Looks great overall. Just a couple
> > > more comments:
> > >
> > > Client status code:
> > > - v0 requests -> current version (0) of those requests.
> > > - Fetch response has a throttled flag instead of throttle time - I
> > > think you intended the latter.
> > > - Can you make it clear that the quota status is a new field
> > > called throttleTimeMs (or equivalent). It would help if some of
> > > that is moved (or repeated) in compatibility/migration plan.
> > > - So you would need to upgrade brokers first, then the clients.
> > > While upgrading the brokers (via a rolling bounce) the brokers
> > > cannot start using the latest fetch-request version immediately
> > > (for replica fetches). Since there will be older brokers in the mix
> > > those brokers would not be able to read v1 fetch requests. So all
> > > the brokers should be upgraded before switching to the latest
> > > fetch request version. This is similar to what Gwen proposed in
> > > KIP-2/KAFKA-1809 and I think we will need to use the
> > > inter-broker protocol version config.
> > >
> > > Rejected alternatives-quota-distribution.B: notes that this is the
> > > most elegant model, but does not explain why it was rejected. I
> > > think this was because we would then need some sort of gossip
> > > between brokers since partitions are across the cluster. Can you
> > > confirm?
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Wed, Apr 08, 2015 at 05:45:34AM +0000, Aditya Auradkar wrote:
> > > > Hey everyone,
> > > >
> > > > Following up after today's hangout. After discussing the client side
> > > metrics piece internally, we've incorporated that section into the KIP.
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > >
> > > > Since there appears to be sufficient consensus, I'm going to start a
> > > voting thread.
> > > >
> > > > Thanks,
> > > > Aditya
> > > > ________________________________________
> > > > From: Gwen Shapira [gshap...@cloudera.com]
> > > > Sent: Tuesday, April 07, 2015 11:31 AM
> > > > To: Sriharsha Chintalapani
> > > > Cc: dev@kafka.apache.org
> > > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > >
> > > > Yeah, I was not suggesting adding auth to metrics - I think this
> > > needlessly
> > > > complicates everything.
> > > > But we need to assume that client developers will not have access to
> > the
> > > > broker metrics (because in secure environment they probably won't).
> > > >
> > > > Gwen
> > > >
> > > > On Tue, Apr 7, 2015 at 11:20 AM, Sriharsha Chintalapani <
> > ka...@harsha.io
> > > >
> > > > wrote:
> > > >
> > > > > Having auth on top of metrics is going to be lot more difficult.
> How
> > > are
> > > > > we going to restrict metrics reporter which run as part of kafka
> > server
> > > > > they will have access to all the metrics and they can publish to
> > > ganglia
> > > > > etc.. I look at the metrics as a read-only info. As you said
> metrics
> > > for
> > > > > all the topics can be visible but what actions are we looking that
> > can
> > > be
> > > > > non-secure based on metrics alone? . This probably can be part of
> > > KIP-11
> > > > > discussion.
> > > > > Having said that it will be great if the throttling details can be
> > > > > exposed as part of the response to the client. Instead of looking
> at
> > > > > metrics , client can depend on the response to slow down if its
> being
> > > > > throttled. This allows us the clients can be self-reliant based on
> > the
> > > > > response .
> > > > >
> > > > > --
> > > > > Harsha
> > > > >
> > > > >
> > > > > On April 7, 2015 at 9:55:41 AM, Gwen Shapira (
> gshap...@cloudera.com)
> > > > > wrote:
> > > > >
> > > > > Re (1):
> > > > > We have no authorization story on the metrics collected by brokers,
> > so
> > > I
> > > > > assume that access to broker metrics means knowing exactly which
> > topics
> > > > > exist and their throughputs. (Prath and Don, correct me if I got it
> > > > > wrong...)
> > > > > Secure environments will strictly control access to this
> information,
> > > so I
> > > > > am pretty sure the client developers will not have access to server
> > > > > metrics
> > > > > at all.
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Tue, Apr 7, 2015 at 7:41 AM, Jay Kreps <jay.kr...@gmail.com>
> > wrote:
> > > > >
> > > > > > Totally. But is that the only use? What I wanted to flesh out was
> > > > > whether
> > > > > > the goal was:
> > > > > > 1. Expose throttling in the client metrics
> > > > > > 2. Enable programmatic response (i.e. stop sending stuff or
> > something
> > > > > like
> > > > > > that)
> > > > > >
> > > > > > I think I kind of understand (1) but let's get specific on the
> > > metric we
> > > > > > would be adding and what exactly you would expose in a dashboard.
> > For
> > > > > > example if the goal is just monitoring do I really want a boolean
> > > flag
> > > > > for
> > > > > > is_throttled or do I want to know how much I am being throttled
> > (i.e.
> > > > > > throttle_pct might indicate the percent of your request time that
> > was
> > > > > due
> > > > > > to throttling or something like that)? If I am 1% throttled that
> > may
> > > be
> > > > > > irrelevant but 99% throttled would be quite relevant? Not sure I
> > > agree,
> > > > > > just throwing that out there...
> > > > > >
> > > > > > For (2) the prior discussion seemed to kind of allude to this
> but I
> > > > > can't
> > > > > > really come up with a use case. Is there one?
> > > > > >
> > > > > > If it is just (1) I think the question is whether it really helps
> > > much
> > > > > to
> > > > > > have the metric on the client vs the server. I suppose this is a
> > bit
> > > > > > environment specific. If you have a central metrics system it
> > > shouldn't
> > > > > > make any difference, but if you don't I suppose it does.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Mon, Apr 6, 2015 at 7:57 PM, Gwen Shapira <
> > gshap...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Here's a wild guess:
> > > > > > >
> > > > > > > An app developer included a Kafka Producer in his app, and is
> not
> > > > > happy
> > > > > > > with the throughput. He doesn't have visibility into the
> brokers
> > > since
> > > > > > they
> > > > > > > are owned by a different team. Obviously the first instinct of
> a
> > > > > > developer
> > > > > > > who knows that throttling exists is to blame throttling for any
> > > > > slowdown
> > > > > > in
> > > > > > > the app.
> > > > > > > If he doesn't have a way to know from the responses whether or
> > not
> > > his
> > > > > > app
> > > > > > > is throttled, he may end up calling Aditya at 4am asked "Hey,
> is
> > my
> > > > > app
> > > > > > > throttled?".
> > > > > > >
> > > > > > > I assume Aditya is trying to avoid this scenario.
> > > > > > >
> > > > > > > On Mon, Apr 6, 2015 at 7:47 PM, Jay Kreps <jay.kr...@gmail.com
> >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hey Aditya,
> > > > > > > >
> > > > > > > > 2. I kind of buy it, but I really like to understand the
> > details
> > > of
> > > > > the
> > > > > > > use
> > > > > > > > case before we make protocol changes. What changes are you
> > > proposing
> > > > > in
> > > > > > > the
> > > > > > > > clients for monitoring and how would that be used?
> > > > > > > >
> > > > > > > > -Jay
> > > > > > > >
> > > > > > > > On Mon, Apr 6, 2015 at 10:36 AM, Aditya Auradkar <
> > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > Hi Jay,
> > > > > > > > >
> > > > > > > > > 2. At this time, the proposed response format changes are
> > only
> > > for
> > > > > > > > > monitoring/informing clients. As Jun mentioned, we get
> > instance
> > > > > level
> > > > > > > > > monitoring in this case since each instance that got
> > throttled
> > > > > will
> > > > > > > have
> > > > > > > > a
> > > > > > > > > metric confirming the same. Without client level monitoring
> > for
> > > > > this,
> > > > > > > > it's
> > > > > > > > > hard for application developers to find if they are being
> > > > > throttled
> > > > > > > since
> > > > > > > > > they will also have to be aware of all the brokers in the
> > > cluster.
> > > > > > This
> > > > > > > > is
> > > > > > > > > quite problematic for large clusters.
> > > > > > > > >
> > > > > > > > > It seems nice for app developers to not have to think about
> > > kafka
> > > > > > > > internal
> > > > > > > > > metrics and only focus on the metrics exposed on their
> > > instances.
> > > > > > > > Analogous
> > > > > > > > > to having client-sde request latency metrics. Basically, we
> > > want
> > > > > an
> > > > > > > easy
> > > > > > > > > way for clients to be aware if they are being throttled.
> > > > > > > > >
> > > > > > > > > 4. For purgatory v delay queue, I think we are on the same
> > > page. I
> > > > > > feel
> > > > > > > > it
> > > > > > > > > is nicer to use the purgatory but I'm happy to use a
> > > DelayQueue if
> > > > > > > there
> > > > > > > > > are performance implications. I don't know enough about the
> > > > > current
> > > > > > and
> > > > > > > > > Yasuhiro's new implementation to be sure one way or the
> > other.
> > > > > > > > >
> > > > > > > > > Stepping back, I think these two things are the only
> > remaining
> > > > > point
> > > > > > of
> > > > > > > > > discussion within the current proposal. Any concerns if I
> > > started
> > > > > a
> > > > > > > > voting
> > > > > > > > > thread on the proposal after the KIP discussion tomorrow?
> > > > > (assuming
> > > > > > we
> > > > > > > > > reach consensus on these items)
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aditya
> > > > > > > > > ________________________________________
> > > > > > > > > From: Jay Kreps [jay.kr...@gmail.com]
> > > > > > > > > Sent: Saturday, April 04, 2015 1:36 PM
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > >
> > > > > > > > > Hey Aditya,
> > > > > > > > >
> > > > > > > > > 2. For the return flag I'm not terribly particular. If we
> > want
> > > to
> > > > > add
> > > > > > > it
> > > > > > > > > let's fully think through how it will be used. The only
> > > concern I
> > > > > > have
> > > > > > > is
> > > > > > > > > adding to the protocol without really thinking through the
> > use
> > > > > cases.
> > > > > > > So
> > > > > > > > > let's work out the APIs we want to add to the Java consumer
> > and
> > > > > > > producer
> > > > > > > > > and the use cases for how clients will make use of these.
> For
> > > my
> > > > > > part I
> > > > > > > > > actually don't see much use other than monitoring since it
> > > isn't
> > > > > an
> > > > > > > error
> > > > > > > > > condition to be at your quota. And if it is just
> monitoring I
> > > > > don't
> > > > > > > see a
> > > > > > > > > big enough difference between having the monitoring on the
> > > > > > server-side
> > > > > > > > > versus in the clients to justify putting it in the
> protocol.
> > > But I
> > > > > > > think
> > > > > > > > > you guys may have other use cases in mind of how a client
> > would
> > > > > make
> > > > > > > some
> > > > > > > > > use of this? Let's work that out. I also don't feel
> strongly
> > > about
> > > > > > > it--it
> > > > > > > > > wouldn't be *bad* to have the monitoring available on the
> > > client,
> > > > > > just
> > > > > > > > > doesn't seem that much better.
> > > > > > > > >
> > > > > > > > > 4. For the purgatory vs delay queue I think is arguably
> nicer
> > > to
> > > > > > reuse
> > > > > > > > the
> > > > > > > > > purgatory we just have to be ultra-conscious of
> efficiency. I
> > > > > think
> > > > > > our
> > > > > > > > > goal is to turn quotas on across the board, so at LinkedIn
> > that
> > > > > would
> > > > > > > > mean
> > > > > > > > > potentially every request will need a small delay. I
> haven't
> > > > > worked
> > > > > > out
> > > > > > > > the
> > > > > > > > > efficiency implications of this choice, so as long as we do
> > > that
> > > > > I'm
> > > > > > > > happy.
> > > > > > > > >
> > > > > > > > > -Jay
> > > > > > > > >
> > > > > > > > > On Fri, Apr 3, 2015 at 1:10 PM, Aditya Auradkar <
> > > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Some responses to Jay's points.
> > > > > > > > > >
> > > > > > > > > > 1. Using commas - Cool.
> > > > > > > > > >
> > > > > > > > > > 2. Adding return flag - I'm inclined to agree with Joel
> > that
> > > > > this
> > > > > > is
> > > > > > > > good
> > > > > > > > > > to have in the initial implementation.
> > > > > > > > > >
> > > > > > > > > > 3. Config - +1. I'll remove it from the KIP. We can
> discuss
> > > this
> > > > > in
> > > > > > > > > > parallel.
> > > > > > > > > >
> > > > > > > > > > 4. Purgatory vs Delay queue - I feel that it is simpler
> to
> > > reuse
> > > > > > the
> > > > > > > > > > existing purgatories for both delayed produce and fetch
> > > > > requests.
> > > > > > > IIUC,
> > > > > > > > > all
> > > > > > > > > > we need for quotas is a minWait parameter for
> > > DelayedOperation
> > > > > (or
> > > > > > > > > > something equivalent) since there is already a max wait.
> > The
> > > > > > > completion
> > > > > > > > > > criteria can check if minWait time has elapsed before
> > > declaring
> > > > > the
> > > > > > > > > > operation complete. For this to impact performance, a
> > > > > significant
> > > > > > > > number
> > > > > > > > > of
> > > > > > > > > > clients may need to exceed their quota at the same time
> and
> > > even
> > > > > > then
> > > > > > > > I'm
> > > > > > > > > > not very clear on the scope of the impact. Two layers of
> > > delays
> > > > > > might
> > > > > > > > add
> > > > > > > > > > complexity to the implementation which I'm hoping to
> avoid.
> > > > > > > > > >
> > > > > > > > > > Aditya
> > > > > > > > > >
> > > > > > > > > > ________________________________________
> > > > > > > > > > From: Joel Koshy [jjkosh...@gmail.com]
> > > > > > > > > > Sent: Friday, April 03, 2015 12:48 PM
> > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > >
> > > > > > > > > > Aditya, thanks for the updated KIP and Jay/Jun thanks for
> > the
> > > > > > > > > > comments. Couple of comments in-line:
> > > > > > > > > >
> > > > > > > > > > > 2. I would advocate for adding the return flag when we
> > next
> > > > > bump
> > > > > > > the
> > > > > > > > > > > request format version just to avoid proliferation. I
> > agree
> > > > > this
> > > > > > > is a
> > > > > > > > > > good
> > > > > > > > > > > thing to know about, but at the moment I don't think we
> > > have a
> > > > > > very
> > > > > > > > > well
> > > > > > > > > > > flushed out idea of how the client would actually make
> > use
> > > of
> > > > > > this
> > > > > > > > > info.
> > > > > > > > > > I
> > > > > > > > > >
> > > > > > > > > > I'm somewhat inclined to having something appropriate off
> > the
> > > > > bat -
> > > > > > > > > > mainly because (i) clients really should know that they
> > have
> > > > > been
> > > > > > > > > > throttled (ii) a smart producer/consumer implementation
> > would
> > > > > want
> > > > > > to
> > > > > > > > > > know how much to back off. So perhaps this and
> > > config-management
> > > > > > > > > > should be moved to a separate discussion, but it would be
> > > good
> > > > > to
> > > > > > > have
> > > > > > > > > > this discussion going and incorporated into the first
> quota
> > > > > > > > > > implementation.
> > > > > > > > > >
> > > > > > > > > > > 3. Config--I think we need to generalize the topic
> stuff
> > > so we
> > > > > > can
> > > > > > > > > > override
> > > > > > > > > > > at multiple levels. We have topic and client, but I
> > suspect
> > > > > > "user"
> > > > > > > > and
> > > > > > > > > > > "broker" will also be important. I recommend we take
> > config
> > > > > stuff
> > > > > > > out
> > > > > > > > > of
> > > > > > > > > > > this KIP since we really need to fully think through a
> > > > > proposal
> > > > > > > that
> > > > > > > > > will
> > > > > > > > > > > cover all these types of overrides.
> > > > > > > > > >
> > > > > > > > > > +1 - it is definitely orthogonal to the core quota
> > > > > implementation
> > > > > > > > > > (although necessary for its operability). Having a
> > > > > config-related
> > > > > > > > > > discussion in this KIP would only draw out the discussion
> > and
> > > > > vote
> > > > > > > > > > even if the core quota design looks good to everyone.
> > > > > > > > > >
> > > > > > > > > > So basically I think we can remove the portions on
> dynamic
> > > > > config
> > > > > > as
> > > > > > > > > > well as the response format but I really think we should
> > > close
> > > > > on
> > > > > > > > > > those while the implementation is in progress and before
> > > quotas
> > > > > is
> > > > > > > > > > officially released.
> > > > > > > > > >
> > > > > > > > > > > 4. Instead of using purgatories to implement the delay
> > > would
> > > > > it
> > > > > > > make
> > > > > > > > > more
> > > > > > > > > > > sense to just use a delay queue? I think all the
> > additional
> > > > > stuff
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > purgatory other than the delay queue doesn't make sense
> > as
> > > the
> > > > > > > quota
> > > > > > > > > is a
> > > > > > > > > > > hard N ms penalty with no chance of early eviction. If
> > > there
> > > > > is
> > > > > > no
> > > > > > > > perf
> > > > > > > > > > > penalty for the full purgatory that may be fine (even
> > > good) to
> > > > > > > reuse,
> > > > > > > > > > but I
> > > > > > > > > > > haven't looked into that.
> > > > > > > > > >
> > > > > > > > > > A simple delay queue sounds good - I think Aditya was
> also
> > > > > trying
> > > > > > to
> > > > > > > > > > avoid adding a new quota purgatory. i.e., it may be
> > possible
> > > to
> > > > > use
> > > > > > > > > > the existing purgatory instances to enforce quotas. That
> > may
> > > be
> > > > > > > > > > simpler, but would be incur a slight perf penalty if too
> > many
> > > > > > clients
> > > > > > > > > > are being throttled.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Joel
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -Jay
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 3, 2015 at 10:45 AM, Aditya Auradkar <
> > > > > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Update, I added a proposal on doing dynamic client
> based
> > > > > > > > configuration
> > > > > > > > > > >> that can be used for quotas.
> > > > > > > > > > >>
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > > >>
> > > > > > > > > > >> Please take a look and let me know if there are any
> > > concerns.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Aditya
> > > > > > > > > > >> ________________________________________
> > > > > > > > > > >> From: Aditya Auradkar
> > > > > > > > > > >> Sent: Friday, April 03, 2015 10:10 AM
> > > > > > > > > > >> To: dev@kafka.apache.org
> > > > > > > > > > >> Subject: RE: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks Jun.
> > > > > > > > > > >>
> > > > > > > > > > >> Some thoughts:
> > > > > > > > > > >>
> > > > > > > > > > >> 10) I think it is better we throttle regardless of the
> > > > > > > produce/fetch
> > > > > > > > > > >> version. This is a nice feature where clients can tell
> > if
> > > > > they
> > > > > > are
> > > > > > > > > being
> > > > > > > > > > >> throttled or not. If we only throttle newer clients,
> > then
> > > we
> > > > > > have
> > > > > > > > > > >> inconsistent behavior across clients in a multi-tenant
> > > > > cluster.
> > > > > > > > Having
> > > > > > > > > > >> quota metrics on the client side is also a nice
> > incentive
> > > to
> > > > > > > upgrade
> > > > > > > > > > client
> > > > > > > > > > >> versions.
> > > > > > > > > > >>
> > > > > > > > > > >> 11) I think we can call metric.record(fetchSize)
> before
> > > > > adding
> > > > > > the
> > > > > > > > > > >> delayedFetch request into the purgatory. This will
> give
> > us
> > > > > the
> > > > > > > > > estimated
> > > > > > > > > > >> delay of the request up-front. The timeout on the
> > > > > DelayedFetch
> > > > > > is
> > > > > > > > the
> > > > > > > > > > >> Max(maxWait, quotaDelay). The DelayedFetch completion
> > > > > criteria
> > > > > > can
> > > > > > > > > > change a
> > > > > > > > > > >> little to accomodate quotas.
> > > > > > > > > > >>
> > > > > > > > > > >> - I agree the quota code should return the estimated
> > delay
> > > > > time
> > > > > > in
> > > > > > > > > > >> QuotaViolationException.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Aditya
> > > > > > > > > > >>
> > > > > > > > > > >> ________________________________________
> > > > > > > > > > >> From: Jun Rao [j...@confluent.io]
> > > > > > > > > > >> Sent: Friday, April 03, 2015 9:16 AM
> > > > > > > > > > >> To: dev@kafka.apache.org
> > > > > > > > > > >> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks for the update.
> > > > > > > > > > >>
> > > > > > > > > > >> 10. About whether to return a new field in the
> response
> > to
> > > > > > > indicate
> > > > > > > > > > >> throttling. Earlier, the plan was to not change the
> > > response
> > > > > > > format
> > > > > > > > > and
> > > > > > > > > > >> just have a metric on the broker to indicate whether a
> > > > > clientId
> > > > > > is
> > > > > > > > > > >> throttled or not. The issue is that we don't know
> > whether
> > > a
> > > > > > > > particular
> > > > > > > > > > >> clientId instance is throttled or not (since there
> could
> > > be
> > > > > > > multiple
> > > > > > > > > > >> clients with the same clientId). Your proposal of
> adding
> > > an
> > > > > > > > > isThrottled
> > > > > > > > > > >> field in the response addresses and seems better.
> Then,
> > > do we
> > > > > > just
> > > > > > > > > > throttle
> > > > > > > > > > >> the new version of produce/fetch request or both the
> old
> > > and
> > > > > the
> > > > > > > new
> > > > > > > > > > >> versions? Also, we probably still need a separate
> metric
> > > on
> > > > > the
> > > > > > > > broker
> > > > > > > > > > side
> > > > > > > > > > >> to indicate whether a clientId is throttled or not.
> > > > > > > > > > >>
> > > > > > > > > > >> 11. Just to clarify. For fetch requests, when will
> > > > > > > > > > metric.record(fetchSize)
> > > > > > > > > > >> be called? Is it when we are ready to send the fetch
> > > response
> > > > > > > (after
> > > > > > > > > > >> minBytes and maxWait are satisfied)?
> > > > > > > > > > >>
> > > > > > > > > > >> As an implementation detail, it may be useful for the
> > > quota
> > > > > code
> > > > > > > to
> > > > > > > > > > return
> > > > > > > > > > >> an estimated delay time (to bring the measurement
> within
> > > the
> > > > > > > limit)
> > > > > > > > in
> > > > > > > > > > >> QuotaViolationException.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >>
> > > > > > > > > > >> Jun
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Apr 1, 2015 at 3:27 PM, Aditya Auradkar <
> > > > > > > > > > >> aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hey everyone,
> > > > > > > > > > >> >
> > > > > > > > > > >> > I've made changes to the KIP to capture our
> > discussions
> > > > > over
> > > > > > the
> > > > > > > > > last
> > > > > > > > > > >> > couple of weeks.
> > > > > > > > > > >> >
> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > > >> >
> > > > > > > > > > >> > I'll start a voting thread after people have had a
> > > chance
> > > > > to
> > > > > > > > > > >> read/comment.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > Aditya
> > > > > > > > > > >> >
> > > > > > > > > > >> > ________________________________________
> > > > > > > > > > >> > From: Steven Wu [stevenz...@gmail.com]
> > > > > > > > > > >> > Sent: Friday, March 20, 2015 9:14 AM
> > > > > > > > > > >> > To: dev@kafka.apache.org
> > > > > > > > > > >> > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > > >> >
> > > > > > > > > > >> > +1 on Jun's suggestion of maintaining one set/style
> of
> > > > > metrics
> > > > > > > at
> > > > > > > > > > broker.
> > > > > > > > > > >> > In Netflix, we have to convert the yammer metrics to
> > > servo
> > > > > > > metrics
> > > > > > > > > at
> > > > > > > > > > >> > broker. it will be painful to know some metrics are
> > in a
> > > > > > > different
> > > > > > > > > > style
> > > > > > > > > > >> > and get to be handled differently.
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Fri, Mar 20, 2015 at 8:17 AM, Jun Rao <
> > > j...@confluent.io>
> > > > >
> > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Not so sure. People who use quota will definitely
> > > want to
> > > > > > > > monitor
> > > > > > > > > > the
> > > > > > > > > > >> new
> > > > > > > > > > >> > > metrics at the client id level. Then they will
> need
> > to
> > > > > deal
> > > > > > > with
> > > > > > > > > > those
> > > > > > > > > > >> > > metrics differently from the rest of the metrics.
> It
> > > > > would
> > > > > > be
> > > > > > > > > > better if
> > > > > > > > > > >> > we
> > > > > > > > > > >> > > can hide this complexity from the users.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Jun
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Thu, Mar 19, 2015 at 10:45 PM, Joel Koshy <
> > > > > > > > jjkosh...@gmail.com
> > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Actually thinking again - since these will be a
> > few
> > > new
> > > > > > > > metrics
> > > > > > > > > at
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > client id level (bytes in and bytes out to start
> > > with)
> > > > > > maybe
> > > > > > > > it
> > > > > > > > > is
> > > > > > > > > > >> fine
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > have the two type of metrics coexist and we can
> > > migrate
> > > > > > the
> > > > > > > > > > existing
> > > > > > > > > > >> > > > metrics in parallel.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Thursday, March 19, 2015, Joel Koshy <
> > > > > > > jjkosh...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > That is a valid concern but in that case I
> think
> > > it
> > > > > > would
> > > > > > > be
> > > > > > > > > > better
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > > just migrate completely to the new metrics
> > package
> > > > > > first.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > On Thursday, March 19, 2015, Jun Rao <
> > > > > j...@confluent.io
> > > > > > > > > > >> > > > > <javascript:_e(%7B%7D,'cvml','
> j...@confluent.io
> > > ');>>
> > > > > > > wrote:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >> Hmm, I was thinking a bit differently on the
> > > metrics
> > > > > > > > stuff. I
> > > > > > > > > > >> think
> > > > > > > > > > >> > it
> > > > > > > > > > >> > > > >> would be confusing to have some metrics
> defined
> > > in
> > > > > the
> > > > > > > new
> > > > > > > > > > metrics
> > > > > > > > > > >> > > > package
> > > > > > > > > > >> > > > >> while some others defined in Coda Hale. Those
> > > > > metrics
> > > > > > > will
> > > > > > > > > look
> > > > > > > > > > >> > > > different
> > > > > > > > > > >> > > > >> (e.g., rates in Coda Hale will have special
> > > > > attributes
> > > > > > > such
> > > > > > > > > as
> > > > > > > > > > >> > > > >> 1-min-average). People may need different
> ways
> > to
> > > > > > export
> > > > > > > > the
> > > > > > > > > > >> metrics
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > >> external systems such as Graphite. So,
> instead
> > of
> > > > > using
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > >> > > metrics
> > > > > > > > > > >> > > > >> package on the broker, I was thinking that we
> > can
> > > > > just
> > > > > > > > > > implement a
> > > > > > > > > > >> > > > >> QuotaMetrics that wraps the Coda Hale
> metrics.
> > > The
> > > > > > > > > > implementation
> > > > > > > > > > >> > can
> > > > > > > > > > >> > > be
> > > > > > > > > > >> > > > >> the same as what's in the new metrics
> package.
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Thanks,
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Jun
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> On Thu, Mar 19, 2015 at 8:09 PM, Jay Kreps <
> > > > > > > > > > jay.kr...@gmail.com>
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> > Yeah I was saying was that we are blocked
> on
> > > > > picking
> > > > > > an
> > > > > > > > > > approach
> > > > > > > > > > >> > for
> > > > > > > > > > >> > > > >> > metrics but not necessarily the full
> > > conversion.
> > > > > > > Clearly
> > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > >> > pick
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > >> new
> > > > > > > > > > >> > > > >> > metrics package we would need to implement
> > the
> > > two
> > > > > > > > metrics
> > > > > > > > > we
> > > > > > > > > > >> want
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > >> quota
> > > > > > > > > > >> > > > >> > on. But the conversion of the remaining
> > metrics
> > > > > can
> > > > > > be
> > > > > > > > done
> > > > > > > > > > >> > > > >> asynchronously.
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > -Jay
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > On Thu, Mar 19, 2015 at 5:56 PM, Joel
> Koshy <
> > > > > > > > > > >> jjkosh...@gmail.com>
> > > > > > > > > > >> > > > >> wrote:
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > > > in KAFKA-1930). I agree that this KIP
> > > doesn't
> > > > > > need
> > > > > > > to
> > > > > > > > > > block
> > > > > > > > > > >> on
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> > > > migration of the metrics package.
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > Can you clarify the above? i.e., if we
> are
> > > going
> > > > > to
> > > > > > > > quota
> > > > > > > > > > on
> > > > > > > > > > >> > > > something
> > > > > > > > > > >> > > > >> > > then we would want to have migrated that
> > > metric
> > > > > > over
> > > > > > > > > > right? Or
> > > > > > > > > > >> > do
> > > > > > > > > > >> > > > you
> > > > > > > > > > >> > > > >> > > mean we don't need to complete the
> > migration
> > > of
> > > > > all
> > > > > > > > > > metrics to
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> > > metrics package right?
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > I think most of us now feel that the
> delay
> > +
> > > no
> > > > > > error
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > >> good
> > > > > > > > > > >> > > > >> > > approach, but it would be good to make
> sure
> > > > > > everyone
> > > > > > > is
> > > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > >> > > same
> > > > > > > > > > >> > > > >> > > page.
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > As Aditya requested a couple of days ago
> I
> > > think
> > > > > we
> > > > > > > > > should
> > > > > > > > > > go
> > > > > > > > > > >> > over
> > > > > > > > > > >> > > > >> > > this at the next KIP hangout.
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > Joel
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > On Thu, Mar 19, 2015 at 09:24:09AM -0700,
> > Jun
> > > > > Rao
> > > > > > > > wrote:
> > > > > > > > > > >> > > > >> > > > 1. Delay + no error seems reasonable to
> > me.
> > > > > > > However,
> > > > > > > > I
> > > > > > > > > do
> > > > > > > > > > >> feel
> > > > > > > > > > >> > > > that
> > > > > > > > > > >> > > > >> we
> > > > > > > > > > >> > > > >> > > need
> > > > > > > > > > >> > > > >> > > > to give the client an indicator that
> it's
> > > > > being
> > > > > > > > > > throttled,
> > > > > > > > > > >> > > instead
> > > > > > > > > > >> > > > >> of
> > > > > > > > > > >> > > > >> > > doing
> > > > > > > > > > >> > > > >> > > > this silently. For that, we probably
> need
> > > to
> > > > > > evolve
> > > > > > > > the
> > > > > > > > > > >> > > > >> produce/fetch
> > > > > > > > > > >> > > > >> > > > protocol to include an extra status
> field
> > > in
> > > > > the
> > > > > > > > > > response.
> > > > > > > > > > >> We
> > > > > > > > > > >> > > > >> probably
> > > > > > > > > > >> > > > >> > > need
> > > > > > > > > > >> > > > >> > > > to think more about whether we just
> want
> > to
> > > > > > return
> > > > > > > a
> > > > > > > > > > simple
> > > > > > > > > > >> > > status
> > > > > > > > > > >> > > > >> code
> > > > > > > > > > >> > > > >> > > > (e.g., 1 = throttled) or a value that
> > > > > indicates
> > > > > > how
> > > > > > > > > much
> > > > > > > > > > is
> > > > > > > > > > >> > > being
> > > > > > > > > > >> > > > >> > > throttled.
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > 2. We probably need to improve the
> > > histogram
> > > > > > > support
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >> > new
> > > > > > > > > > >> > > > >> metrics
> > > > > > > > > > >> > > > >> > > > package before we can use it more
> widely
> > on
> > > > > the
> > > > > > > > server
> > > > > > > > > > side
> > > > > > > > > > >> > > (left
> > > > > > > > > > >> > > > a
> > > > > > > > > > >> > > > >> > > comment
> > > > > > > > > > >> > > > >> > > > in KAFKA-1930). I agree that this KIP
> > > doesn't
> > > > > > need
> > > > > > > to
> > > > > > > > > > block
> > > > > > > > > > >> on
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> > > > migration of the metrics package.
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > Jun
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > On Wed, Mar 18, 2015 at 4:02 PM, Aditya
> > > > > Auradkar
> > > > > > <
> > > > > > > > > > >> > > > >> > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > > Hey everyone,
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Thanks for the great discussion.
> There
> > > are
> > > > > > > > currently
> > > > > > > > > a
> > > > > > > > > > few
> > > > > > > > > > >> > > > points
> > > > > > > > > > >> > > > >> on
> > > > > > > > > > >> > > > >> > > this
> > > > > > > > > > >> > > > >> > > > > KIP that need addressing and I want
> to
> > > make
> > > > > > sure
> > > > > > > we
> > > > > > > > > > are on
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> same
> > > > > > > > > > >> > > > >> > > page
> > > > > > > > > > >> > > > >> > > > > about those.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > 1. Append and delay response vs delay
> > and
> > > > > > return
> > > > > > > > > error
> > > > > > > > > > >> > > > >> > > > > - I think we've discussed the pros
> and
> > > cons
> > > > > of
> > > > > > > each
> > > > > > > > > > >> approach
> > > > > > > > > > >> > > but
> > > > > > > > > > >> > > > >> > > haven't
> > > > > > > > > > >> > > > >> > > > > chosen an approach yet. Where does
> > > everyone
> > > > > > stand
> > > > > > > > on
> > > > > > > > > > this
> > > > > > > > > > >> > > issue?
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > 2. Metrics Migration and usage in
> > quotas
> > > > > > > > > > >> > > > >> > > > > - The metrics library in clients has
> a
> > > > > notion
> > > > > > of
> > > > > > > > > quotas
> > > > > > > > > > >> that
> > > > > > > > > > >> > > we
> > > > > > > > > > >> > > > >> > should
> > > > > > > > > > >> > > > >> > > > > reuse. For that to happen, we need to
> > > > > migrate
> > > > > > the
> > > > > > > > > > server
> > > > > > > > > > >> to
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> new
> > > > > > > > > > >> > > > >> > > metrics
> > > > > > > > > > >> > > > >> > > > > package.
> > > > > > > > > > >> > > > >> > > > > - Need more clarification on how to
> > > compute
> > > > > > > > > throttling
> > > > > > > > > > >> time
> > > > > > > > > > >> > > and
> > > > > > > > > > >> > > > >> > > windowing
> > > > > > > > > > >> > > > >> > > > > for quotas.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > I'm going to start a new KIP to
> discuss
> > > > > metrics
> > > > > > > > > > migration
> > > > > > > > > > >> > > > >> separately.
> > > > > > > > > > >> > > > >> > > That
> > > > > > > > > > >> > > > >> > > > > will also contain a section on
> quotas.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > 3. Dynamic Configuration management -
> > > Being
> > > > > > > > discussed
> > > > > > > > > > in
> > > > > > > > > > >> > > KIP-5.
> > > > > > > > > > >> > > > >> > > Basically
> > > > > > > > > > >> > > > >> > > > > we need something that will model
> > default
> > > > > > quotas
> > > > > > > > and
> > > > > > > > > > allow
> > > > > > > > > > >> > > > >> per-client
> > > > > > > > > > >> > > > >> > > > > overrides.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Is there something else that I'm
> > missing?
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Thanks,
> > > > > > > > > > >> > > > >> > > > > Aditya
> > > > > > > > > > >> > > > >> > > > >
> > ________________________________________
> > > > > > > > > > >> > > > >> > > > > From: Jay Kreps [jay.kr...@gmail.com
> ]
> > > > > > > > > > >> > > > >> > > > > Sent: Wednesday, March 18, 2015 2:10
> PM
> > > > > > > > > > >> > > > >> > > > > To: dev@kafka.apache.org
> > > > > > > > > > >> > > > >> > > > > Subject: Re: [KIP-DISCUSSION] KIP-13
> > > Quotas
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Hey Steven,
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > The current proposal is actually to
> > > enforce
> > > > > > > quotas
> > > > > > > > at
> > > > > > > > > > the
> > > > > > > > > > >> > > > >> > > > > client/application level, NOT the
> topic
> > > > > level.
> > > > > > So
> > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > >> > have
> > > > > > > > > > >> > > a
> > > > > > > > > > >> > > > >> > service
> > > > > > > > > > >> > > > >> > > > > with a few dozen instances the quota
> is
> > > > > against
> > > > > > > all
> > > > > > > > > of
> > > > > > > > > > >> those
> > > > > > > > > > >> > > > >> > instances
> > > > > > > > > > >> > > > >> > > > > added up across all their topics. So
> > > > > actually
> > > > > > the
> > > > > > > > > > effect
> > > > > > > > > > >> > would
> > > > > > > > > > >> > > > be
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > same
> > > > > > > > > > >> > > > >> > > > > either way but throttling gives the
> > > producer
> > > > > > the
> > > > > > > > > > choice of
> > > > > > > > > > >> > > > either
> > > > > > > > > > >> > > > >> > > blocking
> > > > > > > > > > >> > > > >> > > > > or dropping.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > -Jay
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > On Tue, Mar 17, 2015 at 10:08 AM,
> > Steven
> > > Wu
> > > > > <
> > > > > > > > > > >> > > > stevenz...@gmail.com
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > > wrote:
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > > Jay,
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > let's say an app produces to 10
> > > different
> > > > > > > topics.
> > > > > > > > > > one of
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> topic
> > > > > > > > > > >> > > > >> > is
> > > > > > > > > > >> > > > >> > > > > sent
> > > > > > > > > > >> > > > >> > > > > > from a library. due to whatever
> > > > > > condition/bug,
> > > > > > > > this
> > > > > > > > > > lib
> > > > > > > > > > >> > > starts
> > > > > > > > > > >> > > > >> to
> > > > > > > > > > >> > > > >> > > send
> > > > > > > > > > >> > > > >> > > > > > messages over the quota. if we go
> > with
> > > the
> > > > > > > > delayed
> > > > > > > > > > >> > response
> > > > > > > > > > >> > > > >> > > approach, it
> > > > > > > > > > >> > > > >> > > > > > will cause the whole shared
> > > > > RecordAccumulator
> > > > > > > > > buffer
> > > > > > > > > > to
> > > > > > > > > > >> be
> > > > > > > > > > >> > > > >> filled
> > > > > > > > > > >> > > > >> > up.
> > > > > > > > > > >> > > > >> > > > > that
> > > > > > > > > > >> > > > >> > > > > > will penalize other 9 topics who
> are
> > > > > within
> > > > > > the
> > > > > > > > > > quota.
> > > > > > > > > > >> > that
> > > > > > > > > > >> > > is
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > > > > unfairness point that Ewen and I
> were
> > > > > trying
> > > > > > to
> > > > > > > > > make.
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > if broker just drop the msg and
> > return
> > > an
> > > > > > > > > > error/status
> > > > > > > > > > >> > code
> > > > > > > > > > >> > > > >> > > indicates the
> > > > > > > > > > >> > > > >> > > > > > drop and why. then producer can
> just
> > > move
> > > > > on
> > > > > > > and
> > > > > > > > > > accept
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> drop.
> > > > > > > > > > >> > > > >> > > shared
> > > > > > > > > > >> > > > >> > > > > > buffer won't be saturated and
> other 9
> > > > > topics
> > > > > > > > won't
> > > > > > > > > be
> > > > > > > > > > >> > > > penalized.
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > > >> > > > >> > > > > > Steven
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > On Tue, Mar 17, 2015 at 9:44 AM,
> Jay
> > > Kreps
> > > > > <
> > > > > > > > > > >> > > > jay.kr...@gmail.com
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > > wrote:
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > > Hey Steven,
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > It is true that hitting the quota
> > > will
> > > > > > cause
> > > > > > > > > > >> > back-pressure
> > > > > > > > > > >> > > > on
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > > > > producer.
> > > > > > > > > > >> > > > >> > > > > > > But the solution is simple, a
> > > producer
> > > > > that
> > > > > > > > wants
> > > > > > > > > > to
> > > > > > > > > > >> > avoid
> > > > > > > > > > >> > > > >> this
> > > > > > > > > > >> > > > >> > > should
> > > > > > > > > > >> > > > >> > > > > > stay
> > > > > > > > > > >> > > > >> > > > > > > under its quota. In other words
> > this
> > > is
> > > > > a
> > > > > > > > > contract
> > > > > > > > > > >> > between
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > >> > > cluster
> > > > > > > > > > >> > > > >> > > > > > and
> > > > > > > > > > >> > > > >> > > > > > > the client, with each side having
> > > > > something
> > > > > > > to
> > > > > > > > > > uphold.
> > > > > > > > > > >> > > Quite
> > > > > > > > > > >> > > > >> > > possibly
> > > > > > > > > > >> > > > >> > > > > the
> > > > > > > > > > >> > > > >> > > > > > > same thing will happen in the
> > > absence of
> > > > > a
> > > > > > > > > quota, a
> > > > > > > > > > >> > client
> > > > > > > > > > >> > > > >> that
> > > > > > > > > > >> > > > >> > > > > produces
> > > > > > > > > > >> > > > >> > > > > > an
> > > > > > > > > > >> > > > >> > > > > > > unexpected amount of load will
> hit
> > > the
> > > > > > limits
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> > > server
> > > > > > > > > > >> > > > >> and
> > > > > > > > > > >> > > > >> > > > > > experience
> > > > > > > > > > >> > > > >> > > > > > > backpressure. Quotas just allow
> you
> > > to
> > > > > set
> > > > > > > that
> > > > > > > > > > same
> > > > > > > > > > >> > limit
> > > > > > > > > > >> > > > at
> > > > > > > > > > >> > > > >> > > something
> > > > > > > > > > >> > > > >> > > > > > > lower than 100% of all resources
> on
> > > the
> > > > > > > server,
> > > > > > > > > > which
> > > > > > > > > > >> is
> > > > > > > > > > >> > > > >> useful
> > > > > > > > > > >> > > > >> > > for a
> > > > > > > > > > >> > > > >> > > > > > > shared cluster.
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > -Jay
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > On Mon, Mar 16, 2015 at 11:34 PM,
> > > Steven
> > > > > > Wu <
> > > > > > > > > > >> > > > >> > stevenz...@gmail.com>
> > > > > > > > > > >> > > > >> > > > > > wrote:
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > wait. we create one kafka
> > producer
> > > for
> > > > > > each
> > > > > > > > > > cluster.
> > > > > > > > > > >> > > each
> > > > > > > > > > >> > > > >> > > cluster can
> > > > > > > > > > >> > > > >> > > > > > > have
> > > > > > > > > > >> > > > >> > > > > > > > many topics. if producer buffer
> > got
> > > > > > filled
> > > > > > > up
> > > > > > > > > > due to
> > > > > > > > > > >> > > > delayed
> > > > > > > > > > >> > > > >> > > response
> > > > > > > > > > >> > > > >> > > > > > for
> > > > > > > > > > >> > > > >> > > > > > > > one throttled topic, won't that
> > > > > penalize
> > > > > > > > other
> > > > > > > > > > >> topics
> > > > > > > > > > >> > > > >> unfairly?
> > > > > > > > > > >> > > > >> > > it
> > > > > > > > > > >> > > > >> > > > > > seems
> > > > > > > > > > >> > > > >> > > > > > > to
> > > > > > > > > > >> > > > >> > > > > > > > me that broker should just
> return
> > > > > error
> > > > > > > > without
> > > > > > > > > > >> delay.
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > sorry that I am chatting to
> > myself
> > > :)
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > On Mon, Mar 16, 2015 at 11:29
> PM,
> > > > > Steven
> > > > > > > Wu <
> > > > > > > > > > >> > > > >> > > stevenz...@gmail.com>
> > > > > > > > > > >> > > > >> > > > > > > wrote:
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > > I think I can answer my own
> > > > > question.
> > > > > > > > delayed
> > > > > > > > > > >> > response
> > > > > > > > > > >> > > > >> will
> > > > > > > > > > >> > > > >> > > cause
> > > > > > > > > > >> > > > >> > > > > the
> > > > > > > > > > >> > > > >> > > > > > > > > producer buffer to be full,
> > which
> > > > > then
> > > > > > > > result
> > > > > > > > > > in
> > > > > > > > > > >> > > either
> > > > > > > > > > >> > > > >> > thread
> > > > > > > > > > >> > > > >> > > > > > blocking
> > > > > > > > > > >> > > > >> > > > > > > > or
> > > > > > > > > > >> > > > >> > > > > > > > > message drop.
> > > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > > On Mon, Mar 16, 2015 at 11:24
> > PM,
> > > > > > Steven
> > > > > > > > Wu <
> > > > > > > > > > >> > > > >> > > stevenz...@gmail.com>
> > > > > > > > > > >> > > > >> > > > > > > > wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > >> please correct me if I am
> > > missing
> > > > > sth
> > > > > > > > here.
> > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > >> > not
> > > > > > > > > > >> > > > >> > > understanding
> > > > > > > > > > >> > > > >> > > > > > how
> > > > > > > > > > >> > > > >> > > > > > > > >> would throttle work without
> > > > > > > > > > cooperation/back-off
> > > > > > > > > > >> > from
> > > > > > > > > > >> > > > >> > > producer.
> > > > > > > > > > >> > > > >> > > > > new
> > > > > > > > > > >> > > > >> > > > > > > Java
> > > > > > > > > > >> > > > >> > > > > > > > >> producer supports
> non-blocking
> > > API.
> > > > > > why
> > > > > > > > > would
> > > > > > > > > > >> > delayed
> > > > > > > > > > >> > > > >> > > response be
> > > > > > > > > > >> > > > >> > > > > > able
> > > > > > > > > > >> > > > >> > > > > > > > to
> > > > > > > > > > >> > > > >> > > > > > > > >> slow down producer? producer
> > > will
> > > > > > > continue
> > > > > > > > > to
> > > > > > > > > > >> fire
> > > > > > > > > > >> > > > async
> > > > > > > > > > >> > > > >> > > sends.
> > > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >> On Mon, Mar 16, 2015 at
> 10:58
> > > PM,
> > > > > > > Guozhang
> > > > > > > > > > Wang <
> > > > > > > > > > >> > > > >> > > > > wangg...@gmail.com
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > >> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> I think we are really
> > > discussing
> > > > > two
> > > > > > > > > separate
> > > > > > > > > > >> > issues
> > > > > > > > > > >> > > > >> here:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> 1. Whether we should a)
> > > > > > > > > > >> > > > >> > > > >
> > > append-then-block-then-returnOKButThrottled
> > > > > > > > > > >> > > > >> > > > > > > or
> > > > > > > > > > >> > > > >> > > > > > > > b)
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > block-then-returnFailDuetoThrottled
> > > > > > for
> > > > > > > > > quota
> > > > > > > > > > >> > > actions
> > > > > > > > > > >> > > > on
> > > > > > > > > > >> > > > >> > > produce
> > > > > > > > > > >> > > > >> > > > > > > > >>> requests.
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> Both these approaches
> assume
> > > some
> > > > > > kind
> > > > > > > of
> > > > > > > > > > >> > > > >> well-behaveness
> > > > > > > > > > >> > > > >> > of
> > > > > > > > > > >> > > > >> > > the
> > > > > > > > > > >> > > > >> > > > > > > > clients:
> > > > > > > > > > >> > > > >> > > > > > > > >>> option a) assumes the
> client
> > > sets
> > > > > an
> > > > > > > > proper
> > > > > > > > > > >> > timeout
> > > > > > > > > > >> > > > >> value
> > > > > > > > > > >> > > > >> > > while
> > > > > > > > > > >> > > > >> > > > > can
> > > > > > > > > > >> > > > >> > > > > > > > just
> > > > > > > > > > >> > > > >> > > > > > > > >>> ignore "OKButThrottled"
> > > response,
> > > > > > while
> > > > > > > > > > option
> > > > > > > > > > >> b)
> > > > > > > > > > >> > > > >> assumes
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > > > > client
> > > > > > > > > > >> > > > >> > > > > > > > >>> handles the
> > > "FailDuetoThrottled"
> > > > > > > > > > appropriately.
> > > > > > > > > > >> > For
> > > > > > > > > > >> > > > any
> > > > > > > > > > >> > > > >> > > malicious
> > > > > > > > > > >> > > > >> > > > > > > > clients
> > > > > > > > > > >> > > > >> > > > > > > > >>> that, for example, just
> keep
> > > > > retrying
> > > > > > > > > either
> > > > > > > > > > >> > > > >> intentionally
> > > > > > > > > > >> > > > >> > or
> > > > > > > > > > >> > > > >> > > > > not,
> > > > > > > > > > >> > > > >> > > > > > > > >>> neither
> > > > > > > > > > >> > > > >> > > > > > > > >>> of these approaches are
> > > actually
> > > > > > > > effective.
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> 2. For "OKButThrottled" and
> > > > > > > > > > "FailDuetoThrottled"
> > > > > > > > > > >> > > > >> responses,
> > > > > > > > > > >> > > > >> > > shall
> > > > > > > > > > >> > > > >> > > > > > we
> > > > > > > > > > >> > > > >> > > > > > > > >>> encode
> > > > > > > > > > >> > > > >> > > > > > > > >>> them as error codes or
> > augment
> > > the
> > > > > > > > protocol
> > > > > > > > > > to
> > > > > > > > > > >> > use a
> > > > > > > > > > >> > > > >> > separate
> > > > > > > > > > >> > > > >> > > > > field
> > > > > > > > > > >> > > > >> > > > > > > > >>> indicating "status codes".
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> Today we have already
> > > incorporated
> > > > > > some
> > > > > > > > > > status
> > > > > > > > > > >> > code
> > > > > > > > > > >> > > as
> > > > > > > > > > >> > > > >> > error
> > > > > > > > > > >> > > > >> > > > > codes
> > > > > > > > > > >> > > > >> > > > > > in
> > > > > > > > > > >> > > > >> > > > > > > > the
> > > > > > > > > > >> > > > >> > > > > > > > >>> responses, e.g.
> > > > > ReplicaNotAvailable
> > > > > > in
> > > > > > > > > > >> > > > MetadataResponse,
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > pros
> > > > > > > > > > >> > > > >> > > > > > of
> > > > > > > > > > >> > > > >> > > > > > > > this
> > > > > > > > > > >> > > > >> > > > > > > > >>> is of course using a single
> > > field
> > > > > for
> > > > > > > > > > response
> > > > > > > > > > >> > > status
> > > > > > > > > > >> > > > >> like
> > > > > > > > > > >> > > > >> > > the
> > > > > > > > > > >> > > > >> > > > > HTTP
> > > > > > > > > > >> > > > >> > > > > > > > >>> status
> > > > > > > > > > >> > > > >> > > > > > > > >>> codes, while the cons is
> that
> > > it
> > > > > > > requires
> > > > > > > > > > >> clients
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > >> handle
> > > > > > > > > > >> > > > >> > > the
> > > > > > > > > > >> > > > >> > > > > > error
> > > > > > > > > > >> > > > >> > > > > > > > >>> codes
> > > > > > > > > > >> > > > >> > > > > > > > >>> carefully.
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> I think maybe we can
> actually
> > > > > extend
> > > > > > > the
> > > > > > > > > > >> > single-code
> > > > > > > > > > >> > > > >> > > approach to
> > > > > > > > > > >> > > > >> > > > > > > > overcome
> > > > > > > > > > >> > > > >> > > > > > > > >>> its drawbacks, that is,
> wrap
> > > the
> > > > > > error
> > > > > > > > > codes
> > > > > > > > > > >> > > semantics
> > > > > > > > > > >> > > > >> to
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > > > users
> > > > > > > > > > >> > > > >> > > > > > > so
> > > > > > > > > > >> > > > >> > > > > > > > >>> that
> > > > > > > > > > >> > > > >> > > > > > > > >>> users do not need to handle
> > the
> > > > > codes
> > > > > > > > > > >> one-by-one.
> > > > > > > > > > >> > > More
> > > > > > > > > > >> > > > >> > > > > concretely,
> > > > > > > > > > >> > > > >> > > > > > > > >>> following Jay's example the
> > > client
> > > > > > > could
> > > > > > > > > > write
> > > > > > > > > > >> > sth.
> > > > > > > > > > >> > > > like
> > > > > > > > > > >> > > > >> > > this:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> if(error.isOK())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // status code is good or
> the
> > > > > > code
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > >> > > simply
> > > > > > > > > > >> > > > >> > > ignored for
> > > > > > > > > > >> > > > >> > > > > > > this
> > > > > > > > > > >> > > > >> > > > > > > > >>> request type, process the
> > > request
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.needsRetry())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // throttled, transient
> > error,
> > > > > > > etc:
> > > > > > > > > > retry
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.isFatal())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // non-retriable errors,
> etc:
> > > > > > > > notify /
> > > > > > > > > > >> > > terminate
> > > > > > > > > > >> > > > /
> > > > > > > > > > >> > > > >> > other
> > > > > > > > > > >> > > > >> > > > > > > handling
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> Only when the clients
> really
> > > want
> > > > > to
> > > > > > > > > handle,
> > > > > > > > > > for
> > > > > > > > > > >> > > > example
> > > > > > > > > > >> > > > >> > > > > > > > >>> FailDuetoThrottled
> > > > > > > > > > >> > > > >> > > > > > > > >>> status code specifically,
> it
> > > needs
> > > > > > to:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> if(error.isOK())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // status code is good or
> the
> > > > > > code
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > >> > > simply
> > > > > > > > > > >> > > > >> > > ignored for
> > > > > > > > > > >> > > > >> > > > > > > this
> > > > > > > > > > >> > > > >> > > > > > > > >>> request type, process the
> > > request
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error ==
> > > > > > FailDuetoThrottled )
> > > > > > > > > > >> > > > >> > > > > > > > >>> // throttled: log it
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.needsRetry())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // transient error, etc:
> > retry
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.isFatal())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // non-retriable errors,
> etc:
> > > > > > > > notify /
> > > > > > > > > > >> > > terminate
> > > > > > > > > > >> > > > /
> > > > > > > > > > >> > > > >> > other
> > > > > > > > > > >> > > > >> > > > > > > handling
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> And for implementation we
> can
> > > > > > probably
> > > > > > > > > group
> > > > > > > > > > the
> > > > > > > > > > >> > > codes
> > > > > > > > > > >> > > > >> > > > > accordingly
> > > > > > > > > > >> > > > >> > > > > > > like
> > > > > > > > > > >> > > > >> > > > > > > > >>> HTTP status code such that
> we
> > > can
> > > > > do:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> boolean Error.isOK() {
> > > > > > > > > > >> > > > >> > > > > > > > >>> return code < 300 && code
> >=
> > > 200;
> > > > > > > > > > >> > > > >> > > > > > > > >>> }
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> Guozhang
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> On Mon, Mar 16, 2015 at
> 10:24
> > > PM,
> > > > > > Ewen
> > > > > > > > > > >> > > > Cheslack-Postava
> > > > > > > > > > >> > > > >> <
> > > > > > > > > > >> > > > >> > > > > > > > >>> e...@confluent.io>
> > > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Agreed that trying to
> > > shoehorn
> > > > > > > > non-error
> > > > > > > > > > codes
> > > > > > > > > > >> > > into
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > error
> > > > > > > > > > >> > > > >> > > > > > field
> > > > > > > > > > >> > > > >> > > > > > > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> a
> > > > > > > > > > >> > > > >> > > > > > > > >>> > bad idea. It makes it
> *way*
> > > too
> > > > > > easy
> > > > > > > to
> > > > > > > > > > write
> > > > > > > > > > >> > code
> > > > > > > > > > >> > > > >> that
> > > > > > > > > > >> > > > >> > > looks
> > > > > > > > > > >> > > > >> > > > > > (and
> > > > > > > > > > >> > > > >> > > > > > > > >>> should
> > > > > > > > > > >> > > > >> > > > > > > > >>> > be) correct but is
> actually
> > > > > > > incorrect.
> > > > > > > > If
> > > > > > > > > > >> > > > necessary, I
> > > > > > > > > > >> > > > >> > > think
> > > > > > > > > > >> > > > >> > > > > it's
> > > > > > > > > > >> > > > >> > > > > > > > much
> > > > > > > > > > >> > > > >> > > > > > > > >>> > better to to spend a
> couple
> > > of
> > > > > > extra
> > > > > > > > > bytes
> > > > > > > > > > to
> > > > > > > > > > >> > > encode
> > > > > > > > > > >> > > > >> that
> > > > > > > > > > >> > > > >> > > > > > > information
> > > > > > > > > > >> > > > >> > > > > > > > >>> > separately (a "status" or
> > > > > "warning"
> > > > > > > > > > section of
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> > > response).
> > > > > > > > > > >> > > > >> > > > > An
> > > > > > > > > > >> > > > >> > > > > > > > >>> indication
> > > > > > > > > > >> > > > >> > > > > > > > >>> > that throttling is
> > occurring
> > > is
> > > > > > > > something
> > > > > > > > > > I'd
> > > > > > > > > > >> > > expect
> > > > > > > > > > >> > > > >> to
> > > > > > > > > > >> > > > >> > be
> > > > > > > > > > >> > > > >> > > > > > > indicated
> > > > > > > > > > >> > > > >> > > > > > > > >>> by a
> > > > > > > > > > >> > > > >> > > > > > > > >>> > bit flag in the response
> > > rather
> > > > > > than
> > > > > > > as
> > > > > > > > > an
> > > > > > > > > > >> error
> > > > > > > > > > >> > > > code.
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Gwen - I think an error
> > code
> > > > > makes
> > > > > > > > sense
> > > > > > > > > > when
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> request
> > > > > > > > > > >> > > > >> > > > > > actually
> > > > > > > > > > >> > > > >> > > > > > > > >>> failed.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Option B, which Jun was
> > > > > advocating,
> > > > > > > > would
> > > > > > > > > > have
> > > > > > > > > > >> > > > >> appended
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > > > > > messages
> > > > > > > > > > >> > > > >> > > > > > > > >>> > successfully. If the
> > > > > rate-limiting
> > > > > > > case
> > > > > > > > > > you're
> > > > > > > > > > >> > > > talking
> > > > > > > > > > >> > > > >> > > about
> > > > > > > > > > >> > > > >> > > > > had
> > > > > > > > > > >> > > > >> > > > > > > > >>> > successfully committed
> the
> > > > > > messages,
> > > > > > > I
> > > > > > > > > > would
> > > > > > > > > > >> say
> > > > > > > > > > >> > > > >> that's
> > > > > > > > > > >> > > > >> > > also a
> > > > > > > > > > >> > > > >> > > > > > bad
> > > > > > > > > > >> > > > >> > > > > > > > use
> > > > > > > > > > >> > > > >> > > > > > > > >>> of
> > > > > > > > > > >> > > > >> > > > > > > > >>> > error codes.
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > On Mon, Mar 16, 2015 at
> > 10:16
> > > > > PM,
> > > > > > > Gwen
> > > > > > > > > > >> Shapira <
> > > > > > > > > > >> > > > >> > > > > > > > gshap...@cloudera.com>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > We discussed an error
> > code
> > > for
> > > > > > > > > > rate-limiting
> > > > > > > > > > >> > > > (which
> > > > > > > > > > >> > > > >> I
> > > > > > > > > > >> > > > >> > > think
> > > > > > > > > > >> > > > >> > > > > > made
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > sense), isn't it a
> > similar
> > > > > case?
> > > > > > > > > > >> > > > >> > > > > > > > >>> > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > On Mon, Mar 16, 2015 at
> > > 10:10
> > > > > PM,
> > > > > > > Jay
> > > > > > > > > > Kreps
> > > > > > > > > > >> <
> > > > > > > > > > >> > > > >> > > > > > jay.kr...@gmail.com
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > My concern is that as
> > > soon
> > > > > as
> > > > > > you
> > > > > > > > > start
> > > > > > > > > > >> > > encoding
> > > > > > > > > > >> > > > >> > > non-error
> > > > > > > > > > >> > > > >> > > > > > > > response
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > information into
> error
> > > codes
> > > > > > the
> > > > > > > > next
> > > > > > > > > > >> > question
> > > > > > > > > > >> > > > is
> > > > > > > > > > >> > > > >> > what
> > > > > > > > > > >> > > > >> > > to
> > > > > > > > > > >> > > > >> > > > > do
> > > > > > > > > > >> > > > >> > > > > > if
> > > > > > > > > > >> > > > >> > > > > > > > two
> > > > > > > > > > >> > > > >> > > > > > > > >>> > such
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > codes apply (i.e. you
> > > have a
> > > > > > > > replica
> > > > > > > > > > down
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > >> > > response
> > > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> > quota'd). I
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > think I am trying to
> > > argue
> > > > > that
> > > > > > > > error
> > > > > > > > > > >> should
> > > > > > > > > > >> > > > mean
> > > > > > > > > > >> > > > >> > "why
> > > > > > > > > > >> > > > >> > > we
> > > > > > > > > > >> > > > >> > > > > > > failed
> > > > > > > > > > >> > > > >> > > > > > > > >>> your
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > request", for which
> > there
> > > > > will
> > > > > > > > really
> > > > > > > > > > only
> > > > > > > > > > >> > be
> > > > > > > > > > >> > > > one
> > > > > > > > > > >> > > > >> > > reason,
> > > > > > > > > > >> > > > >> > > > > and
> > > > > > > > > > >> > > > >> > > > > > > any
> > > > > > > > > > >> > > > >> > > > > > > > >>> other
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > useful information we
> > > want
> > > > > to
> > > > > > > send
> > > > > > > > > > back is
> > > > > > > > > > >> > > just
> > > > > > > > > > >> > > > >> > another
> > > > > > > > > > >> > > > >> > > > > field
> > > > > > > > > > >> > > > >> > > > > > > in
> > > > > > > > > > >> > > > >> > > > > > > > >>> the
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > response.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > -Jay
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > On Mon, Mar 16, 2015
> at
> > > 9:51
> > > > > > PM,
> > > > > > > > Gwen
> > > > > > > > > > >> > Shapira
> > > > > > > > > > >> > > <
> > > > > > > > > > >> > > > >> > > > > > > > >>> gshap...@cloudera.com>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> I think its not too
> > > late to
> > > > > > > > reserve
> > > > > > > > > a
> > > > > > > > > > set
> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > >> error
> > > > > > > > > > >> > > > >> > > codes
> > > > > > > > > > >> > > > >> > > > > > > > >>> (200-299?)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> for "non-error"
> codes.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> It won't be backward
> > > > > > compatible
> > > > > > > > > (i.e.
> > > > > > > > > > >> > clients
> > > > > > > > > > >> > > > >> that
> > > > > > > > > > >> > > > >> > > > > currently
> > > > > > > > > > >> > > > >> > > > > > > do
> > > > > > > > > > >> > > > >> > > > > > > > >>> "else
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> throw" will throw on
> > > > > > > non-errors),
> > > > > > > > > but
> > > > > > > > > > >> > perhaps
> > > > > > > > > > >> > > > its
> > > > > > > > > > >> > > > >> > > > > > worthwhile.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> On Mon, Mar 16, 2015
> > at
> > > > > 9:42
> > > > > > PM,
> > > > > > > > Jay
> > > > > > > > > > >> Kreps
> > > > > > > > > > >> > <
> > > > > > > > > > >> > > > >> > > > > > > jay.kr...@gmail.com
> > > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Hey Jun,
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > I'd really really
> > > really
> > > > > > like
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > >> > > that.
> > > > > > > > > > >> > > > >> > Having
> > > > > > > > > > >> > > > >> > > just
> > > > > > > > > > >> > > > >> > > > > > > > spent a
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > bunch of
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > time on the
> clients,
> > > > > using
> > > > > > the
> > > > > > > > > error
> > > > > > > > > > >> > codes
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > >> > encode
> > > > > > > > > > >> > > > >> > > > > other
> > > > > > > > > > >> > > > >> > > > > > > > >>> > information
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > about the response
> > is
> > > > > super
> > > > > > > > > > dangerous.
> > > > > > > > > > >> > The
> > > > > > > > > > >> > > > >> error
> > > > > > > > > > >> > > > >> > > > > handling
> > > > > > > > > > >> > > > >> > > > > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> one of
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > the
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > hardest parts of
> the
> > > > > client
> > > > > > > > > > (Guozhang
> > > > > > > > > > >> > chime
> > > > > > > > > > >> > > > in
> > > > > > > > > > >> > > > >> > > here).
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Generally the
> error
> > > > > handling
> > > > > > > > looks
> > > > > > > > > > like
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > if(error == none)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // good, process
> the
> > > > > > > > request
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else if(error ==
> > > > > > > > KNOWN_ERROR_1)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // handle known
> > error
> > > 1
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else if(error ==
> > > > > > > > KNOWN_ERROR_2)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // handle known
> > error
> > > 2
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > throw
> > > > > > > > > > >> > > Errors.forCode(error).exception();
> > > > > > > > > > >> > > > >> //
> > > > > > > > > > >> > > > >> > or
> > > > > > > > > > >> > > > >> > > some
> > > > > > > > > > >> > > > >> > > > > > > other
> > > > > > > > > > >> > > > >> > > > > > > > >>> > default
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > behavior
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > This works because
> > we
> > > > > have a
> > > > > > > > > > convention
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > >> and
> > > > > > > > > > >> > > > >> > > error
> > > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> something
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > that
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > prevented your
> > getting
> > > > > the
> > > > > > > > > response
> > > > > > > > > > so
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> default
> > > > > > > > > > >> > > > >> > > > > > handling
> > > > > > > > > > >> > > > >> > > > > > > > >>> case is
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > sane
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > and forward
> > > compatible.
> > > > > It
> > > > > > is
> > > > > > > > > > tempting
> > > > > > > > > > >> to
> > > > > > > > > > >> > > use
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > error
> > > > > > > > > > >> > > > >> > > > > > code
> > > > > > > > > > >> > > > >> > > > > > > > to
> > > > > > > > > > >> > > > >> > > > > > > > >>> > convey
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > information in the
> > > > > success
> > > > > > > case.
> > > > > > > > > For
> > > > > > > > > > >> > > example
> > > > > > > > > > >> > > > we
> > > > > > > > > > >> > > > >> > > could
> > > > > > > > > > >> > > > >> > > > > use
> > > > > > > > > > >> > > > >> > > > > > > > error
> > > > > > > > > > >> > > > >> > > > > > > > >>> > codes
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > to
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > encode whether
> > quotas
> > > > > were
> > > > > > > > > enforced,
> > > > > > > > > > >> > > whether
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > request
> > > > > > > > > > >> > > > >> > > > > > was
> > > > > > > > > > >> > > > >> > > > > > > > >>> served
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > out
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> of
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > cache, whether the
> > > stock
> > > > > > > market
> > > > > > > > is
> > > > > > > > > > up
> > > > > > > > > > >> > > today,
> > > > > > > > > > >> > > > or
> > > > > > > > > > >> > > > >> > > > > whatever.
> > > > > > > > > > >> > > > >> > > > > > > The
> > > > > > > > > > >> > > > >> > > > > > > > >>> > problem
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > that since these
> are
> > > not
> > > > > > > errors
> > > > > > > > as
> > > > > > > > > > far
> > > > > > > > > > >> as
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> > > client is
> > > > > > > > > > >> > > > >> > > > > > > > >>> concerned it
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> should
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > not throw an
> > exception
> > > > > but
> > > > > > > > process
> > > > > > > > > > the
> > > > > > > > > > >> > > > >> response,
> > > > > > > > > > >> > > > >> > > but now
> > > > > > > > > > >> > > > >> > > > > > we
> > > > > > > > > > >> > > > >> > > > > > > > >>> created
> > > > > > > > > > >> > > > >> > > > > > > > >>> > an
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > explicit
> requirement
> > > that
> > > > > > that
> > > > > > > > > > error be
> > > > > > > > > > >> > > > handled
> > > > > > > > > > >> > > > >> > > > > explicitly
> > > > > > > > > > >> > > > >> > > > > > > > >>> since it
> > > > > > > > > > >> > > > >> > > > > > > > >>> > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > different. I
> really
> > > think
> > > > > > that
> > > > > > > > > this
> > > > > > > > > > >> kind
> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > >> > > information
> > > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > > >> > > > >> > > > > > > not
> > > > > > > > > > >> > > > >> > > > > > > > >>> an
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > error,
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> it
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > is just
> information,
> > > and
> > > > > if
> > > > > > we
> > > > > > > > > want
> > > > > > > > > > it
> > > > > > > > > > >> in
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> > > response
> > > > > > > > > > >> > > > >> > > > > we
> > > > > > > > > > >> > > > >> > > > > > > > >>> should do
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > the
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > right thing and
> add
> > a
> > > new
> > > > > > > field
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > >> > > > >> response.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > I think you saw
> the
> > > Samza
> > > > > > bug
> > > > > > > > that
> > > > > > > > > > was
> > > > > > > > > > >> > > > >> literally
> > > > > > > > > > >> > > > >> > an
> > > > > > > > > > >> > > > >> > > > > > example
> > > > > > > > > > >> > > > >> > > > > > > of
> > > > > > > > > > >> > > > >> > > > > > > > >>> this
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > happening and
> > leading
> > > to
> > > > > an
> > > > > > > > > infinite
> > > > > > > > > > >> > retry
> > > > > > > > > > >> > > > >> loop.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Further more I
> > really
> > > > > want
> > > > > > to
> > > > > > > > > > emphasize
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > >> > hitting
> > > > > > > > > > >> > > > >> > > > > your
> > > > > > > > > > >> > > > >> > > > > > > > quota
> > > > > > > > > > >> > > > >> > > > > > > > >>> in
> > > > > > > > > > >> > > > >> > > > > > > > >>> > the
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > design that Adi
> has
> > > > > proposed
> > > > > > > is
> > > > > > > > > > >> actually
> > > > > > > > > > >> > > not
> > > > > > > > > > >> > > > an
> > > > > > > > > > >> > > > >> > > error
> > > > > > > > > > >> > > > >> > > > > > > > condition
> > > > > > > > > > >> > > > >> > > > > > > > >>> at
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > all.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> It
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > is totally
> > reasonable
> > > in
> > > > > any
> > > > > > > > > > bootstrap
> > > > > > > > > > >> > > > >> situation
> > > > > > > > > > >> > > > >> > to
> > > > > > > > > > >> > > > >> > > > > > > > >>> intentionally
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > want to
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > run at the limit
> the
> > > > > system
> > > > > > > > > imposes
> > > > > > > > > > on
> > > > > > > > > > >> > you.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > -Jay
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > On Mon, Mar 16,
> 2015
> > > at
> > > > > 4:27
> > > > > > > PM,
> > > > > > > > > Jun
> > > > > > > > > > >> Rao
> > > > > > > > > > >> > <
> > > > > > > > > > >> > > > >> > > > > > j...@confluent.io>
> > > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> It's probably
> > useful
> > > for
> > > > > a
> > > > > > > > client
> > > > > > > > > > to
> > > > > > > > > > >> > know
> > > > > > > > > > >> > > > >> whether
> > > > > > > > > > >> > > > >> > > its
> > > > > > > > > > >> > > > >> > > > > > > > requests
> > > > > > > > > > >> > > > >> > > > > > > > >>> are
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> throttled or not
> > > (e.g.,
> > > > > for
> > > > > > > > > > monitoring
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > > >> > > alerting).
> > > > > > > > > > >> > > > >> > > > > > From
> > > > > > > > > > >> > > > >> > > > > > > > that
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> perspective,
> > option B
> > > > > > (delay
> > > > > > > > the
> > > > > > > > > > >> > requests
> > > > > > > > > > >> > > > and
> > > > > > > > > > >> > > > >> > > return an
> > > > > > > > > > >> > > > >> > > > > > > > error)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > seems
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> better.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> Thanks,
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> Jun
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> On Wed, Mar 4,
> 2015
> > > at
> > > > > 3:51
> > > > > > > PM,
> > > > > > > > > > Aditya
> > > > > > > > > > >> > > > >> Auradkar <
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > aaurad...@linkedin.com.invalid
> > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Posted a KIP
> for
> > > > > quotas
> > > > > > in
> > > > > > > > > kafka.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >>
> > > > > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Appreciate any
> > > > > feedback.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Aditya
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > --
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Thanks,
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Ewen
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> --
> > > > > > > > > > >> > > > >> > > > > > > > >>> -- Guozhang
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > --
> > > > > > > > > > >> > > > > Sent from Gmail Mobile
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > --
> > > > > > > > > > >> > > > Sent from Gmail Mobile
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
