Re: [KIP-DISCUSSION] KIP-13 Quotas

[Guozhang Wang]

I think KAFKA-2063 (bounding fetch response) is still under discussion, and
may not be got it in time with KAFKA-1927.

On Thu, Apr 9, 2015 at 4:49 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> I think it's reasonable to batch the protocol changes together. In
> addition to the protocol changes, is someone actively driving the server
> side changes/KIP process for KAFKA-2063?
>
> Thanks,
> Aditya
>
> ________________________________________
> From: Jun Rao [j...@confluent.io]
> Sent: Thursday, April 09, 2015 8:59 AM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Since we are also thinking about evolving the fetch request protocol in
> KAFKA-2063 (bound fetch response size), perhaps it's worth thinking through
> if we can just evolve the protocol once.
>
> Thanks,
>
> Jun
>
> On Wed, Apr 8, 2015 at 10:43 AM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Thanks for the detailed review. I've addressed your comments.
> >
> > For rejected alternatives, we've rejected per-partition distribution
> > because we choose client based quotas where there is no notion of
> > partitions. I've explained in a bit more detail in that section.
> >
> > Aditya
> >
> > ________________________________________
> > From: Joel Koshy [jjkosh...@gmail.com]
> > Sent: Wednesday, April 08, 2015 6:30 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> >
> > Thanks for updating the wiki. Looks great overall. Just a couple
> > more comments:
> >
> > Client status code:
> > - v0 requests -> current version (0) of those requests.
> > - Fetch response has a throttled flag instead of throttle time -  I
> >   think you intended the latter.
> > - Can you make it clear that the quota status is a new field
> >   called throttleTimeMs (or equivalent). It would help if some of
> >   that is moved (or repeated) in compatibility/migration plan.
> > - So you would need to upgrade brokers first, then the clients.
> >   While upgrading the brokers (via a rolling bounce) the brokers
> >   cannot start using the latest fetch-request version immediately
> >   (for replica fetches). Since there will be older brokers in the mix
> >   those brokers would not be able to read v1 fetch requests. So all
> >   the brokers should be upgraded before switching to the latest
> >   fetch request version. This is similar to what Gwen proposed in
> >   KIP-2/KAFKA-1809 and I think we will need to use the
> >   inter-broker protocol version config.
> >
> > Rejected alternatives-quota-distribution.B: notes that this is the
> > most elegant model, but does not explain why it was rejected. I
> > think this was because we would then need some sort of gossip
> > between brokers since partitions are across the cluster. Can you
> > confirm?
> >
> > Thanks,
> >
> > Joel
> >
> > On Wed, Apr 08, 2015 at 05:45:34AM +0000, Aditya Auradkar wrote:
> > > Hey everyone,
> > >
> > > Following up after today's hangout. After discussing the client side
> > metrics piece internally, we've incorporated that section into the KIP.
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > >
> > > Since there appears to be sufficient consensus, I'm going to start a
> > voting thread.
> > >
> > > Thanks,
> > > Aditya
> > > ________________________________________
> > > From: Gwen Shapira [gshap...@cloudera.com]
> > > Sent: Tuesday, April 07, 2015 11:31 AM
> > > To: Sriharsha Chintalapani
> > > Cc: dev@kafka.apache.org
> > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > >
> > > Yeah, I was not suggesting adding auth to metrics - I think this
> > needlessly
> > > complicates everything.
> > > But we need to assume that client developers will not have access to
> the
> > > broker metrics (because in secure environment they probably won't).
> > >
> > > Gwen
> > >
> > > On Tue, Apr 7, 2015 at 11:20 AM, Sriharsha Chintalapani <
> ka...@harsha.io
> > >
> > > wrote:
> > >
> > > > Having auth  on top of metrics is going to be lot more difficult. How
> > are
> > > > we going to restrict metrics reporter which run as part of kafka
> server
> > > > they will have access to all the metrics and they can publish to
> > ganglia
> > > > etc..  I look at the metrics as a read-only info. As you said metrics
> > for
> > > > all the topics can be visible but what actions are we looking that
> can
> > be
> > > > non-secure based on metrics alone? . This probably can be part of
> > KIP-11
> > > > discussion.
> > > >  Having said that it will be great if the throttling details can be
> > > > exposed as part of the response to the client. Instead of looking at
> > > > metrics , client can depend on the response to slow down if its being
> > > > throttled.  This allows us the clients can be self-reliant based on
> the
> > > > response .
> > > >
> > > > --
> > > > Harsha
> > > >
> > > >
> > > > On April 7, 2015 at 9:55:41 AM, Gwen Shapira (gshap...@cloudera.com)
> > > > wrote:
> > > >
> > > > Re (1):
> > > > We have no authorization story on the metrics collected by brokers,
> so
> > I
> > > > assume that access to broker metrics means knowing exactly which
> topics
> > > > exist and their throughputs. (Prath and Don, correct me if I got it
> > > > wrong...)
> > > > Secure environments will strictly control access to this information,
> > so I
> > > > am pretty sure the client developers will not have access to server
> > > > metrics
> > > > at all.
> > > >
> > > > Gwen
> > > >
> > > > On Tue, Apr 7, 2015 at 7:41 AM, Jay Kreps <jay.kr...@gmail.com>
> wrote:
> > > >
> > > > > Totally. But is that the only use? What I wanted to flesh out was
> > > > whether
> > > > > the goal was:
> > > > > 1. Expose throttling in the client metrics
> > > > > 2. Enable programmatic response (i.e. stop sending stuff or
> something
> > > > like
> > > > > that)
> > > > >
> > > > > I think I kind of understand (1) but let's get specific on the
> > metric we
> > > > > would be adding and what exactly you would expose in a dashboard.
> For
> > > > > example if the goal is just monitoring do I really want a boolean
> > flag
> > > > for
> > > > > is_throttled or do I want to know how much I am being throttled
> (i.e.
> > > > > throttle_pct might indicate the percent of your request time that
> was
> > > > due
> > > > > to throttling or something like that)? If I am 1% throttled that
> may
> > be
> > > > > irrelevant but 99% throttled would be quite relevant? Not sure I
> > agree,
> > > > > just throwing that out there...
> > > > >
> > > > > For (2) the prior discussion seemed to kind of allude to this but I
> > > > can't
> > > > > really come up with a use case. Is there one?
> > > > >
> > > > > If it is just (1) I think the question is whether it really helps
> > much
> > > > to
> > > > > have the metric on the client vs the server. I suppose this is a
> bit
> > > > > environment specific. If you have a central metrics system it
> > shouldn't
> > > > > make any difference, but if you don't I suppose it does.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Mon, Apr 6, 2015 at 7:57 PM, Gwen Shapira <
> gshap...@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > Here's a wild guess:
> > > > > >
> > > > > > An app developer included a Kafka Producer in his app, and is not
> > > > happy
> > > > > > with the throughput. He doesn't have visibility into the brokers
> > since
> > > > > they
> > > > > > are owned by a different team. Obviously the first instinct of a
> > > > > developer
> > > > > > who knows that throttling exists is to blame throttling for any
> > > > slowdown
> > > > > in
> > > > > > the app.
> > > > > > If he doesn't have a way to know from the responses whether or
> not
> > his
> > > > > app
> > > > > > is throttled, he may end up calling Aditya at 4am asked "Hey, is
> my
> > > > app
> > > > > > throttled?".
> > > > > >
> > > > > > I assume Aditya is trying to avoid this scenario.
> > > > > >
> > > > > > On Mon, Apr 6, 2015 at 7:47 PM, Jay Kreps <jay.kr...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hey Aditya,
> > > > > > >
> > > > > > > 2. I kind of buy it, but I really like to understand the
> details
> > of
> > > > the
> > > > > > use
> > > > > > > case before we make protocol changes. What changes are you
> > proposing
> > > > in
> > > > > > the
> > > > > > > clients for monitoring and how would that be used?
> > > > > > >
> > > > > > > -Jay
> > > > > > >
> > > > > > > On Mon, Apr 6, 2015 at 10:36 AM, Aditya Auradkar <
> > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > >
> > > > > > > > Hi Jay,
> > > > > > > >
> > > > > > > > 2. At this time, the proposed response format changes are
> only
> > for
> > > > > > > > monitoring/informing clients. As Jun mentioned, we get
> instance
> > > > level
> > > > > > > > monitoring in this case since each instance that got
> throttled
> > > > will
> > > > > > have
> > > > > > > a
> > > > > > > > metric confirming the same. Without client level monitoring
> for
> > > > this,
> > > > > > > it's
> > > > > > > > hard for application developers to find if they are being
> > > > throttled
> > > > > > since
> > > > > > > > they will also have to be aware of all the brokers in the
> > cluster.
> > > > > This
> > > > > > > is
> > > > > > > > quite problematic for large clusters.
> > > > > > > >
> > > > > > > > It seems nice for app developers to not have to think about
> > kafka
> > > > > > > internal
> > > > > > > > metrics and only focus on the metrics exposed on their
> > instances.
> > > > > > > Analogous
> > > > > > > > to having client-sde request latency metrics. Basically, we
> > want
> > > > an
> > > > > > easy
> > > > > > > > way for clients to be aware if they are being throttled.
> > > > > > > >
> > > > > > > > 4. For purgatory v delay queue, I think we are on the same
> > page. I
> > > > > feel
> > > > > > > it
> > > > > > > > is nicer to use the purgatory but I'm happy to use a
> > DelayQueue if
> > > > > > there
> > > > > > > > are performance implications. I don't know enough about the
> > > > current
> > > > > and
> > > > > > > > Yasuhiro's new implementation to be sure one way or the
> other.
> > > > > > > >
> > > > > > > > Stepping back, I think these two things are the only
> remaining
> > > > point
> > > > > of
> > > > > > > > discussion within the current proposal. Any concerns if I
> > started
> > > > a
> > > > > > > voting
> > > > > > > > thread on the proposal after the KIP discussion tomorrow?
> > > > (assuming
> > > > > we
> > > > > > > > reach consensus on these items)
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Aditya
> > > > > > > > ________________________________________
> > > > > > > > From: Jay Kreps [jay.kr...@gmail.com]
> > > > > > > > Sent: Saturday, April 04, 2015 1:36 PM
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > >
> > > > > > > > Hey Aditya,
> > > > > > > >
> > > > > > > > 2. For the return flag I'm not terribly particular. If we
> want
> > to
> > > > add
> > > > > > it
> > > > > > > > let's fully think through how it will be used. The only
> > concern I
> > > > > have
> > > > > > is
> > > > > > > > adding to the protocol without really thinking through the
> use
> > > > cases.
> > > > > > So
> > > > > > > > let's work out the APIs we want to add to the Java consumer
> and
> > > > > > producer
> > > > > > > > and the use cases for how clients will make use of these. For
> > my
> > > > > part I
> > > > > > > > actually don't see much use other than monitoring since it
> > isn't
> > > > an
> > > > > > error
> > > > > > > > condition to be at your quota. And if it is just monitoring I
> > > > don't
> > > > > > see a
> > > > > > > > big enough difference between having the monitoring on the
> > > > > server-side
> > > > > > > > versus in the clients to justify putting it in the protocol.
> > But I
> > > > > > think
> > > > > > > > you guys may have other use cases in mind of how a client
> would
> > > > make
> > > > > > some
> > > > > > > > use of this? Let's work that out. I also don't feel strongly
> > about
> > > > > > it--it
> > > > > > > > wouldn't be *bad* to have the monitoring available on the
> > client,
> > > > > just
> > > > > > > > doesn't seem that much better.
> > > > > > > >
> > > > > > > > 4. For the purgatory vs delay queue I think is arguably nicer
> > to
> > > > > reuse
> > > > > > > the
> > > > > > > > purgatory we just have to be ultra-conscious of efficiency. I
> > > > think
> > > > > our
> > > > > > > > goal is to turn quotas on across the board, so at LinkedIn
> that
> > > > would
> > > > > > > mean
> > > > > > > > potentially every request will need a small delay. I haven't
> > > > worked
> > > > > out
> > > > > > > the
> > > > > > > > efficiency implications of this choice, so as long as we do
> > that
> > > > I'm
> > > > > > > happy.
> > > > > > > >
> > > > > > > > -Jay
> > > > > > > >
> > > > > > > > On Fri, Apr 3, 2015 at 1:10 PM, Aditya Auradkar <
> > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > Some responses to Jay's points.
> > > > > > > > >
> > > > > > > > > 1. Using commas - Cool.
> > > > > > > > >
> > > > > > > > > 2. Adding return flag - I'm inclined to agree with Joel
> that
> > > > this
> > > > > is
> > > > > > > good
> > > > > > > > > to have in the initial implementation.
> > > > > > > > >
> > > > > > > > > 3. Config - +1. I'll remove it from the KIP. We can discuss
> > this
> > > > in
> > > > > > > > > parallel.
> > > > > > > > >
> > > > > > > > > 4. Purgatory vs Delay queue - I feel that it is simpler to
> > reuse
> > > > > the
> > > > > > > > > existing purgatories for both delayed produce and fetch
> > > > requests.
> > > > > > IIUC,
> > > > > > > > all
> > > > > > > > > we need for quotas is a minWait parameter for
> > DelayedOperation
> > > > (or
> > > > > > > > > something equivalent) since there is already a max wait.
> The
> > > > > > completion
> > > > > > > > > criteria can check if minWait time has elapsed before
> > declaring
> > > > the
> > > > > > > > > operation complete. For this to impact performance, a
> > > > significant
> > > > > > > number
> > > > > > > > of
> > > > > > > > > clients may need to exceed their quota at the same time and
> > even
> > > > > then
> > > > > > > I'm
> > > > > > > > > not very clear on the scope of the impact. Two layers of
> > delays
> > > > > might
> > > > > > > add
> > > > > > > > > complexity to the implementation which I'm hoping to avoid.
> > > > > > > > >
> > > > > > > > > Aditya
> > > > > > > > >
> > > > > > > > > ________________________________________
> > > > > > > > > From: Joel Koshy [jjkosh...@gmail.com]
> > > > > > > > > Sent: Friday, April 03, 2015 12:48 PM
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > >
> > > > > > > > > Aditya, thanks for the updated KIP and Jay/Jun thanks for
> the
> > > > > > > > > comments. Couple of comments in-line:
> > > > > > > > >
> > > > > > > > > > 2. I would advocate for adding the return flag when we
> next
> > > > bump
> > > > > > the
> > > > > > > > > > request format version just to avoid proliferation. I
> agree
> > > > this
> > > > > > is a
> > > > > > > > > good
> > > > > > > > > > thing to know about, but at the moment I don't think we
> > have a
> > > > > very
> > > > > > > > well
> > > > > > > > > > flushed out idea of how the client would actually make
> use
> > of
> > > > > this
> > > > > > > > info.
> > > > > > > > > I
> > > > > > > > >
> > > > > > > > > I'm somewhat inclined to having something appropriate off
> the
> > > > bat -
> > > > > > > > > mainly because (i) clients really should know that they
> have
> > > > been
> > > > > > > > > throttled (ii) a smart producer/consumer implementation
> would
> > > > want
> > > > > to
> > > > > > > > > know how much to back off. So perhaps this and
> > config-management
> > > > > > > > > should be moved to a separate discussion, but it would be
> > good
> > > > to
> > > > > > have
> > > > > > > > > this discussion going and incorporated into the first quota
> > > > > > > > > implementation.
> > > > > > > > >
> > > > > > > > > > 3. Config--I think we need to generalize the topic stuff
> > so we
> > > > > can
> > > > > > > > > override
> > > > > > > > > > at multiple levels. We have topic and client, but I
> suspect
> > > > > "user"
> > > > > > > and
> > > > > > > > > > "broker" will also be important. I recommend we take
> config
> > > > stuff
> > > > > > out
> > > > > > > > of
> > > > > > > > > > this KIP since we really need to fully think through a
> > > > proposal
> > > > > > that
> > > > > > > > will
> > > > > > > > > > cover all these types of overrides.
> > > > > > > > >
> > > > > > > > > +1 - it is definitely orthogonal to the core quota
> > > > implementation
> > > > > > > > > (although necessary for its operability). Having a
> > > > config-related
> > > > > > > > > discussion in this KIP would only draw out the discussion
> and
> > > > vote
> > > > > > > > > even if the core quota design looks good to everyone.
> > > > > > > > >
> > > > > > > > > So basically I think we can remove the portions on dynamic
> > > > config
> > > > > as
> > > > > > > > > well as the response format but I really think we should
> > close
> > > > on
> > > > > > > > > those while the implementation is in progress and before
> > quotas
> > > > is
> > > > > > > > > officially released.
> > > > > > > > >
> > > > > > > > > > 4. Instead of using purgatories to implement the delay
> > would
> > > > it
> > > > > > make
> > > > > > > > more
> > > > > > > > > > sense to just use a delay queue? I think all the
> additional
> > > > stuff
> > > > > > in
> > > > > > > > the
> > > > > > > > > > purgatory other than the delay queue doesn't make sense
> as
> > the
> > > > > > quota
> > > > > > > > is a
> > > > > > > > > > hard N ms penalty with no chance of early eviction. If
> > there
> > > > is
> > > > > no
> > > > > > > perf
> > > > > > > > > > penalty for the full purgatory that may be fine (even
> > good) to
> > > > > > reuse,
> > > > > > > > > but I
> > > > > > > > > > haven't looked into that.
> > > > > > > > >
> > > > > > > > > A simple delay queue sounds good - I think Aditya was also
> > > > trying
> > > > > to
> > > > > > > > > avoid adding a new quota purgatory. i.e., it may be
> possible
> > to
> > > > use
> > > > > > > > > the existing purgatory instances to enforce quotas. That
> may
> > be
> > > > > > > > > simpler, but would be incur a slight perf penalty if too
> many
> > > > > clients
> > > > > > > > > are being throttled.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Joel
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -Jay
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 3, 2015 at 10:45 AM, Aditya Auradkar <
> > > > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > >> Update, I added a proposal on doing dynamic client based
> > > > > > > configuration
> > > > > > > > > >> that can be used for quotas.
> > > > > > > > > >>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > >>
> > > > > > > > > >> Please take a look and let me know if there are any
> > concerns.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Aditya
> > > > > > > > > >> ________________________________________
> > > > > > > > > >> From: Aditya Auradkar
> > > > > > > > > >> Sent: Friday, April 03, 2015 10:10 AM
> > > > > > > > > >> To: dev@kafka.apache.org
> > > > > > > > > >> Subject: RE: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > >>
> > > > > > > > > >> Thanks Jun.
> > > > > > > > > >>
> > > > > > > > > >> Some thoughts:
> > > > > > > > > >>
> > > > > > > > > >> 10) I think it is better we throttle regardless of the
> > > > > > produce/fetch
> > > > > > > > > >> version. This is a nice feature where clients can tell
> if
> > > > they
> > > > > are
> > > > > > > > being
> > > > > > > > > >> throttled or not. If we only throttle newer clients,
> then
> > we
> > > > > have
> > > > > > > > > >> inconsistent behavior across clients in a multi-tenant
> > > > cluster.
> > > > > > > Having
> > > > > > > > > >> quota metrics on the client side is also a nice
> incentive
> > to
> > > > > > upgrade
> > > > > > > > > client
> > > > > > > > > >> versions.
> > > > > > > > > >>
> > > > > > > > > >> 11) I think we can call metric.record(fetchSize) before
> > > > adding
> > > > > the
> > > > > > > > > >> delayedFetch request into the purgatory. This will give
> us
> > > > the
> > > > > > > > estimated
> > > > > > > > > >> delay of the request up-front. The timeout on the
> > > > DelayedFetch
> > > > > is
> > > > > > > the
> > > > > > > > > >> Max(maxWait, quotaDelay). The DelayedFetch completion
> > > > criteria
> > > > > can
> > > > > > > > > change a
> > > > > > > > > >> little to accomodate quotas.
> > > > > > > > > >>
> > > > > > > > > >> - I agree the quota code should return the estimated
> delay
> > > > time
> > > > > in
> > > > > > > > > >> QuotaViolationException.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Aditya
> > > > > > > > > >>
> > > > > > > > > >> ________________________________________
> > > > > > > > > >> From: Jun Rao [j...@confluent.io]
> > > > > > > > > >> Sent: Friday, April 03, 2015 9:16 AM
> > > > > > > > > >> To: dev@kafka.apache.org
> > > > > > > > > >> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > >>
> > > > > > > > > >> Thanks for the update.
> > > > > > > > > >>
> > > > > > > > > >> 10. About whether to return a new field in the response
> to
> > > > > > indicate
> > > > > > > > > >> throttling. Earlier, the plan was to not change the
> > response
> > > > > > format
> > > > > > > > and
> > > > > > > > > >> just have a metric on the broker to indicate whether a
> > > > clientId
> > > > > is
> > > > > > > > > >> throttled or not. The issue is that we don't know
> whether
> > a
> > > > > > > particular
> > > > > > > > > >> clientId instance is throttled or not (since there could
> > be
> > > > > > multiple
> > > > > > > > > >> clients with the same clientId). Your proposal of adding
> > an
> > > > > > > > isThrottled
> > > > > > > > > >> field in the response addresses and seems better. Then,
> > do we
> > > > > just
> > > > > > > > > throttle
> > > > > > > > > >> the new version of produce/fetch request or both the old
> > and
> > > > the
> > > > > > new
> > > > > > > > > >> versions? Also, we probably still need a separate metric
> > on
> > > > the
> > > > > > > broker
> > > > > > > > > side
> > > > > > > > > >> to indicate whether a clientId is throttled or not.
> > > > > > > > > >>
> > > > > > > > > >> 11. Just to clarify. For fetch requests, when will
> > > > > > > > > metric.record(fetchSize)
> > > > > > > > > >> be called? Is it when we are ready to send the fetch
> > response
> > > > > > (after
> > > > > > > > > >> minBytes and maxWait are satisfied)?
> > > > > > > > > >>
> > > > > > > > > >> As an implementation detail, it may be useful for the
> > quota
> > > > code
> > > > > > to
> > > > > > > > > return
> > > > > > > > > >> an estimated delay time (to bring the measurement within
> > the
> > > > > > limit)
> > > > > > > in
> > > > > > > > > >> QuotaViolationException.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >>
> > > > > > > > > >> Jun
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Apr 1, 2015 at 3:27 PM, Aditya Auradkar <
> > > > > > > > > >> aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hey everyone,
> > > > > > > > > >> >
> > > > > > > > > >> > I've made changes to the KIP to capture our
> discussions
> > > > over
> > > > > the
> > > > > > > > last
> > > > > > > > > >> > couple of weeks.
> > > > > > > > > >> >
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > >> >
> > > > > > > > > >> > I'll start a voting thread after people have had a
> > chance
> > > > to
> > > > > > > > > >> read/comment.
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > Aditya
> > > > > > > > > >> >
> > > > > > > > > >> > ________________________________________
> > > > > > > > > >> > From: Steven Wu [stevenz...@gmail.com]
> > > > > > > > > >> > Sent: Friday, March 20, 2015 9:14 AM
> > > > > > > > > >> > To: dev@kafka.apache.org
> > > > > > > > > >> > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > >> >
> > > > > > > > > >> > +1 on Jun's suggestion of maintaining one set/style of
> > > > metrics
> > > > > > at
> > > > > > > > > broker.
> > > > > > > > > >> > In Netflix, we have to convert the yammer metrics to
> > servo
> > > > > > metrics
> > > > > > > > at
> > > > > > > > > >> > broker. it will be painful to know some metrics are
> in a
> > > > > > different
> > > > > > > > > style
> > > > > > > > > >> > and get to be handled differently.
> > > > > > > > > >> >
> > > > > > > > > >> > On Fri, Mar 20, 2015 at 8:17 AM, Jun Rao <
> > j...@confluent.io>
> > > >
> > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Not so sure. People who use quota will definitely
> > want to
> > > > > > > monitor
> > > > > > > > > the
> > > > > > > > > >> new
> > > > > > > > > >> > > metrics at the client id level. Then they will need
> to
> > > > deal
> > > > > > with
> > > > > > > > > those
> > > > > > > > > >> > > metrics differently from the rest of the metrics. It
> > > > would
> > > > > be
> > > > > > > > > better if
> > > > > > > > > >> > we
> > > > > > > > > >> > > can hide this complexity from the users.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Jun
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Thu, Mar 19, 2015 at 10:45 PM, Joel Koshy <
> > > > > > > jjkosh...@gmail.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Actually thinking again - since these will be a
> few
> > new
> > > > > > > metrics
> > > > > > > > at
> > > > > > > > > >> the
> > > > > > > > > >> > > > client id level (bytes in and bytes out to start
> > with)
> > > > > maybe
> > > > > > > it
> > > > > > > > is
> > > > > > > > > >> fine
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > have the two type of metrics coexist and we can
> > migrate
> > > > > the
> > > > > > > > > existing
> > > > > > > > > >> > > > metrics in parallel.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On Thursday, March 19, 2015, Joel Koshy <
> > > > > > jjkosh...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > That is a valid concern but in that case I think
> > it
> > > > > would
> > > > > > be
> > > > > > > > > better
> > > > > > > > > >> > to
> > > > > > > > > >> > > > > just migrate completely to the new metrics
> package
> > > > > first.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > On Thursday, March 19, 2015, Jun Rao <
> > > > j...@confluent.io
> > > > > > > > > >> > > > > <javascript:_e(%7B%7D,'cvml','j...@confluent.io
> > ');>>
> > > > > > wrote:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >> Hmm, I was thinking a bit differently on the
> > metrics
> > > > > > > stuff. I
> > > > > > > > > >> think
> > > > > > > > > >> > it
> > > > > > > > > >> > > > >> would be confusing to have some metrics defined
> > in
> > > > the
> > > > > > new
> > > > > > > > > metrics
> > > > > > > > > >> > > > package
> > > > > > > > > >> > > > >> while some others defined in Coda Hale. Those
> > > > metrics
> > > > > > will
> > > > > > > > look
> > > > > > > > > >> > > > different
> > > > > > > > > >> > > > >> (e.g., rates in Coda Hale will have special
> > > > attributes
> > > > > > such
> > > > > > > > as
> > > > > > > > > >> > > > >> 1-min-average). People may need different ways
> to
> > > > > export
> > > > > > > the
> > > > > > > > > >> metrics
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > >> external systems such as Graphite. So, instead
> of
> > > > using
> > > > > > the
> > > > > > > > new
> > > > > > > > > >> > > metrics
> > > > > > > > > >> > > > >> package on the broker, I was thinking that we
> can
> > > > just
> > > > > > > > > implement a
> > > > > > > > > >> > > > >> QuotaMetrics that wraps the Coda Hale metrics.
> > The
> > > > > > > > > implementation
> > > > > > > > > >> > can
> > > > > > > > > >> > > be
> > > > > > > > > >> > > > >> the same as what's in the new metrics package.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Thanks,
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Jun
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> On Thu, Mar 19, 2015 at 8:09 PM, Jay Kreps <
> > > > > > > > > jay.kr...@gmail.com>
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> > Yeah I was saying was that we are blocked on
> > > > picking
> > > > > an
> > > > > > > > > approach
> > > > > > > > > >> > for
> > > > > > > > > >> > > > >> > metrics but not necessarily the full
> > conversion.
> > > > > > Clearly
> > > > > > > if
> > > > > > > > > we
> > > > > > > > > >> > pick
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > >> new
> > > > > > > > > >> > > > >> > metrics package we would need to implement
> the
> > two
> > > > > > > metrics
> > > > > > > > we
> > > > > > > > > >> want
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > >> quota
> > > > > > > > > >> > > > >> > on. But the conversion of the remaining
> metrics
> > > > can
> > > > > be
> > > > > > > done
> > > > > > > > > >> > > > >> asynchronously.
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > -Jay
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > On Thu, Mar 19, 2015 at 5:56 PM, Joel Koshy <
> > > > > > > > > >> jjkosh...@gmail.com>
> > > > > > > > > >> > > > >> wrote:
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > > > in KAFKA-1930). I agree that this KIP
> > doesn't
> > > > > need
> > > > > > to
> > > > > > > > > block
> > > > > > > > > >> on
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > >> > > > migration of the metrics package.
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > Can you clarify the above? i.e., if we are
> > going
> > > > to
> > > > > > > quota
> > > > > > > > > on
> > > > > > > > > >> > > > something
> > > > > > > > > >> > > > >> > > then we would want to have migrated that
> > metric
> > > > > over
> > > > > > > > > right? Or
> > > > > > > > > >> > do
> > > > > > > > > >> > > > you
> > > > > > > > > >> > > > >> > > mean we don't need to complete the
> migration
> > of
> > > > all
> > > > > > > > > metrics to
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> > > metrics package right?
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > I think most of us now feel that the delay
> +
> > no
> > > > > error
> > > > > > > is
> > > > > > > > a
> > > > > > > > > >> good
> > > > > > > > > >> > > > >> > > approach, but it would be good to make sure
> > > > > everyone
> > > > > > is
> > > > > > > > on
> > > > > > > > > the
> > > > > > > > > >> > > same
> > > > > > > > > >> > > > >> > > page.
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > As Aditya requested a couple of days ago I
> > think
> > > > we
> > > > > > > > should
> > > > > > > > > go
> > > > > > > > > >> > over
> > > > > > > > > >> > > > >> > > this at the next KIP hangout.
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > Joel
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > On Thu, Mar 19, 2015 at 09:24:09AM -0700,
> Jun
> > > > Rao
> > > > > > > wrote:
> > > > > > > > > >> > > > >> > > > 1. Delay + no error seems reasonable to
> me.
> > > > > > However,
> > > > > > > I
> > > > > > > > do
> > > > > > > > > >> feel
> > > > > > > > > >> > > > that
> > > > > > > > > >> > > > >> we
> > > > > > > > > >> > > > >> > > need
> > > > > > > > > >> > > > >> > > > to give the client an indicator that it's
> > > > being
> > > > > > > > > throttled,
> > > > > > > > > >> > > instead
> > > > > > > > > >> > > > >> of
> > > > > > > > > >> > > > >> > > doing
> > > > > > > > > >> > > > >> > > > this silently. For that, we probably need
> > to
> > > > > evolve
> > > > > > > the
> > > > > > > > > >> > > > >> produce/fetch
> > > > > > > > > >> > > > >> > > > protocol to include an extra status field
> > in
> > > > the
> > > > > > > > > response.
> > > > > > > > > >> We
> > > > > > > > > >> > > > >> probably
> > > > > > > > > >> > > > >> > > need
> > > > > > > > > >> > > > >> > > > to think more about whether we just want
> to
> > > > > return
> > > > > > a
> > > > > > > > > simple
> > > > > > > > > >> > > status
> > > > > > > > > >> > > > >> code
> > > > > > > > > >> > > > >> > > > (e.g., 1 = throttled) or a value that
> > > > indicates
> > > > > how
> > > > > > > > much
> > > > > > > > > is
> > > > > > > > > >> > > being
> > > > > > > > > >> > > > >> > > throttled.
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > > > 2. We probably need to improve the
> > histogram
> > > > > > support
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >> > new
> > > > > > > > > >> > > > >> metrics
> > > > > > > > > >> > > > >> > > > package before we can use it more widely
> on
> > > > the
> > > > > > > server
> > > > > > > > > side
> > > > > > > > > >> > > (left
> > > > > > > > > >> > > > a
> > > > > > > > > >> > > > >> > > comment
> > > > > > > > > >> > > > >> > > > in KAFKA-1930). I agree that this KIP
> > doesn't
> > > > > need
> > > > > > to
> > > > > > > > > block
> > > > > > > > > >> on
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > >> > > > migration of the metrics package.
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > > > Jun
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > > > On Wed, Mar 18, 2015 at 4:02 PM, Aditya
> > > > Auradkar
> > > > > <
> > > > > > > > > >> > > > >> > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > > > > Hey everyone,
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > Thanks for the great discussion. There
> > are
> > > > > > > currently
> > > > > > > > a
> > > > > > > > > few
> > > > > > > > > >> > > > points
> > > > > > > > > >> > > > >> on
> > > > > > > > > >> > > > >> > > this
> > > > > > > > > >> > > > >> > > > > KIP that need addressing and I want to
> > make
> > > > > sure
> > > > > > we
> > > > > > > > > are on
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> same
> > > > > > > > > >> > > > >> > > page
> > > > > > > > > >> > > > >> > > > > about those.
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > 1. Append and delay response vs delay
> and
> > > > > return
> > > > > > > > error
> > > > > > > > > >> > > > >> > > > > - I think we've discussed the pros and
> > cons
> > > > of
> > > > > > each
> > > > > > > > > >> approach
> > > > > > > > > >> > > but
> > > > > > > > > >> > > > >> > > haven't
> > > > > > > > > >> > > > >> > > > > chosen an approach yet. Where does
> > everyone
> > > > > stand
> > > > > > > on
> > > > > > > > > this
> > > > > > > > > >> > > issue?
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > 2. Metrics Migration and usage in
> quotas
> > > > > > > > > >> > > > >> > > > > - The metrics library in clients has a
> > > > notion
> > > > > of
> > > > > > > > quotas
> > > > > > > > > >> that
> > > > > > > > > >> > > we
> > > > > > > > > >> > > > >> > should
> > > > > > > > > >> > > > >> > > > > reuse. For that to happen, we need to
> > > > migrate
> > > > > the
> > > > > > > > > server
> > > > > > > > > >> to
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > >> new
> > > > > > > > > >> > > > >> > > metrics
> > > > > > > > > >> > > > >> > > > > package.
> > > > > > > > > >> > > > >> > > > > - Need more clarification on how to
> > compute
> > > > > > > > throttling
> > > > > > > > > >> time
> > > > > > > > > >> > > and
> > > > > > > > > >> > > > >> > > windowing
> > > > > > > > > >> > > > >> > > > > for quotas.
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > I'm going to start a new KIP to discuss
> > > > metrics
> > > > > > > > > migration
> > > > > > > > > >> > > > >> separately.
> > > > > > > > > >> > > > >> > > That
> > > > > > > > > >> > > > >> > > > > will also contain a section on quotas.
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > 3. Dynamic Configuration management -
> > Being
> > > > > > > discussed
> > > > > > > > > in
> > > > > > > > > >> > > KIP-5.
> > > > > > > > > >> > > > >> > > Basically
> > > > > > > > > >> > > > >> > > > > we need something that will model
> default
> > > > > quotas
> > > > > > > and
> > > > > > > > > allow
> > > > > > > > > >> > > > >> per-client
> > > > > > > > > >> > > > >> > > > > overrides.
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > Is there something else that I'm
> missing?
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > Thanks,
> > > > > > > > > >> > > > >> > > > > Aditya
> > > > > > > > > >> > > > >> > > > >
> ________________________________________
> > > > > > > > > >> > > > >> > > > > From: Jay Kreps [jay.kr...@gmail.com]
> > > > > > > > > >> > > > >> > > > > Sent: Wednesday, March 18, 2015 2:10 PM
> > > > > > > > > >> > > > >> > > > > To: dev@kafka.apache.org
> > > > > > > > > >> > > > >> > > > > Subject: Re: [KIP-DISCUSSION] KIP-13
> > Quotas
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > Hey Steven,
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > The current proposal is actually to
> > enforce
> > > > > > quotas
> > > > > > > at
> > > > > > > > > the
> > > > > > > > > >> > > > >> > > > > client/application level, NOT the topic
> > > > level.
> > > > > So
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > >> > have
> > > > > > > > > >> > > a
> > > > > > > > > >> > > > >> > service
> > > > > > > > > >> > > > >> > > > > with a few dozen instances the quota is
> > > > against
> > > > > > all
> > > > > > > > of
> > > > > > > > > >> those
> > > > > > > > > >> > > > >> > instances
> > > > > > > > > >> > > > >> > > > > added up across all their topics. So
> > > > actually
> > > > > the
> > > > > > > > > effect
> > > > > > > > > >> > would
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> > > same
> > > > > > > > > >> > > > >> > > > > either way but throttling gives the
> > producer
> > > > > the
> > > > > > > > > choice of
> > > > > > > > > >> > > > either
> > > > > > > > > >> > > > >> > > blocking
> > > > > > > > > >> > > > >> > > > > or dropping.
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > -Jay
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > On Tue, Mar 17, 2015 at 10:08 AM,
> Steven
> > Wu
> > > > <
> > > > > > > > > >> > > > stevenz...@gmail.com
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > > wrote:
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > > Jay,
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > let's say an app produces to 10
> > different
> > > > > > topics.
> > > > > > > > > one of
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> topic
> > > > > > > > > >> > > > >> > is
> > > > > > > > > >> > > > >> > > > > sent
> > > > > > > > > >> > > > >> > > > > > from a library. due to whatever
> > > > > condition/bug,
> > > > > > > this
> > > > > > > > > lib
> > > > > > > > > >> > > starts
> > > > > > > > > >> > > > >> to
> > > > > > > > > >> > > > >> > > send
> > > > > > > > > >> > > > >> > > > > > messages over the quota. if we go
> with
> > the
> > > > > > > delayed
> > > > > > > > > >> > response
> > > > > > > > > >> > > > >> > > approach, it
> > > > > > > > > >> > > > >> > > > > > will cause the whole shared
> > > > RecordAccumulator
> > > > > > > > buffer
> > > > > > > > > to
> > > > > > > > > >> be
> > > > > > > > > >> > > > >> filled
> > > > > > > > > >> > > > >> > up.
> > > > > > > > > >> > > > >> > > > > that
> > > > > > > > > >> > > > >> > > > > > will penalize other 9 topics who are
> > > > within
> > > > > the
> > > > > > > > > quota.
> > > > > > > > > >> > that
> > > > > > > > > >> > > is
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> > > > > > unfairness point that Ewen and I were
> > > > trying
> > > > > to
> > > > > > > > make.
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > if broker just drop the msg and
> return
> > an
> > > > > > > > > error/status
> > > > > > > > > >> > code
> > > > > > > > > >> > > > >> > > indicates the
> > > > > > > > > >> > > > >> > > > > > drop and why. then producer can just
> > move
> > > > on
> > > > > > and
> > > > > > > > > accept
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> drop.
> > > > > > > > > >> > > > >> > > shared
> > > > > > > > > >> > > > >> > > > > > buffer won't be saturated and other 9
> > > > topics
> > > > > > > won't
> > > > > > > > be
> > > > > > > > > >> > > > penalized.
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > >> > > > >> > > > > > Steven
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > On Tue, Mar 17, 2015 at 9:44 AM, Jay
> > Kreps
> > > > <
> > > > > > > > > >> > > > jay.kr...@gmail.com
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > > wrote:
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > > Hey Steven,
> > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > >> > > > >> > > > > > > It is true that hitting the quota
> > will
> > > > > cause
> > > > > > > > > >> > back-pressure
> > > > > > > > > >> > > > on
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> > > > > > producer.
> > > > > > > > > >> > > > >> > > > > > > But the solution is simple, a
> > producer
> > > > that
> > > > > > > wants
> > > > > > > > > to
> > > > > > > > > >> > avoid
> > > > > > > > > >> > > > >> this
> > > > > > > > > >> > > > >> > > should
> > > > > > > > > >> > > > >> > > > > > stay
> > > > > > > > > >> > > > >> > > > > > > under its quota. In other words
> this
> > is
> > > > a
> > > > > > > > contract
> > > > > > > > > >> > between
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > >> > > cluster
> > > > > > > > > >> > > > >> > > > > > and
> > > > > > > > > >> > > > >> > > > > > > the client, with each side having
> > > > something
> > > > > > to
> > > > > > > > > uphold.
> > > > > > > > > >> > > Quite
> > > > > > > > > >> > > > >> > > possibly
> > > > > > > > > >> > > > >> > > > > the
> > > > > > > > > >> > > > >> > > > > > > same thing will happen in the
> > absence of
> > > > a
> > > > > > > > quota, a
> > > > > > > > > >> > client
> > > > > > > > > >> > > > >> that
> > > > > > > > > >> > > > >> > > > > produces
> > > > > > > > > >> > > > >> > > > > > an
> > > > > > > > > >> > > > >> > > > > > > unexpected amount of load will hit
> > the
> > > > > limits
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >> > > server
> > > > > > > > > >> > > > >> and
> > > > > > > > > >> > > > >> > > > > > experience
> > > > > > > > > >> > > > >> > > > > > > backpressure. Quotas just allow you
> > to
> > > > set
> > > > > > that
> > > > > > > > > same
> > > > > > > > > >> > limit
> > > > > > > > > >> > > > at
> > > > > > > > > >> > > > >> > > something
> > > > > > > > > >> > > > >> > > > > > > lower than 100% of all resources on
> > the
> > > > > > server,
> > > > > > > > > which
> > > > > > > > > >> is
> > > > > > > > > >> > > > >> useful
> > > > > > > > > >> > > > >> > > for a
> > > > > > > > > >> > > > >> > > > > > > shared cluster.
> > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > >> > > > >> > > > > > > -Jay
> > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > >> > > > >> > > > > > > On Mon, Mar 16, 2015 at 11:34 PM,
> > Steven
> > > > > Wu <
> > > > > > > > > >> > > > >> > stevenz...@gmail.com>
> > > > > > > > > >> > > > >> > > > > > wrote:
> > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > wait. we create one kafka
> producer
> > for
> > > > > each
> > > > > > > > > cluster.
> > > > > > > > > >> > > each
> > > > > > > > > >> > > > >> > > cluster can
> > > > > > > > > >> > > > >> > > > > > > have
> > > > > > > > > >> > > > >> > > > > > > > many topics. if producer buffer
> got
> > > > > filled
> > > > > > up
> > > > > > > > > due to
> > > > > > > > > >> > > > delayed
> > > > > > > > > >> > > > >> > > response
> > > > > > > > > >> > > > >> > > > > > for
> > > > > > > > > >> > > > >> > > > > > > > one throttled topic, won't that
> > > > penalize
> > > > > > > other
> > > > > > > > > >> topics
> > > > > > > > > >> > > > >> unfairly?
> > > > > > > > > >> > > > >> > > it
> > > > > > > > > >> > > > >> > > > > > seems
> > > > > > > > > >> > > > >> > > > > > > to
> > > > > > > > > >> > > > >> > > > > > > > me that broker should just return
> > > > error
> > > > > > > without
> > > > > > > > > >> delay.
> > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > sorry that I am chatting to
> myself
> > :)
> > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > On Mon, Mar 16, 2015 at 11:29 PM,
> > > > Steven
> > > > > > Wu <
> > > > > > > > > >> > > > >> > > stevenz...@gmail.com>
> > > > > > > > > >> > > > >> > > > > > > wrote:
> > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > > I think I can answer my own
> > > > question.
> > > > > > > delayed
> > > > > > > > > >> > response
> > > > > > > > > >> > > > >> will
> > > > > > > > > >> > > > >> > > cause
> > > > > > > > > >> > > > >> > > > > the
> > > > > > > > > >> > > > >> > > > > > > > > producer buffer to be full,
> which
> > > > then
> > > > > > > result
> > > > > > > > > in
> > > > > > > > > >> > > either
> > > > > > > > > >> > > > >> > thread
> > > > > > > > > >> > > > >> > > > > > blocking
> > > > > > > > > >> > > > >> > > > > > > > or
> > > > > > > > > >> > > > >> > > > > > > > > message drop.
> > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > > On Mon, Mar 16, 2015 at 11:24
> PM,
> > > > > Steven
> > > > > > > Wu <
> > > > > > > > > >> > > > >> > > stevenz...@gmail.com>
> > > > > > > > > >> > > > >> > > > > > > > wrote:
> > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > >> please correct me if I am
> > missing
> > > > sth
> > > > > > > here.
> > > > > > > > I
> > > > > > > > > am
> > > > > > > > > >> > not
> > > > > > > > > >> > > > >> > > understanding
> > > > > > > > > >> > > > >> > > > > > how
> > > > > > > > > >> > > > >> > > > > > > > >> would throttle work without
> > > > > > > > > cooperation/back-off
> > > > > > > > > >> > from
> > > > > > > > > >> > > > >> > > producer.
> > > > > > > > > >> > > > >> > > > > new
> > > > > > > > > >> > > > >> > > > > > > Java
> > > > > > > > > >> > > > >> > > > > > > > >> producer supports non-blocking
> > API.
> > > > > why
> > > > > > > > would
> > > > > > > > > >> > delayed
> > > > > > > > > >> > > > >> > > response be
> > > > > > > > > >> > > > >> > > > > > able
> > > > > > > > > >> > > > >> > > > > > > > to
> > > > > > > > > >> > > > >> > > > > > > > >> slow down producer? producer
> > will
> > > > > > continue
> > > > > > > > to
> > > > > > > > > >> fire
> > > > > > > > > >> > > > async
> > > > > > > > > >> > > > >> > > sends.
> > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > >> > > > >> > > > > > > > >> On Mon, Mar 16, 2015 at 10:58
> > PM,
> > > > > > Guozhang
> > > > > > > > > Wang <
> > > > > > > > > >> > > > >> > > > > wangg...@gmail.com
> > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > >> wrote:
> > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > >> > > > >> > > > > > > > >>> I think we are really
> > discussing
> > > > two
> > > > > > > > separate
> > > > > > > > > >> > issues
> > > > > > > > > >> > > > >> here:
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> 1. Whether we should a)
> > > > > > > > > >> > > > >> > > > >
> > append-then-block-then-returnOKButThrottled
> > > > > > > > > >> > > > >> > > > > > > or
> > > > > > > > > >> > > > >> > > > > > > > b)
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > block-then-returnFailDuetoThrottled
> > > > > for
> > > > > > > > quota
> > > > > > > > > >> > > actions
> > > > > > > > > >> > > > on
> > > > > > > > > >> > > > >> > > produce
> > > > > > > > > >> > > > >> > > > > > > > >>> requests.
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> Both these approaches assume
> > some
> > > > > kind
> > > > > > of
> > > > > > > > > >> > > > >> well-behaveness
> > > > > > > > > >> > > > >> > of
> > > > > > > > > >> > > > >> > > the
> > > > > > > > > >> > > > >> > > > > > > > clients:
> > > > > > > > > >> > > > >> > > > > > > > >>> option a) assumes the client
> > sets
> > > > an
> > > > > > > proper
> > > > > > > > > >> > timeout
> > > > > > > > > >> > > > >> value
> > > > > > > > > >> > > > >> > > while
> > > > > > > > > >> > > > >> > > > > can
> > > > > > > > > >> > > > >> > > > > > > > just
> > > > > > > > > >> > > > >> > > > > > > > >>> ignore "OKButThrottled"
> > response,
> > > > > while
> > > > > > > > > option
> > > > > > > > > >> b)
> > > > > > > > > >> > > > >> assumes
> > > > > > > > > >> > > > >> > the
> > > > > > > > > >> > > > >> > > > > > client
> > > > > > > > > >> > > > >> > > > > > > > >>> handles the
> > "FailDuetoThrottled"
> > > > > > > > > appropriately.
> > > > > > > > > >> > For
> > > > > > > > > >> > > > any
> > > > > > > > > >> > > > >> > > malicious
> > > > > > > > > >> > > > >> > > > > > > > clients
> > > > > > > > > >> > > > >> > > > > > > > >>> that, for example, just keep
> > > > retrying
> > > > > > > > either
> > > > > > > > > >> > > > >> intentionally
> > > > > > > > > >> > > > >> > or
> > > > > > > > > >> > > > >> > > > > not,
> > > > > > > > > >> > > > >> > > > > > > > >>> neither
> > > > > > > > > >> > > > >> > > > > > > > >>> of these approaches are
> > actually
> > > > > > > effective.
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> 2. For "OKButThrottled" and
> > > > > > > > > "FailDuetoThrottled"
> > > > > > > > > >> > > > >> responses,
> > > > > > > > > >> > > > >> > > shall
> > > > > > > > > >> > > > >> > > > > > we
> > > > > > > > > >> > > > >> > > > > > > > >>> encode
> > > > > > > > > >> > > > >> > > > > > > > >>> them as error codes or
> augment
> > the
> > > > > > > protocol
> > > > > > > > > to
> > > > > > > > > >> > use a
> > > > > > > > > >> > > > >> > separate
> > > > > > > > > >> > > > >> > > > > field
> > > > > > > > > >> > > > >> > > > > > > > >>> indicating "status codes".
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> Today we have already
> > incorporated
> > > > > some
> > > > > > > > > status
> > > > > > > > > >> > code
> > > > > > > > > >> > > as
> > > > > > > > > >> > > > >> > error
> > > > > > > > > >> > > > >> > > > > codes
> > > > > > > > > >> > > > >> > > > > > in
> > > > > > > > > >> > > > >> > > > > > > > the
> > > > > > > > > >> > > > >> > > > > > > > >>> responses, e.g.
> > > > ReplicaNotAvailable
> > > > > in
> > > > > > > > > >> > > > MetadataResponse,
> > > > > > > > > >> > > > >> > the
> > > > > > > > > >> > > > >> > > pros
> > > > > > > > > >> > > > >> > > > > > of
> > > > > > > > > >> > > > >> > > > > > > > this
> > > > > > > > > >> > > > >> > > > > > > > >>> is of course using a single
> > field
> > > > for
> > > > > > > > > response
> > > > > > > > > >> > > status
> > > > > > > > > >> > > > >> like
> > > > > > > > > >> > > > >> > > the
> > > > > > > > > >> > > > >> > > > > HTTP
> > > > > > > > > >> > > > >> > > > > > > > >>> status
> > > > > > > > > >> > > > >> > > > > > > > >>> codes, while the cons is that
> > it
> > > > > > requires
> > > > > > > > > >> clients
> > > > > > > > > >> > to
> > > > > > > > > >> > > > >> handle
> > > > > > > > > >> > > > >> > > the
> > > > > > > > > >> > > > >> > > > > > error
> > > > > > > > > >> > > > >> > > > > > > > >>> codes
> > > > > > > > > >> > > > >> > > > > > > > >>> carefully.
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> I think maybe we can actually
> > > > extend
> > > > > > the
> > > > > > > > > >> > single-code
> > > > > > > > > >> > > > >> > > approach to
> > > > > > > > > >> > > > >> > > > > > > > overcome
> > > > > > > > > >> > > > >> > > > > > > > >>> its drawbacks, that is, wrap
> > the
> > > > > error
> > > > > > > > codes
> > > > > > > > > >> > > semantics
> > > > > > > > > >> > > > >> to
> > > > > > > > > >> > > > >> > the
> > > > > > > > > >> > > > >> > > > > users
> > > > > > > > > >> > > > >> > > > > > > so
> > > > > > > > > >> > > > >> > > > > > > > >>> that
> > > > > > > > > >> > > > >> > > > > > > > >>> users do not need to handle
> the
> > > > codes
> > > > > > > > > >> one-by-one.
> > > > > > > > > >> > > More
> > > > > > > > > >> > > > >> > > > > concretely,
> > > > > > > > > >> > > > >> > > > > > > > >>> following Jay's example the
> > client
> > > > > > could
> > > > > > > > > write
> > > > > > > > > >> > sth.
> > > > > > > > > >> > > > like
> > > > > > > > > >> > > > >> > > this:
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> if(error.isOK())
> > > > > > > > > >> > > > >> > > > > > > > >>> // status code is good or the
> > > > > code
> > > > > > > can
> > > > > > > > > be
> > > > > > > > > >> > > simply
> > > > > > > > > >> > > > >> > > ignored for
> > > > > > > > > >> > > > >> > > > > > > this
> > > > > > > > > >> > > > >> > > > > > > > >>> request type, process the
> > request
> > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.needsRetry())
> > > > > > > > > >> > > > >> > > > > > > > >>> // throttled, transient
> error,
> > > > > > etc:
> > > > > > > > > retry
> > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.isFatal())
> > > > > > > > > >> > > > >> > > > > > > > >>> // non-retriable errors, etc:
> > > > > > > notify /
> > > > > > > > > >> > > terminate
> > > > > > > > > >> > > > /
> > > > > > > > > >> > > > >> > other
> > > > > > > > > >> > > > >> > > > > > > handling
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> Only when the clients really
> > want
> > > > to
> > > > > > > > handle,
> > > > > > > > > for
> > > > > > > > > >> > > > example
> > > > > > > > > >> > > > >> > > > > > > > >>> FailDuetoThrottled
> > > > > > > > > >> > > > >> > > > > > > > >>> status code specifically, it
> > needs
> > > > > to:
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> if(error.isOK())
> > > > > > > > > >> > > > >> > > > > > > > >>> // status code is good or the
> > > > > code
> > > > > > > can
> > > > > > > > > be
> > > > > > > > > >> > > simply
> > > > > > > > > >> > > > >> > > ignored for
> > > > > > > > > >> > > > >> > > > > > > this
> > > > > > > > > >> > > > >> > > > > > > > >>> request type, process the
> > request
> > > > > > > > > >> > > > >> > > > > > > > >>> else if(error ==
> > > > > FailDuetoThrottled )
> > > > > > > > > >> > > > >> > > > > > > > >>> // throttled: log it
> > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.needsRetry())
> > > > > > > > > >> > > > >> > > > > > > > >>> // transient error, etc:
> retry
> > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.isFatal())
> > > > > > > > > >> > > > >> > > > > > > > >>> // non-retriable errors, etc:
> > > > > > > notify /
> > > > > > > > > >> > > terminate
> > > > > > > > > >> > > > /
> > > > > > > > > >> > > > >> > other
> > > > > > > > > >> > > > >> > > > > > > handling
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> And for implementation we can
> > > > > probably
> > > > > > > > group
> > > > > > > > > the
> > > > > > > > > >> > > codes
> > > > > > > > > >> > > > >> > > > > accordingly
> > > > > > > > > >> > > > >> > > > > > > like
> > > > > > > > > >> > > > >> > > > > > > > >>> HTTP status code such that we
> > can
> > > > do:
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> boolean Error.isOK() {
> > > > > > > > > >> > > > >> > > > > > > > >>> return code < 300 && code >=
> > 200;
> > > > > > > > > >> > > > >> > > > > > > > >>> }
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> Guozhang
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> On Mon, Mar 16, 2015 at 10:24
> > PM,
> > > > > Ewen
> > > > > > > > > >> > > > Cheslack-Postava
> > > > > > > > > >> > > > >> <
> > > > > > > > > >> > > > >> > > > > > > > >>> e...@confluent.io>
> > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> > Agreed that trying to
> > shoehorn
> > > > > > > non-error
> > > > > > > > > codes
> > > > > > > > > >> > > into
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> > > error
> > > > > > > > > >> > > > >> > > > > > field
> > > > > > > > > >> > > > >> > > > > > > > is
> > > > > > > > > >> > > > >> > > > > > > > >>> a
> > > > > > > > > >> > > > >> > > > > > > > >>> > bad idea. It makes it *way*
> > too
> > > > > easy
> > > > > > to
> > > > > > > > > write
> > > > > > > > > >> > code
> > > > > > > > > >> > > > >> that
> > > > > > > > > >> > > > >> > > looks
> > > > > > > > > >> > > > >> > > > > > (and
> > > > > > > > > >> > > > >> > > > > > > > >>> should
> > > > > > > > > >> > > > >> > > > > > > > >>> > be) correct but is actually
> > > > > > incorrect.
> > > > > > > If
> > > > > > > > > >> > > > necessary, I
> > > > > > > > > >> > > > >> > > think
> > > > > > > > > >> > > > >> > > > > it's
> > > > > > > > > >> > > > >> > > > > > > > much
> > > > > > > > > >> > > > >> > > > > > > > >>> > better to to spend a couple
> > of
> > > > > extra
> > > > > > > > bytes
> > > > > > > > > to
> > > > > > > > > >> > > encode
> > > > > > > > > >> > > > >> that
> > > > > > > > > >> > > > >> > > > > > > information
> > > > > > > > > >> > > > >> > > > > > > > >>> > separately (a "status" or
> > > > "warning"
> > > > > > > > > section of
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> > > response).
> > > > > > > > > >> > > > >> > > > > An
> > > > > > > > > >> > > > >> > > > > > > > >>> indication
> > > > > > > > > >> > > > >> > > > > > > > >>> > that throttling is
> occurring
> > is
> > > > > > > something
> > > > > > > > > I'd
> > > > > > > > > >> > > expect
> > > > > > > > > >> > > > >> to
> > > > > > > > > >> > > > >> > be
> > > > > > > > > >> > > > >> > > > > > > indicated
> > > > > > > > > >> > > > >> > > > > > > > >>> by a
> > > > > > > > > >> > > > >> > > > > > > > >>> > bit flag in the response
> > rather
> > > > > than
> > > > > > as
> > > > > > > > an
> > > > > > > > > >> error
> > > > > > > > > >> > > > code.
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > Gwen - I think an error
> code
> > > > makes
> > > > > > > sense
> > > > > > > > > when
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> request
> > > > > > > > > >> > > > >> > > > > > actually
> > > > > > > > > >> > > > >> > > > > > > > >>> failed.
> > > > > > > > > >> > > > >> > > > > > > > >>> > Option B, which Jun was
> > > > advocating,
> > > > > > > would
> > > > > > > > > have
> > > > > > > > > >> > > > >> appended
> > > > > > > > > >> > > > >> > the
> > > > > > > > > >> > > > >> > > > > > > messages
> > > > > > > > > >> > > > >> > > > > > > > >>> > successfully. If the
> > > > rate-limiting
> > > > > > case
> > > > > > > > > you're
> > > > > > > > > >> > > > talking
> > > > > > > > > >> > > > >> > > about
> > > > > > > > > >> > > > >> > > > > had
> > > > > > > > > >> > > > >> > > > > > > > >>> > successfully committed the
> > > > > messages,
> > > > > > I
> > > > > > > > > would
> > > > > > > > > >> say
> > > > > > > > > >> > > > >> that's
> > > > > > > > > >> > > > >> > > also a
> > > > > > > > > >> > > > >> > > > > > bad
> > > > > > > > > >> > > > >> > > > > > > > use
> > > > > > > > > >> > > > >> > > > > > > > >>> of
> > > > > > > > > >> > > > >> > > > > > > > >>> > error codes.
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > On Mon, Mar 16, 2015 at
> 10:16
> > > > PM,
> > > > > > Gwen
> > > > > > > > > >> Shapira <
> > > > > > > > > >> > > > >> > > > > > > > gshap...@cloudera.com>
> > > > > > > > > >> > > > >> > > > > > > > >>> > wrote:
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > We discussed an error
> code
> > for
> > > > > > > > > rate-limiting
> > > > > > > > > >> > > > (which
> > > > > > > > > >> > > > >> I
> > > > > > > > > >> > > > >> > > think
> > > > > > > > > >> > > > >> > > > > > made
> > > > > > > > > >> > > > >> > > > > > > > >>> > > sense), isn't it a
> similar
> > > > case?
> > > > > > > > > >> > > > >> > > > > > > > >>> > >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > On Mon, Mar 16, 2015 at
> > 10:10
> > > > PM,
> > > > > > Jay
> > > > > > > > > Kreps
> > > > > > > > > >> <
> > > > > > > > > >> > > > >> > > > > > jay.kr...@gmail.com
> > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > My concern is that as
> > soon
> > > > as
> > > > > you
> > > > > > > > start
> > > > > > > > > >> > > encoding
> > > > > > > > > >> > > > >> > > non-error
> > > > > > > > > >> > > > >> > > > > > > > response
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > information into error
> > codes
> > > > > the
> > > > > > > next
> > > > > > > > > >> > question
> > > > > > > > > >> > > > is
> > > > > > > > > >> > > > >> > what
> > > > > > > > > >> > > > >> > > to
> > > > > > > > > >> > > > >> > > > > do
> > > > > > > > > >> > > > >> > > > > > if
> > > > > > > > > >> > > > >> > > > > > > > two
> > > > > > > > > >> > > > >> > > > > > > > >>> > such
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > codes apply (i.e. you
> > have a
> > > > > > > replica
> > > > > > > > > down
> > > > > > > > > >> > and
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > >> > > response
> > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > >> > > > >> > > > > > > > >>> > quota'd). I
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > think I am trying to
> > argue
> > > > that
> > > > > > > error
> > > > > > > > > >> should
> > > > > > > > > >> > > > mean
> > > > > > > > > >> > > > >> > "why
> > > > > > > > > >> > > > >> > > we
> > > > > > > > > >> > > > >> > > > > > > failed
> > > > > > > > > >> > > > >> > > > > > > > >>> your
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > request", for which
> there
> > > > will
> > > > > > > really
> > > > > > > > > only
> > > > > > > > > >> > be
> > > > > > > > > >> > > > one
> > > > > > > > > >> > > > >> > > reason,
> > > > > > > > > >> > > > >> > > > > and
> > > > > > > > > >> > > > >> > > > > > > any
> > > > > > > > > >> > > > >> > > > > > > > >>> other
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > useful information we
> > want
> > > > to
> > > > > > send
> > > > > > > > > back is
> > > > > > > > > >> > > just
> > > > > > > > > >> > > > >> > another
> > > > > > > > > >> > > > >> > > > > field
> > > > > > > > > >> > > > >> > > > > > > in
> > > > > > > > > >> > > > >> > > > > > > > >>> the
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > response.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > -Jay
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > > On Mon, Mar 16, 2015 at
> > 9:51
> > > > > PM,
> > > > > > > Gwen
> > > > > > > > > >> > Shapira
> > > > > > > > > >> > > <
> > > > > > > > > >> > > > >> > > > > > > > >>> gshap...@cloudera.com>
> > > > > > > > > >> > > > >> > > > > > > > >>> > > wrote:
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> I think its not too
> > late to
> > > > > > > reserve
> > > > > > > > a
> > > > > > > > > set
> > > > > > > > > >> > of
> > > > > > > > > >> > > > >> error
> > > > > > > > > >> > > > >> > > codes
> > > > > > > > > >> > > > >> > > > > > > > >>> (200-299?)
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> for "non-error" codes.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> It won't be backward
> > > > > compatible
> > > > > > > > (i.e.
> > > > > > > > > >> > clients
> > > > > > > > > >> > > > >> that
> > > > > > > > > >> > > > >> > > > > currently
> > > > > > > > > >> > > > >> > > > > > > do
> > > > > > > > > >> > > > >> > > > > > > > >>> "else
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> throw" will throw on
> > > > > > non-errors),
> > > > > > > > but
> > > > > > > > > >> > perhaps
> > > > > > > > > >> > > > its
> > > > > > > > > >> > > > >> > > > > > worthwhile.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> On Mon, Mar 16, 2015
> at
> > > > 9:42
> > > > > PM,
> > > > > > > Jay
> > > > > > > > > >> Kreps
> > > > > > > > > >> > <
> > > > > > > > > >> > > > >> > > > > > > jay.kr...@gmail.com
> > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > >> > > > >> > > > > > > > >>> > wrote:
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Hey Jun,
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > I'd really really
> > really
> > > > > like
> > > > > > to
> > > > > > > > > avoid
> > > > > > > > > >> > > that.
> > > > > > > > > >> > > > >> > Having
> > > > > > > > > >> > > > >> > > just
> > > > > > > > > >> > > > >> > > > > > > > spent a
> > > > > > > > > >> > > > >> > > > > > > > >>> > > bunch of
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > time on the clients,
> > > > using
> > > > > the
> > > > > > > > error
> > > > > > > > > >> > codes
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > >> > encode
> > > > > > > > > >> > > > >> > > > > other
> > > > > > > > > >> > > > >> > > > > > > > >>> > information
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > about the response
> is
> > > > super
> > > > > > > > > dangerous.
> > > > > > > > > >> > The
> > > > > > > > > >> > > > >> error
> > > > > > > > > >> > > > >> > > > > handling
> > > > > > > > > >> > > > >> > > > > > is
> > > > > > > > > >> > > > >> > > > > > > > >>> one of
> > > > > > > > > >> > > > >> > > > > > > > >>> > > the
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > hardest parts of the
> > > > client
> > > > > > > > > (Guozhang
> > > > > > > > > >> > chime
> > > > > > > > > >> > > > in
> > > > > > > > > >> > > > >> > > here).
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Generally the error
> > > > handling
> > > > > > > looks
> > > > > > > > > like
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > if(error == none)
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // good, process the
> > > > > > > request
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else if(error ==
> > > > > > > KNOWN_ERROR_1)
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // handle known
> error
> > 1
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else if(error ==
> > > > > > > KNOWN_ERROR_2)
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // handle known
> error
> > 2
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > throw
> > > > > > > > > >> > > Errors.forCode(error).exception();
> > > > > > > > > >> > > > >> //
> > > > > > > > > >> > > > >> > or
> > > > > > > > > >> > > > >> > > some
> > > > > > > > > >> > > > >> > > > > > > other
> > > > > > > > > >> > > > >> > > > > > > > >>> > default
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > behavior
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > This works because
> we
> > > > have a
> > > > > > > > > convention
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > >> and
> > > > > > > > > >> > > > >> > > error
> > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > >> > > > >> > > > > > > > >>> something
> > > > > > > > > >> > > > >> > > > > > > > >>> > > that
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > prevented your
> getting
> > > > the
> > > > > > > > response
> > > > > > > > > so
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> default
> > > > > > > > > >> > > > >> > > > > > handling
> > > > > > > > > >> > > > >> > > > > > > > >>> case is
> > > > > > > > > >> > > > >> > > > > > > > >>> > > sane
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > and forward
> > compatible.
> > > > It
> > > > > is
> > > > > > > > > tempting
> > > > > > > > > >> to
> > > > > > > > > >> > > use
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> > > error
> > > > > > > > > >> > > > >> > > > > > code
> > > > > > > > > >> > > > >> > > > > > > > to
> > > > > > > > > >> > > > >> > > > > > > > >>> > convey
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > information in the
> > > > success
> > > > > > case.
> > > > > > > > For
> > > > > > > > > >> > > example
> > > > > > > > > >> > > > we
> > > > > > > > > >> > > > >> > > could
> > > > > > > > > >> > > > >> > > > > use
> > > > > > > > > >> > > > >> > > > > > > > error
> > > > > > > > > >> > > > >> > > > > > > > >>> > codes
> > > > > > > > > >> > > > >> > > > > > > > >>> > > to
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > encode whether
> quotas
> > > > were
> > > > > > > > enforced,
> > > > > > > > > >> > > whether
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> > > request
> > > > > > > > > >> > > > >> > > > > > was
> > > > > > > > > >> > > > >> > > > > > > > >>> served
> > > > > > > > > >> > > > >> > > > > > > > >>> > > out
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> of
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > cache, whether the
> > stock
> > > > > > market
> > > > > > > is
> > > > > > > > > up
> > > > > > > > > >> > > today,
> > > > > > > > > >> > > > or
> > > > > > > > > >> > > > >> > > > > whatever.
> > > > > > > > > >> > > > >> > > > > > > The
> > > > > > > > > >> > > > >> > > > > > > > >>> > problem
> > > > > > > > > >> > > > >> > > > > > > > >>> > > is
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > that since these are
> > not
> > > > > > errors
> > > > > > > as
> > > > > > > > > far
> > > > > > > > > >> as
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > >> > > client is
> > > > > > > > > >> > > > >> > > > > > > > >>> concerned it
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> should
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > not throw an
> exception
> > > > but
> > > > > > > process
> > > > > > > > > the
> > > > > > > > > >> > > > >> response,
> > > > > > > > > >> > > > >> > > but now
> > > > > > > > > >> > > > >> > > > > > we
> > > > > > > > > >> > > > >> > > > > > > > >>> created
> > > > > > > > > >> > > > >> > > > > > > > >>> > an
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > explicit requirement
> > that
> > > > > that
> > > > > > > > > error be
> > > > > > > > > >> > > > handled
> > > > > > > > > >> > > > >> > > > > explicitly
> > > > > > > > > >> > > > >> > > > > > > > >>> since it
> > > > > > > > > >> > > > >> > > > > > > > >>> > is
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > different. I really
> > think
> > > > > that
> > > > > > > > this
> > > > > > > > > >> kind
> > > > > > > > > >> > of
> > > > > > > > > >> > > > >> > > information
> > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > >> > > > >> > > > > > > not
> > > > > > > > > >> > > > >> > > > > > > > >>> an
> > > > > > > > > >> > > > >> > > > > > > > >>> > > error,
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> it
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > is just information,
> > and
> > > > if
> > > > > we
> > > > > > > > want
> > > > > > > > > it
> > > > > > > > > >> in
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > >> > > response
> > > > > > > > > >> > > > >> > > > > we
> > > > > > > > > >> > > > >> > > > > > > > >>> should do
> > > > > > > > > >> > > > >> > > > > > > > >>> > > the
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > right thing and add
> a
> > new
> > > > > > field
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > >> > > > >> response.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > I think you saw the
> > Samza
> > > > > bug
> > > > > > > that
> > > > > > > > > was
> > > > > > > > > >> > > > >> literally
> > > > > > > > > >> > > > >> > an
> > > > > > > > > >> > > > >> > > > > > example
> > > > > > > > > >> > > > >> > > > > > > of
> > > > > > > > > >> > > > >> > > > > > > > >>> this
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > happening and
> leading
> > to
> > > > an
> > > > > > > > infinite
> > > > > > > > > >> > retry
> > > > > > > > > >> > > > >> loop.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Further more I
> really
> > > > want
> > > > > to
> > > > > > > > > emphasize
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > >> > hitting
> > > > > > > > > >> > > > >> > > > > your
> > > > > > > > > >> > > > >> > > > > > > > quota
> > > > > > > > > >> > > > >> > > > > > > > >>> in
> > > > > > > > > >> > > > >> > > > > > > > >>> > the
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > design that Adi has
> > > > proposed
> > > > > > is
> > > > > > > > > >> actually
> > > > > > > > > >> > > not
> > > > > > > > > >> > > > an
> > > > > > > > > >> > > > >> > > error
> > > > > > > > > >> > > > >> > > > > > > > condition
> > > > > > > > > >> > > > >> > > > > > > > >>> at
> > > > > > > > > >> > > > >> > > > > > > > >>> > > all.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> It
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > is totally
> reasonable
> > in
> > > > any
> > > > > > > > > bootstrap
> > > > > > > > > >> > > > >> situation
> > > > > > > > > >> > > > >> > to
> > > > > > > > > >> > > > >> > > > > > > > >>> intentionally
> > > > > > > > > >> > > > >> > > > > > > > >>> > > want to
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > run at the limit the
> > > > system
> > > > > > > > imposes
> > > > > > > > > on
> > > > > > > > > >> > you.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > -Jay
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > On Mon, Mar 16, 2015
> > at
> > > > 4:27
> > > > > > PM,
> > > > > > > > Jun
> > > > > > > > > >> Rao
> > > > > > > > > >> > <
> > > > > > > > > >> > > > >> > > > > > j...@confluent.io>
> > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> It's probably
> useful
> > for
> > > > a
> > > > > > > client
> > > > > > > > > to
> > > > > > > > > >> > know
> > > > > > > > > >> > > > >> whether
> > > > > > > > > >> > > > >> > > its
> > > > > > > > > >> > > > >> > > > > > > > requests
> > > > > > > > > >> > > > >> > > > > > > > >>> are
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> throttled or not
> > (e.g.,
> > > > for
> > > > > > > > > monitoring
> > > > > > > > > >> > and
> > > > > > > > > >> > > > >> > > alerting).
> > > > > > > > > >> > > > >> > > > > > From
> > > > > > > > > >> > > > >> > > > > > > > that
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> perspective,
> option B
> > > > > (delay
> > > > > > > the
> > > > > > > > > >> > requests
> > > > > > > > > >> > > > and
> > > > > > > > > >> > > > >> > > return an
> > > > > > > > > >> > > > >> > > > > > > > error)
> > > > > > > > > >> > > > >> > > > > > > > >>> > seems
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> better.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> Thanks,
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> Jun
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> On Wed, Mar 4, 2015
> > at
> > > > 3:51
> > > > > > PM,
> > > > > > > > > Aditya
> > > > > > > > > >> > > > >> Auradkar <
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > aaurad...@linkedin.com.invalid
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Posted a KIP for
> > > > quotas
> > > > > in
> > > > > > > > kafka.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >>
> > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Appreciate any
> > > > feedback.
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Aditya
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > >> > > > >> > > > > > > > >>> > >
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > > > > >>> > --
> > > > > > > > > >> > > > >> > > > > > > > >>> > Thanks,
> > > > > > > > > >> > > > >> > > > > > > > >>> > Ewen
> > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>> --
> > > > > > > > > >> > > > >> > > > > > > > >>> -- Guozhang
> > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > --
> > > > > > > > > >> > > > > Sent from Gmail Mobile
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > --
> > > > > > > > > >> > > > Sent from Gmail Mobile
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> >
> >
>



-- 
-- Guozhang