Re: [DISCUSS] KIP-55: Secure quotas for authenticated users

[Aditya Auradkar]

Hey Rajini -

If the quota.type is set to 'user', what happens to unauthenticated
clients? They don't supply a principal, so are they essentially
unthrottled?

This may be a nit, but I prefer 'quota.type' options to be
'authenticated-user' and 'client-id' as opposed to 'client' and 'user'. For
a new user, the options 'client' and 'user' sound essentially the same.

Aditya

On Tue, May 24, 2016 at 5:55 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jun,
>
> I have updated the KIP based on your suggestion. Can you take a look?
>
> Thank you,
>
> Rajini
>
> On Tue, May 24, 2016 at 11:20 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Jun,
> >
> > Thank you for the review. I agree that a simple user principal based
> quota
> > is sufficient to allocate broker resources fairly in a multi-user system.
> > Hierarchical quotas proposed in the KIP currently enables clients of a
> user
> > to be rate-limited as well. This allows a user to run multiple clients
> > which don't interfere with each other's quotas. Since there is no clear
> > requirement to support this at the moment, I am happy to limit the scope
> of
> > the KIP to a single-level user-based quota. Will update the KIP today.
> >
> > Regards,
> >
> > Rajini
> >
> > On Mon, May 23, 2016 at 5:24 PM, Jun Rao  wrote:
> >
> >> Rajini,
> >>
> >> Thanks for the KIP. When we first added the quota support, the intention
> >> was to be able to add a quota per application. Since at that time, we
> >> don't
> >> have security yet. We essentially simulated users with client-ids. Now
> >> that
> >> we do have security. It seems that we just need to have a way to set
> quota
> >> at the user level. Setting quota at the combination of users and
> >> client-ids
> >> seems more complicated and I am not sure if there is a good use case.
> >>
> >> Also, the new config quota.secure.enable seems a bit weird. Would it be
> >> better to add a new config quota.type. It defaults to clientId for
> >> backward
> >> compatibility. If one sets it to user, then the default broker level
> quota
> >> is for users w/o a customized quota. In this setting, brokers will also
> >> only take quota set at the user level (i.e., quota set at clientId level
> >> will be ignored).
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Tue, May 3, 2016 at 4:32 AM, Rajini Sivaram <
> >> rajinisiva...@googlemail.com
> >> > wrote:
> >>
> >> > Ewen,
> >> >
> >> > Thank you for the review. I agree that ideally we would have one
> >> definition
> >> > of quotas that handles all cases. But I couldn't quite fit all the
> >> > combinations that are possible today with client-id-based quotas into
> >> the
> >> > new configuration. I think upgrade path is not bad since quotas are
> >> > per-broker. You can configure quotas based on the new configuration,
> set
> >> > quota.secure.enable=true and restart the broker. Since there is no
> >> > requirement for both insecure client-id based quotas and secure
> >> user-based
> >> > quotas to co-exist in a cluster, isn't that sufficient? The
> >> implementation
> >> > does use a unified approach, so if an alternative configuration can be
> >> > defined (perhaps with some acceptable limitations?) which can express
> >> both,
> >> > it will be easy to implement. Suggestions welcome :-)
> >> >
> >> > The cases that the new configuration cannot express, but the old one
> can
> >> > are:
> >> >
> >> >1. SSL/SASL with multiple users, same client ids used by multiple
> >> users,
> >> >client-id based quotas where quotas are shared between multiple
> users
> >> >2. Default quotas for client-ids. In the new configuration, default
> >> >quotas are defined for users and clients with no configured
> sub-quota
> >> > share
> >> >the user's quota.
> >> >
> >> >
> >> >
> >> > On Sat, Apr 30, 2016 at 6:21 AM, Ewen Cheslack-Postava <
> >> e...@confluent.io>
> >> > wrote:
> >> >
> >> > > Rajini,
> >> > >
> >> > > I'm admittedly not very familiar with a lot of this code or
> >> > implementation,
> >> > > so correct me if I'm making any incorrect assumptions.
> >> > >
> >> > > I've only scanned the KIP, but my main concern is the rejection of
> the
> >> > > alternative -- unifying client-id and principal quotas. In
> particular,
> >> > > doesn't this make an upgrade for brokers using those different
> >> approaches
> >> > > difficult since you have to make a hard break between client-id and
> >> > > principal quotas? If people adopt client-id quotas to begin with, it
> >> > seems
> >> > > like we might not be providing a clean upgrade path.
> >> > >
> >> > > As I said, I haven't kept up to date with the details of the
> security
> >> and
> >> > > quota features, but I'd want to make sure we didn't suggest one path
> >> with
> >> > > 0.9, then add another that we can't provide a clean upgrade path to.
> >> > >
> >> > > -Ewen
> >> > >
> >> > > On Fri, Apr 22, 2016 at 7:22 AM, Rajini Sivaram <
> >> > > rajinisiva...@googlemail.com> wrote:
> >> > >

RE: Metrics package discussion

[Aditya Auradkar]

quite a few memory management and GC issues while
> using histograms. In comparison, histograms in the new metrics package
> aren't very well tested.
>
> Thanks,
> Neha
>
> On Wed, Mar 25, 2015 at 8:25 AM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Hey everyone,
> >
> > Picking up this discussion after yesterdays KIP hangout. For anyone who
> > did not join the meeting, we have 2 different metrics packages being used
> > by the clients (custom package) and the server (codahale). We are
> > discussing whether to migrate the server to the new package.
> >
> > What information do we need in order to make a decision?
> >
> > Some pros of the new package:
> > - Using the most recent information by combining data from previous and
> > current samples. I'm not sure how codahale does this so I'll investigate.
> > - We can quota on anything we measure. This is pretty cool IMO. I've
> > investigate the feasibility of adding this feature in codahale.
> > - Hierarchical metrics. For example: we can define a sensor for overall
> > bytes-in/bytes-out and also per-client. Updating the client sensor will
> > cause the global byte rate sensor to get modified too.
> >
> > What are some of the issues with codahale? One previous discussion
> > mentions high memory usage but I don't have any experience with it
> myself.
> >
> > Thanks,
> > Aditya
> >
> >
> >
> >
> >
>
>
> --
> Thanks,
> Neha
>

Re: Review Request 31967: Patch for KAFKA-1546

[Aditya Auradkar]

> On March 26, 2015, 12:07 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/cluster/Replica.scala, lines 52-72
> > <https://reviews.apache.org/r/31967/diff/6/?file=905875#file905875line52>
> >
> > Would it be simpler to instead keeping track of a lastCaughtUpTime and 
> > updating it every time readToEndOfEnd is true? If readToEndOfEnd is false, 
> > we will just leave that value untouched.

Good suggestion. I'll update the patch.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31967/#review77810
-------


On March 25, 2015, 8:27 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31967/
> ---
> 
> (Updated March 25, 2015, 8:27 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1546
> https://issues.apache.org/jira/browse/KAFKA-1546
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> PATCH for KAFKA-1546
> 
> 
> PATCH for KAFKA-1546
> 
> Brief summary of changes:
> - Added a lagBegin metric inside Replica to track the lag in terms of time 
> since the replica did not read from the LEO
> - Using lag begin value in the check for ISR expand and shrink
> - Removed the max lag messages config since it is no longer necessary
> - Returning the initialLogEndOffset in LogReadResult corresponding to the the 
> LEO before actually reading from the log.
> - Unit test cases to test ISR shrinkage and expansion
> 
> Updated KAFKA-1546 patch to reflect Neha and Jun's comments
> 
> 
> Addressing Joel's comments
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> c4bf48a801007ebe7497077d2018d6dffe1677d4 
>   core/src/main/scala/kafka/cluster/Replica.scala 
> bd13c20338ce3d73113224440e858a12814e5adb 
>   core/src/main/scala/kafka/log/Log.scala 
> 06b8ecc5d11a1acfbaf3c693c42bf3ce5b2cd86d 
>   core/src/main/scala/kafka/server/FetchDataInfo.scala 
> 26f278f9b75b1c9c83a720ca9ebd8ab812d19d39 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 46d21c73f1feb3410751899380b35da0c37c975c 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> c5274822c57bf3c1f9e4135c0bdcaa87ee50ce20 
>   core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
> 92152358c95fa9178d71bd1c079af0a0bd8f1da8 
>   core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
> 191251d1340b5e5b2d649b37af3c6c1896d07e6e 
>   core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
> 92d6b2c672f74cdd526f2e98da8f7fb3696a88e3 
>   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
> efb457334bd87e4c5b0dd2c66ae1995993cd0bc1 
> 
> Diff: https://reviews.apache.org/r/31967/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: Review Request 31967: Patch for KAFKA-1546

[Aditya Auradkar]

> On March 25, 2015, 9:25 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/cluster/Replica.scala, lines 60-72
> > <https://reviews.apache.org/r/31967/diff/6/?file=905875#file905875line60>
> >
> > Instead of pass these fiels of logReadResult all the way up to the 
> > replica, I think we can just push this comparison logic down to 
> > replicaManager.readFromLocalLog around line 507, and just keep a boolean in 
> > the logReadResult like isReadFromLogEnd. That value can then be passed all 
> > the way up to Replica as sth. like isLEOCaughtUp.
> > 
> > Also we can change the names of updateReplicaLEO and updateFollowerLEOs 
> > as they now carries the isLEOCaughtUp information as well as LEOs.

I initially started with a boolean and later decided to pass the 
LogOffsetMetadata directly. I agree that it is a bit simpler to pass the 
boolean value so I'm making these changes.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31967/#review77799
---


On March 25, 2015, 8:27 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31967/
> ---
> 
> (Updated March 25, 2015, 8:27 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1546
> https://issues.apache.org/jira/browse/KAFKA-1546
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> PATCH for KAFKA-1546
> 
> 
> PATCH for KAFKA-1546
> 
> Brief summary of changes:
> - Added a lagBegin metric inside Replica to track the lag in terms of time 
> since the replica did not read from the LEO
> - Using lag begin value in the check for ISR expand and shrink
> - Removed the max lag messages config since it is no longer necessary
> - Returning the initialLogEndOffset in LogReadResult corresponding to the the 
> LEO before actually reading from the log.
> - Unit test cases to test ISR shrinkage and expansion
> 
> Updated KAFKA-1546 patch to reflect Neha and Jun's comments
> 
> 
> Addressing Joel's comments
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> c4bf48a801007ebe7497077d2018d6dffe1677d4 
>   core/src/main/scala/kafka/cluster/Replica.scala 
> bd13c20338ce3d73113224440e858a12814e5adb 
>   core/src/main/scala/kafka/log/Log.scala 
> 06b8ecc5d11a1acfbaf3c693c42bf3ce5b2cd86d 
>   core/src/main/scala/kafka/server/FetchDataInfo.scala 
> 26f278f9b75b1c9c83a720ca9ebd8ab812d19d39 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 46d21c73f1feb3410751899380b35da0c37c975c 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> c5274822c57bf3c1f9e4135c0bdcaa87ee50ce20 
>   core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
> 92152358c95fa9178d71bd1c079af0a0bd8f1da8 
>   core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
> 191251d1340b5e5b2d649b37af3c6c1896d07e6e 
>   core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
> 92d6b2c672f74cdd526f2e98da8f7fb3696a88e3 
>   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
> efb457334bd87e4c5b0dd2c66ae1995993cd0bc1 
> 
> Diff: https://reviews.apache.org/r/31967/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: Review Request 31967: Patch for KAFKA-1546

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31967/
---

(Updated March 27, 2015, 12:44 a.m.)


Review request for kafka.


Bugs: KAFKA-1546
https://issues.apache.org/jira/browse/KAFKA-1546


Repository: kafka


Description (updated)
---

PATCH for KAFKA-1546


PATCH for KAFKA-1546

Brief summary of changes:
- Added a lagBegin metric inside Replica to track the lag in terms of time 
since the replica did not read from the LEO
- Using lag begin value in the check for ISR expand and shrink
- Removed the max lag messages config since it is no longer necessary
- Returning the initialLogEndOffset in LogReadResult corresponding to the the 
LEO before actually reading from the log.
- Unit test cases to test ISR shrinkage and expansion

Updated KAFKA-1546 patch to reflect Neha and Jun's comments


Addressing Joel's comments


Addressing Jun and Guozhang's comments


Diffs (updated)
-

  core/src/main/scala/kafka/cluster/Partition.scala 
c4bf48a801007ebe7497077d2018d6dffe1677d4 
  core/src/main/scala/kafka/cluster/Replica.scala 
bd13c20338ce3d73113224440e858a12814e5adb 
  core/src/main/scala/kafka/log/Log.scala 
06b8ecc5d11a1acfbaf3c693c42bf3ce5b2cd86d 
  core/src/main/scala/kafka/server/FetchDataInfo.scala 
26f278f9b75b1c9c83a720ca9ebd8ab812d19d39 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
46d21c73f1feb3410751899380b35da0c37c975c 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
c5274822c57bf3c1f9e4135c0bdcaa87ee50ce20 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
92152358c95fa9178d71bd1c079af0a0bd8f1da8 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
191251d1340b5e5b2d649b37af3c6c1896d07e6e 
  core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
92d6b2c672f74cdd526f2e98da8f7fb3696a88e3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
efb457334bd87e4c5b0dd2c66ae1995993cd0bc1 

Diff: https://reviews.apache.org/r/31967/diff/


Testing
---


Thanks,

Aditya Auradkar

RE: Metrics package discussion

[Aditya Auradkar]

Adding to what Jay said. 

The library maintains 1k samples by default. The UniformSample has a long array 
so about 8k overhead per histogram. The ExponentiallyDecayingSample (which is 
what we use) has a 16 byte overhead per stored sample, so about 16k per 
histogram. So 10k histograms (worst case? metrics per partition and client) is 
about 160MB of memory in the broker.

Copying is also a problem. For  percentiles on HistogramMBean, the 
implementation does a copy of the entire array. For e.g., if we called 
get50Percentile() and get75Percentile(), the entire array would get copied 
twice which is pretty bad if we called each metric on every MBean.

Another point Joel mentioned is that codahale metrics are harder to write tests 
against because we cannot pass in a Clock. 

IMO, if a library is preventing us from adding all the metrics that we want to 
add and we have a viable alternative, we should replace it. It might be short 
term pain but in the long run we will have more useful graphs. 
What do people think? I can start a vote thread on this once we have a couple 
more opinions.

Thanks,
Aditya

From: Jay Kreps [jay.kr...@gmail.com]
Sent: Thursday, March 26, 2015 2:29 PM
To: dev@kafka.apache.org
Subject: Re: Metrics package discussion

Yeah that is a good summary.

The reason we don't use histograms heavily in the server is because of the
memory issues. We originally did use histograms for everything, then we ran
into all these issues, and ripped them out. Whether they are really useful
or not, I don't know. Averages can be pretty misleading so it can be nice
but I don't know that it is critical.

-Jay

On Thu, Mar 26, 2015 at 1:58 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> From what I can tell, Histograms don't seem to be used extensively in the
> Kafka server (only in RequestChannel.scala) and I'm not sure we need them
> for per-client metrics. Topic metrics use meters currently.  Migrating
> graphing, alerting will be quite a significant effort for all users of
> Kafka. Do the potential benefits of the new metrics package outweigh this
> one time migration? In the long run it seems nice to have a unified metrics
> package across clients and server. If we were starting out from scratch
> without any existing deployments, what decision would we take?
>
> I suppose the relative effort in supporting is a useful data point in this
> discussion. We need to throttle based on the current byte rate which should
> be a "Meter" in codahale terms. The Meter implementation uses a 1, 5 and 15
> minute exponential window moving average. The library also does not use the
> most recent samples of data for Metered metrics. For calculating rates, the
> EWMA class has a scheduled task that runs every 5 seconds and adjusts the
> rate using the new data accordingly. In that particular case, I think the
> new library is superior since it is more responsive.  If we do choose to
> remain with Yammer on the server, here are a few ideas on how to support
> quotas with relatively less effort.
>
> - We could have a new type of Meter called "QuotaMeter" that can wrap the
> existing meter code that follows the same pattern that the Sensor does in
> the new metrics library. This QuotaMeter needs to be configured with a
> Quota and it can have a finer grained rate than 1 minute (10 seconds?
> configurable?). Anytime we call "mark()", it update the underlying rates
> and throw a QuotaViolationException if required. This class can either
> extend Meter or be a separate implementation of the Metric superclass that
> every metric implements.
>
> - We can also consider implementing these quotas with the new metrics
> package and have these co-exist with the existing metrics. This leads to 2
> metric packages being used on the server, but they are both pulled in as
> dependencies anyway. Using this for metrics we can quota on may not be a
> bad place to start.
>
> Thanks,
> Aditya
> 
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Wednesday, March 25, 2015 11:08 PM
> To: dev@kafka.apache.org
> Subject: Re: Metrics package discussion
>
> Here was my understanding of the issue last time.
>
> The yammer metrics use a random sample of requests to estimate the
> histogram. This allocates a fairly large array of longs (their values are
> longs rather than floats). A reasonable sample might be 8k entries which
> would give about 64KB per histogram. There are bounds on accuracy, but they
> are only probabilistic. I.e. if you try to get 99% < 5 ms of inaccuracy,
> you will 1% of the time get more than this. This is okay but if you try to
> alert, in which you realize that being wrong 1% of the time is a lot if you
> are co

Re: Review Request 31967: Patch for KAFKA-1546

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31967/
---

(Updated March 27, 2015, 6:58 p.m.)


Review request for kafka.


Bugs: KAFKA-1546
https://issues.apache.org/jira/browse/KAFKA-1546


Repository: kafka


Description (updated)
---

PATCH for KAFKA-1546


PATCH for KAFKA-1546

Brief summary of changes:
- Added a lagBegin metric inside Replica to track the lag in terms of time 
since the replica did not read from the LEO
- Using lag begin value in the check for ISR expand and shrink
- Removed the max lag messages config since it is no longer necessary
- Returning the initialLogEndOffset in LogReadResult corresponding to the the 
LEO before actually reading from the log.
- Unit test cases to test ISR shrinkage and expansion

Updated KAFKA-1546 patch to reflect Neha and Jun's comments


Addressing Joel's comments


Addressing Jun and Guozhang's comments


Addressing Jun's comments


Diffs (updated)
-

  core/src/main/scala/kafka/cluster/Partition.scala 
c4bf48a801007ebe7497077d2018d6dffe1677d4 
  core/src/main/scala/kafka/cluster/Replica.scala 
bd13c20338ce3d73113224440e858a12814e5adb 
  core/src/main/scala/kafka/log/Log.scala 
06b8ecc5d11a1acfbaf3c693c42bf3ce5b2cd86d 
  core/src/main/scala/kafka/server/FetchDataInfo.scala 
26f278f9b75b1c9c83a720ca9ebd8ab812d19d39 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
422451aec5ea0442eb2e4c1ae772885b813904a9 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
44f0026e6676d8d764dd59dbcc6bb7bb727a3ba6 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
92152358c95fa9178d71bd1c079af0a0bd8f1da8 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
191251d1340b5e5b2d649b37af3c6c1896d07e6e 
  core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
92d6b2c672f74cdd526f2e98da8f7fb3696a88e3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
efb457334bd87e4c5b0dd2c66ae1995993cd0bc1 

Diff: https://reviews.apache.org/r/31967/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 31366: Patch for KAFKA-1461

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31366/#review78120
---


Couple of comments. Thanks!


core/src/main/scala/kafka/server/AbstractFetcherThread.scala
<https://reviews.apache.org/r/31366/#comment126590>

Can you add a comment explaning why this flag is used? It only seems to be 
used in doWork() so perhaps it shouldn't be a member variable?



core/src/main/scala/kafka/server/AbstractFetcherThread.scala
<https://reviews.apache.org/r/31366/#comment126589>

I guess you don't need to set this to false if it already is.



core/src/main/scala/kafka/server/AbstractFetcherThread.scala
<https://reviews.apache.org/r/31366/#comment126594>

Perhaps a trace log here before awaiting?



core/src/main/scala/kafka/server/AbstractFetcherThread.scala
<https://reviews.apache.org/r/31366/#comment126602>

Can you add some documentation for this class?


- Aditya Auradkar


On March 27, 2015, 10:31 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31366/
> ---
> 
> (Updated March 27, 2015, 10:31 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1461
> https://issues.apache.org/jira/browse/KAFKA-1461
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1461. Replica fetcher thread does not implement any back-off behavior.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> 5d5cf5897cc37b3595f14bfe9d7cde43456bcc4b 
>   core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
> 96faa7b4ed7c9ba8a3f6f9f114bd94e19b3a7ac0 
> 
> Diff: https://reviews.apache.org/r/31366/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

RE: Metrics package discussion

[Aditya Auradkar]

 them to be an extreme necessity for
> > per-clientid/per-partition metrics and they are more critical for
> > aggregate (global) metrics.
> >
> > So it seems the main benefits of switching to KM metrics are:
> > - Less exposure to YM metrics changes
> > - More control over the actual implementation. E.g., there is
> >   considerable research on implementing approximate-but-good-enough
> >   histograms/percentiles that we can try out
> > - Differences (improvements) from YM metrics such as:
> >   - hierarchical sensors
> >   - integrated with quota enforcement
> >   - mbeans can logically group attributes computed from different
> > sensors. So there is logical grouping (as opposed to a separate
> > mbean per sensor as is the case in YM metrics).
> >
> > The main disadvantages:
> > - Everyone's graphs and alerts will break and need to be updated
> > - Histogram support needs to be tested more/improved
> >
> > The first disadvantage is a big one but we aren't exactly immune to
> > that if we stick with YM.
> >
> > BTW with KM metrics we should also provide reporters (graphite,
> > ganglia) but we probably need to do this anyway since the new clients
> > are on KM metrics.
> >
> > Thanks,
> >
> > Joel
> >
> > On Fri, Mar 27, 2015 at 06:48:48PM +, Aditya Auradkar wrote:
> > > Adding to what Jay said.
> > >
> > > The library maintains 1k samples by default. The UniformSample has a
> > long array so about 8k overhead per histogram. The
> > ExponentiallyDecayingSample (which is what we use) has a 16 byte overhead
> > per stored sample, so about 16k per histogram. So 10k histograms (worst
> > case? metrics per partition and client) is about 160MB of memory in the
> > broker.
> > >
> > > Copying is also a problem. For  percentiles on HistogramMBean, the
> > implementation does a copy of the entire array. For e.g., if we called
> > get50Percentile() and get75Percentile(), the entire array would get
> copied
> > twice which is pretty bad if we called each metric on every MBean.
> > >
> > > Another point Joel mentioned is that codahale metrics are harder to
> > write tests against because we cannot pass in a Clock.
> > >
> > > IMO, if a library is preventing us from adding all the metrics that we
> > want to add and we have a viable alternative, we should replace it. It
> > might be short term pain but in the long run we will have more useful
> > graphs.
> > > What do people think? I can start a vote thread on this once we have a
> > couple more opinions.
> > >
> > > Thanks,
> > > Aditya
> > > 
> > > From: Jay Kreps [jay.kr...@gmail.com]
> > > Sent: Thursday, March 26, 2015 2:29 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: Metrics package discussion
> > >
> > > Yeah that is a good summary.
> > >
> > > The reason we don't use histograms heavily in the server is because of
> > the
> > > memory issues. We originally did use histograms for everything, then we
> > ran
> > > into all these issues, and ripped them out. Whether they are really
> > useful
> > > or not, I don't know. Averages can be pretty misleading so it can be
> nice
> > > but I don't know that it is critical.
> > >
> > > -Jay
> > >
> > > On Thu, Mar 26, 2015 at 1:58 PM, Aditya Auradkar <
> > > aaurad...@linkedin.com.invalid> wrote:
> > >
> > > > From what I can tell, Histograms don't seem to be used extensively in
> > the
> > > > Kafka server (only in RequestChannel.scala) and I'm not sure we need
> > them
> > > > for per-client metrics. Topic metrics use meters currently.
> Migrating
> > > > graphing, alerting will be quite a significant effort for all users
> of
> > > > Kafka. Do the potential benefits of the new metrics package outweigh
> > this
> > > > one time migration? In the long run it seems nice to have a unified
> > metrics
> > > > package across clients and server. If we were starting out from
> scratch
> > > > without any existing deployments, what decision would we take?
> > > >
> > > > I suppose the relative effort in supporting is a useful data point in
> > this
> > > > discussion. We need to throttle based on the current byte rate which
> > should
> > > > be a "Meter" in codahale t

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Hey everyone,

I've made changes to the KIP to capture our discussions over the last couple of 
weeks.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas

I'll start a voting thread after people have had a chance to read/comment.

Thanks,
Aditya


From: Steven Wu [stevenz...@gmail.com]
Sent: Friday, March 20, 2015 9:14 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

+1 on Jun's suggestion of maintaining one set/style of metrics at broker.
In Netflix, we have to convert the yammer metrics to servo metrics at
broker. it will be painful to know some metrics are in a different style
and get to be handled differently.

On Fri, Mar 20, 2015 at 8:17 AM, Jun Rao  wrote:

> Not so sure. People who use quota will definitely want to monitor the new
> metrics at the client id level. Then they will need to deal with those
> metrics differently from the rest of the metrics. It would be better if we
> can hide this complexity from the users.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 19, 2015 at 10:45 PM, Joel Koshy  wrote:
>
> > Actually thinking again - since these will be a few new metrics at the
> > client id level (bytes in and bytes out to start with) maybe it is fine
> to
> > have the two type of metrics coexist and we can migrate the existing
> > metrics in parallel.
> >
> > On Thursday, March 19, 2015, Joel Koshy  wrote:
> >
> > > That is a valid concern but in that case I think it would be better to
> > > just migrate completely to the new metrics package first.
> > >
> > > On Thursday, March 19, 2015, Jun Rao  > > > wrote:
> > >
> > >> Hmm, I was thinking a bit differently on the metrics stuff. I think it
> > >> would be confusing to have some metrics defined in the new metrics
> > package
> > >> while some others defined in Coda Hale. Those metrics will look
> > different
> > >> (e.g., rates in Coda Hale will have special attributes such as
> > >> 1-min-average). People may need different ways to export the metrics
> to
> > >> external systems such as Graphite. So, instead of using the new
> metrics
> > >> package on the broker, I was thinking that we can just implement a
> > >> QuotaMetrics that wraps the Coda Hale metrics. The implementation can
> be
> > >> the same as what's in the new metrics package.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Thu, Mar 19, 2015 at 8:09 PM, Jay Kreps 
> wrote:
> > >>
> > >> > Yeah I was saying was that we are blocked on picking an approach for
> > >> > metrics but not necessarily the full conversion. Clearly if we pick
> > the
> > >> new
> > >> > metrics package we would need to implement the two metrics we want
> to
> > >> quota
> > >> > on. But the conversion of the remaining metrics can be done
> > >> asynchronously.
> > >> >
> > >> > -Jay
> > >> >
> > >> > On Thu, Mar 19, 2015 at 5:56 PM, Joel Koshy 
> > >> wrote:
> > >> >
> > >> > > > in KAFKA-1930). I agree that this KIP doesn't need to block on
> the
> > >> > > > migration of the metrics package.
> > >> > >
> > >> > > Can you clarify the above? i.e., if we are going to quota on
> > something
> > >> > > then we would want to have migrated that metric over right? Or do
> > you
> > >> > > mean we don't need to complete the migration of all metrics to the
> > >> > > metrics package right?
> > >> > >
> > >> > > I think most of us now feel that the delay + no error is a good
> > >> > > approach, but it would be good to make sure everyone is on the
> same
> > >> > > page.
> > >> > >
> > >> > > As Aditya requested a couple of days ago I think we should go over
> > >> > > this at the next KIP hangout.
> > >> > >
> > >> > > Joel
> > >> > >
> > >> > > On Thu, Mar 19, 2015 at 09:24:09AM -0700, Jun Rao wrote:
> > >> > > > 1. Delay + no error seems reasonable to me. However, I do feel
> > that
> > >> we
> > >> > > need
> > >> > > > to give the client an indicator that it's being throttled,
> instead
> > >> of
> > >> > > doing
>

Review Request 32778: Patch for KAFKA-2087

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32778/
---

Review request for kafka.


Bugs: KAFKA-2087
https://issues.apache.org/jira/browse/KAFKA-2087


Repository: kafka


Description
---

PATCH for KAFKA-1546


PATCH for KAFKA-1546

Brief summary of changes:
- Added a lagBegin metric inside Replica to track the lag in terms of time 
since the replica did not read from the LEO
- Using lag begin value in the check for ISR expand and shrink
- Removed the max lag messages config since it is no longer necessary
- Returning the initialLogEndOffset in LogReadResult corresponding to the the 
LEO before actually reading from the log.
- Unit test cases to test ISR shrinkage and expansion

Updated KAFKA-1546 patch to reflect Neha and Jun's comments


Addressing Joel's comments


Addressing Jun and Guozhang's comments


Addressing Jun's comments


Cleaning up javadoc for TopicConfigManager


Diffs
-

  core/src/main/scala/kafka/cluster/Partition.scala 
c4bf48a801007ebe7497077d2018d6dffe1677d4 
  core/src/main/scala/kafka/cluster/Replica.scala 
bd13c20338ce3d73113224440e858a12814e5adb 
  core/src/main/scala/kafka/log/Log.scala 
06b8ecc5d11a1acfbaf3c693c42bf3ce5b2cd86d 
  core/src/main/scala/kafka/server/FetchDataInfo.scala 
26f278f9b75b1c9c83a720ca9ebd8ab812d19d39 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
422451aec5ea0442eb2e4c1ae772885b813904a9 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
44f0026e6676d8d764dd59dbcc6bb7bb727a3ba6 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
47295d40131492aaac786273819b7bc6e22e5486 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
92152358c95fa9178d71bd1c079af0a0bd8f1da8 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
191251d1340b5e5b2d649b37af3c6c1896d07e6e 
  core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
92d6b2c672f74cdd526f2e98da8f7fb3696a88e3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
efb457334bd87e4c5b0dd2c66ae1995993cd0bc1 

Diff: https://reviews.apache.org/r/32778/diff/


Testing
---


Thanks,

Aditya Auradkar

Review Request 32781: Patch for KAFKA-2087

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32781/
---

Review request for kafka.


Bugs: KAFKA-2087
https://issues.apache.org/jira/browse/KAFKA-2087


Repository: kafka


Description
---

Fixing KAFKA-2087


Diffs
-

  core/src/main/scala/kafka/server/TopicConfigManager.scala 47295d4 

Diff: https://reviews.apache.org/r/32781/diff/


Testing
---


Thanks,

Aditya Auradkar

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Thanks Jun.

Some thoughts:

10) I think it is better we throttle regardless of the produce/fetch version. 
This is a nice feature where clients can tell if they are being throttled or 
not. If we only throttle newer clients, then we have inconsistent behavior 
across clients in a multi-tenant cluster. Having quota metrics on the client 
side is also a nice incentive to upgrade client versions.

11) I think we can call metric.record(fetchSize) before adding the delayedFetch 
request into the purgatory. This will give us the estimated delay of the 
request up-front. The timeout on the DelayedFetch is the Max(maxWait, 
quotaDelay). The DelayedFetch completion criteria can change a little to 
accomodate quotas.

- I agree the quota code should return the estimated delay time in 
QuotaViolationException.

Thanks,
Aditya


From: Jun Rao [j...@confluent.io]
Sent: Friday, April 03, 2015 9:16 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Thanks for the update.

10. About whether to return a new field in the response to indicate
throttling. Earlier, the plan was to not change the response format and
just have a metric on the broker to indicate whether a clientId is
throttled or not. The issue is that we don't know whether a particular
clientId instance is throttled or not (since there could be multiple
clients with the same clientId). Your proposal of adding an isThrottled
field in the response addresses and seems better. Then, do we just throttle
the new version of produce/fetch request or both the old and the new
versions? Also, we probably still need a separate metric on the broker side
to indicate whether a clientId is throttled or not.

11. Just to clarify. For fetch requests, when will metric.record(fetchSize)
be called? Is it when we are ready to send the fetch response (after
minBytes and maxWait are satisfied)?

As an implementation detail, it may be useful for the quota code to return
an estimated delay time (to bring the measurement within the limit) in
QuotaViolationException.

Thanks,

Jun

On Wed, Apr 1, 2015 at 3:27 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey everyone,
>
> I've made changes to the KIP to capture our discussions over the last
> couple of weeks.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
>
> I'll start a voting thread after people have had a chance to read/comment.
>
> Thanks,
> Aditya
>
> 
> From: Steven Wu [stevenz...@gmail.com]
> Sent: Friday, March 20, 2015 9:14 AM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> +1 on Jun's suggestion of maintaining one set/style of metrics at broker.
> In Netflix, we have to convert the yammer metrics to servo metrics at
> broker. it will be painful to know some metrics are in a different style
> and get to be handled differently.
>
> On Fri, Mar 20, 2015 at 8:17 AM, Jun Rao  wrote:
>
> > Not so sure. People who use quota will definitely want to monitor the new
> > metrics at the client id level. Then they will need to deal with those
> > metrics differently from the rest of the metrics. It would be better if
> we
> > can hide this complexity from the users.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 19, 2015 at 10:45 PM, Joel Koshy 
> wrote:
> >
> > > Actually thinking again - since these will be a few new metrics at the
> > > client id level (bytes in and bytes out to start with) maybe it is fine
> > to
> > > have the two type of metrics coexist and we can migrate the existing
> > > metrics in parallel.
> > >
> > > On Thursday, March 19, 2015, Joel Koshy  wrote:
> > >
> > > > That is a valid concern but in that case I think it would be better
> to
> > > > just migrate completely to the new metrics package first.
> > > >
> > > > On Thursday, March 19, 2015, Jun Rao  > > > > wrote:
> > > >
> > > >> Hmm, I was thinking a bit differently on the metrics stuff. I think
> it
> > > >> would be confusing to have some metrics defined in the new metrics
> > > package
> > > >> while some others defined in Coda Hale. Those metrics will look
> > > different
> > > >> (e.g., rates in Coda Hale will have special attributes such as
> > > >> 1-min-average). People may need different ways to export the metrics
> > to
> > > >> external systems such as Graphite. So, instead of using the new
> > metrics
> > > >> package on the broker, I was thinking that we can just implement a
> > > >> QuotaMetrics that wraps the

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Update, I added a proposal on doing dynamic client based configuration that can 
be used for quotas.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas

Please take a look and let me know if there are any concerns.

Thanks,
Aditya

From: Aditya Auradkar
Sent: Friday, April 03, 2015 10:10 AM
To: dev@kafka.apache.org
Subject: RE: [KIP-DISCUSSION] KIP-13 Quotas

Thanks Jun.

Some thoughts:

10) I think it is better we throttle regardless of the produce/fetch version. 
This is a nice feature where clients can tell if they are being throttled or 
not. If we only throttle newer clients, then we have inconsistent behavior 
across clients in a multi-tenant cluster. Having quota metrics on the client 
side is also a nice incentive to upgrade client versions.

11) I think we can call metric.record(fetchSize) before adding the delayedFetch 
request into the purgatory. This will give us the estimated delay of the 
request up-front. The timeout on the DelayedFetch is the Max(maxWait, 
quotaDelay). The DelayedFetch completion criteria can change a little to 
accomodate quotas.

- I agree the quota code should return the estimated delay time in 
QuotaViolationException.

Thanks,
Aditya


From: Jun Rao [j...@confluent.io]
Sent: Friday, April 03, 2015 9:16 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Thanks for the update.

10. About whether to return a new field in the response to indicate
throttling. Earlier, the plan was to not change the response format and
just have a metric on the broker to indicate whether a clientId is
throttled or not. The issue is that we don't know whether a particular
clientId instance is throttled or not (since there could be multiple
clients with the same clientId). Your proposal of adding an isThrottled
field in the response addresses and seems better. Then, do we just throttle
the new version of produce/fetch request or both the old and the new
versions? Also, we probably still need a separate metric on the broker side
to indicate whether a clientId is throttled or not.

11. Just to clarify. For fetch requests, when will metric.record(fetchSize)
be called? Is it when we are ready to send the fetch response (after
minBytes and maxWait are satisfied)?

As an implementation detail, it may be useful for the quota code to return
an estimated delay time (to bring the measurement within the limit) in
QuotaViolationException.

Thanks,

Jun

On Wed, Apr 1, 2015 at 3:27 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey everyone,
>
> I've made changes to the KIP to capture our discussions over the last
> couple of weeks.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
>
> I'll start a voting thread after people have had a chance to read/comment.
>
> Thanks,
> Aditya
>
> 
> From: Steven Wu [stevenz...@gmail.com]
> Sent: Friday, March 20, 2015 9:14 AM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> +1 on Jun's suggestion of maintaining one set/style of metrics at broker.
> In Netflix, we have to convert the yammer metrics to servo metrics at
> broker. it will be painful to know some metrics are in a different style
> and get to be handled differently.
>
> On Fri, Mar 20, 2015 at 8:17 AM, Jun Rao  wrote:
>
> > Not so sure. People who use quota will definitely want to monitor the new
> > metrics at the client id level. Then they will need to deal with those
> > metrics differently from the rest of the metrics. It would be better if
> we
> > can hide this complexity from the users.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 19, 2015 at 10:45 PM, Joel Koshy 
> wrote:
> >
> > > Actually thinking again - since these will be a few new metrics at the
> > > client id level (bytes in and bytes out to start with) maybe it is fine
> > to
> > > have the two type of metrics coexist and we can migrate the existing
> > > metrics in parallel.
> > >
> > > On Thursday, March 19, 2015, Joel Koshy  wrote:
> > >
> > > > That is a valid concern but in that case I think it would be better
> to
> > > > just migrate completely to the new metrics package first.
> > > >
> > > > On Thursday, March 19, 2015, Jun Rao  > > > > wrote:
> > > >
> > > >> Hmm, I was thinking a bit differently on the metrics stuff. I think
> it
> > > >> would be confusing to have some metrics defined in the new metrics
> > > package
> > > >> while some others defined in Coda Hale. Those metrics will look
> > > different
> > > >> (

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Some responses to Jay's points.

1. Using commas - Cool.

2. Adding return flag - I'm inclined to agree with Joel that this is good to 
have in the initial implementation. 

3. Config - +1. I'll remove it from the KIP. We can discuss this in parallel.

4. Purgatory vs Delay queue - I feel that it is simpler to reuse the existing 
purgatories for both delayed produce and fetch requests. IIUC, all we need for 
quotas is a minWait parameter for DelayedOperation (or something equivalent) 
since there is already a max wait. The completion criteria can check if minWait 
time has elapsed before declaring the operation complete. For this to impact 
performance, a significant number of clients may need to exceed their quota at 
the same time and even then I'm not very clear on the scope of the impact. Two 
layers of delays might add complexity to the implementation which I'm hoping to 
avoid.

Aditya


From: Joel Koshy [jjkosh...@gmail.com]
Sent: Friday, April 03, 2015 12:48 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Aditya, thanks for the updated KIP and Jay/Jun thanks for the
comments. Couple of comments in-line:

> 2. I would advocate for adding the return flag when we next bump the
> request format version just to avoid proliferation. I agree this is a good
> thing to know about, but at the moment I don't think we have a very well
> flushed out idea of how the client would actually make use of this info. I

I'm somewhat inclined to having something appropriate off the bat -
mainly because (i) clients really should know that they have been
throttled (ii) a smart producer/consumer implementation would want to
know how much to back off. So perhaps this and config-management
should be moved to a separate discussion, but it would be good to have
this discussion going and incorporated into the first quota
implementation.

> 3. Config--I think we need to generalize the topic stuff so we can override
> at multiple levels. We have topic and client, but I suspect "user" and
> "broker" will also be important. I recommend we take config stuff out of
> this KIP since we really need to fully think through a proposal that will
> cover all these types of overrides.

+1 - it is definitely orthogonal to the core quota implementation
(although necessary for its operability). Having a config-related
discussion in this KIP would only draw out the discussion and vote
even if the core quota design looks good to everyone.

So basically I think we can remove the portions on dynamic config as
well as the response format but I really think we should close on
those while the implementation is in progress and before quotas is
officially released.

> 4. Instead of using purgatories to implement the delay would it make more
> sense to just use a delay queue? I think all the additional stuff in the
> purgatory other than the delay queue doesn't make sense as the quota is a
> hard N ms penalty with no chance of early eviction. If there is no perf
> penalty for the full purgatory that may be fine (even good) to reuse, but I
> haven't looked into that.

A simple delay queue sounds good - I think Aditya was also trying to
avoid adding a new quota purgatory. i.e., it may be possible to use
the existing purgatory instances to enforce quotas. That may be
simpler, but would be incur a slight perf penalty if too many clients
are being throttled.

Thanks,

Joel

>
> -Jay
>
> On Fri, Apr 3, 2015 at 10:45 AM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
>> Update, I added a proposal on doing dynamic client based configuration
>> that can be used for quotas.
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
>>
>> Please take a look and let me know if there are any concerns.
>>
>> Thanks,
>> Aditya
>> 
>> From: Aditya Auradkar
>> Sent: Friday, April 03, 2015 10:10 AM
>> To: dev@kafka.apache.org
>> Subject: RE: [KIP-DISCUSSION] KIP-13 Quotas
>>
>> Thanks Jun.
>>
>> Some thoughts:
>>
>> 10) I think it is better we throttle regardless of the produce/fetch
>> version. This is a nice feature where clients can tell if they are being
>> throttled or not. If we only throttle newer clients, then we have
>> inconsistent behavior across clients in a multi-tenant cluster. Having
>> quota metrics on the client side is also a nice incentive to upgrade client
>> versions.
>>
>> 11) I think we can call metric.record(fetchSize) before adding the
>> delayedFetch request into the purgatory. This will give us the estimated
>> delay of the request up-front. The timeout on the DelayedFetch is the
>> Max(maxWait, quotaDelay). The D

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Hi Jay,

2. At this time, the proposed response format changes are only for 
monitoring/informing clients. As Jun mentioned, we get instance level 
monitoring in this case since each instance that got throttled will have a 
metric confirming the same. Without client level monitoring for this, it's hard 
for application developers to find if they are being throttled since they will 
also have to be aware of all the brokers in the cluster. This is quite 
problematic for large clusters.

It seems nice for app developers to not have to think about kafka internal 
metrics and only focus on the metrics exposed on their instances. Analogous to 
having client-sde request latency metrics. Basically, we want an easy way for 
clients to be aware if they are being throttled.

4. For purgatory v delay queue, I think we are on the same page. I feel it is 
nicer to use the purgatory but I'm happy to use a DelayQueue if there are 
performance implications. I don't know enough about the current and Yasuhiro's 
new implementation to be sure one way or the other.

Stepping back, I think these two things are the only remaining point of 
discussion within the current proposal. Any concerns if I started a voting 
thread on the proposal after the KIP discussion tomorrow? (assuming we reach 
consensus on these items)

Thanks,
Aditya

From: Jay Kreps [jay.kr...@gmail.com]
Sent: Saturday, April 04, 2015 1:36 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Hey Aditya,

2. For the return flag I'm not terribly particular. If we want to add it
let's fully think through how it will be used. The only concern I have is
adding to the protocol without really thinking through the use cases. So
let's work out the APIs we want to add to the Java consumer and producer
and the use cases for how clients will make use of these. For my part I
actually don't see much use other than monitoring since it isn't an error
condition to be at your quota. And if it is just monitoring I don't see a
big enough difference between having the monitoring on the server-side
versus in the clients to justify putting it in the protocol. But I think
you guys may have other use cases in mind of how a client would make some
use of this? Let's work that out. I also don't feel strongly about it--it
wouldn't be *bad* to have the monitoring available on the client, just
doesn't seem that much better.

4. For the purgatory vs delay queue I think is arguably nicer to reuse the
purgatory we just have to be ultra-conscious of efficiency. I think our
goal is to turn quotas on across the board, so at LinkedIn that would mean
potentially every request will need a small delay. I haven't worked out the
efficiency implications of this choice, so as long as we do that I'm happy.

-Jay

On Fri, Apr 3, 2015 at 1:10 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Some responses to Jay's points.
>
> 1. Using commas - Cool.
>
> 2. Adding return flag - I'm inclined to agree with Joel that this is good
> to have in the initial implementation.
>
> 3. Config - +1. I'll remove it from the KIP. We can discuss this in
> parallel.
>
> 4. Purgatory vs Delay queue - I feel that it is simpler to reuse the
> existing purgatories for both delayed produce and fetch requests. IIUC, all
> we need for quotas is a minWait parameter for DelayedOperation (or
> something equivalent) since there is already a max wait. The completion
> criteria can check if minWait time has elapsed before declaring the
> operation complete. For this to impact performance, a significant number of
> clients may need to exceed their quota at the same time and even then I'm
> not very clear on the scope of the impact. Two layers of delays might add
> complexity to the implementation which I'm hoping to avoid.
>
> Aditya
>
> 
> From: Joel Koshy [jjkosh...@gmail.com]
> Sent: Friday, April 03, 2015 12:48 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Aditya, thanks for the updated KIP and Jay/Jun thanks for the
> comments. Couple of comments in-line:
>
> > 2. I would advocate for adding the return flag when we next bump the
> > request format version just to avoid proliferation. I agree this is a
> good
> > thing to know about, but at the moment I don't think we have a very well
> > flushed out idea of how the client would actually make use of this info.
> I
>
> I'm somewhat inclined to having something appropriate off the bat -
> mainly because (i) clients really should know that they have been
> throttled (ii) a smart producer/consumer implementation would want to
> know how much to back off. So perhaps this and config-mana

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Our only use case currently is to expose throttling in the client metrics. I 
think it's better to add more information than an isThrottled flag. Initially 
we were thinking of exposing the amount of time the request was delayed but the 
percentage throttling metric Jay mentioned also sounds good.

Here are the exact metrics we could add using KM sensors.

1. Max - The maximum amount of time/percentage that a request was throttled 
during the last window. Users can configure alerting if this exceeds a certain 
threshold.
2. Avg - The average delay over each request sent during the window.

If exposing delay as time, it corresponds to the existing request-latency-avg, 
request-latency-max metrics currently on the producer. 

(2) Enabling programmatic response is harder and we don't have a use-case for 
this.

Thanks,
Aditya


From: Jay Kreps [jay.kr...@gmail.com]
Sent: Tuesday, April 07, 2015 7:41 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Totally. But is that the only use? What I wanted to flesh out was whether
the goal was:
1. Expose throttling in the client metrics
2. Enable programmatic response (i.e. stop sending stuff or something like
that)

I think I kind of understand (1) but let's get specific on the metric we
would be adding and what exactly you would expose  in a dashboard. For
example if the goal is just monitoring do I really want a boolean flag for
is_throttled or do I want to know how much I am being throttled (i.e.
throttle_pct might indicate the percent of your request time that was due
to throttling or something like that)? If I am 1% throttled that may be
irrelevant but 99% throttled would be quite relevant? Not sure I agree,
just throwing that out there...

For (2) the prior discussion seemed to kind of allude to this but I can't
really come up with a use case. Is there one?

If it is just (1) I think the question is whether it really helps much to
have the metric on the client vs the server. I suppose this is a bit
environment specific. If you have a central metrics system it shouldn't
make any difference, but if you don't I suppose it does.

-Jay

On Mon, Apr 6, 2015 at 7:57 PM, Gwen Shapira  wrote:

> Here's a wild guess:
>
> An app developer included a Kafka Producer in his app, and is not happy
> with the throughput. He doesn't have visibility into the brokers since they
> are owned by a different team. Obviously the first instinct of a developer
> who knows that throttling exists is to blame throttling for any slowdown in
> the app.
> If he doesn't have a way to know from the responses whether or not his app
> is throttled, he may end up calling Aditya at 4am asked "Hey, is my app
> throttled?".
>
> I assume Aditya is trying to avoid this scenario.
>
> On Mon, Apr 6, 2015 at 7:47 PM, Jay Kreps  wrote:
>
> > Hey Aditya,
> >
> > 2. I kind of buy it, but I really like to understand the details of the
> use
> > case before we make protocol changes. What changes are you proposing in
> the
> > clients for monitoring and how would that be used?
> >
> > -Jay
> >
> > On Mon, Apr 6, 2015 at 10:36 AM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Hi Jay,
> > >
> > > 2. At this time, the proposed response format changes are only for
> > > monitoring/informing clients. As Jun mentioned, we get instance level
> > > monitoring in this case since each instance that got throttled will
> have
> > a
> > > metric confirming the same. Without client level monitoring for this,
> > it's
> > > hard for application developers to find if they are being throttled
> since
> > > they will also have to be aware of all the brokers in the cluster. This
> > is
> > > quite problematic for large clusters.
> > >
> > > It seems nice for app developers to not have to think about kafka
> > internal
> > > metrics and only focus on the metrics exposed on their instances.
> > Analogous
> > > to having client-sde request latency metrics. Basically, we want an
> easy
> > > way for clients to be aware if they are being throttled.
> > >
> > > 4. For purgatory v delay queue, I think we are on the same page. I feel
> > it
> > > is nicer to use the purgatory but I'm happy to use a DelayQueue if
> there
> > > are performance implications. I don't know enough about the current and
> > > Yasuhiro's new implementation to be sure one way or the other.
> > >
> > > Stepping back, I think these two things are the only remaining point of
> > > discussion within the current proposal. Any concerns if I star

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Hey everyone,

Following up after today's hangout. After discussing the client side metrics 
piece internally, we've incorporated that section into the KIP.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas

Since there appears to be sufficient consensus, I'm going to start a voting 
thread.

Thanks,
Aditya

From: Gwen Shapira [gshap...@cloudera.com]
Sent: Tuesday, April 07, 2015 11:31 AM
To: Sriharsha Chintalapani
Cc: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Yeah, I was not suggesting adding auth to metrics - I think this needlessly
complicates everything.
But we need to assume that client developers will not have access to the
broker metrics (because in secure environment they probably won't).

Gwen

On Tue, Apr 7, 2015 at 11:20 AM, Sriharsha Chintalapani 
wrote:

> Having auth  on top of metrics is going to be lot more difficult. How are
> we going to restrict metrics reporter which run as part of kafka server
> they will have access to all the metrics and they can publish to ganglia
> etc..  I look at the metrics as a read-only info. As you said metrics for
> all the topics can be visible but what actions are we looking that can be
> non-secure based on metrics alone? . This probably can be part of KIP-11
> discussion.
>  Having said that it will be great if the throttling details can be
> exposed as part of the response to the client. Instead of looking at
> metrics , client can depend on the response to slow down if its being
> throttled.  This allows us the clients can be self-reliant based on the
> response .
>
> --
> Harsha
>
>
> On April 7, 2015 at 9:55:41 AM, Gwen Shapira (gshap...@cloudera.com)
> wrote:
>
> Re (1):
> We have no authorization story on the metrics collected by brokers, so I
> assume that access to broker metrics means knowing exactly which topics
> exist and their throughputs. (Prath and Don, correct me if I got it
> wrong...)
> Secure environments will strictly control access to this information, so I
> am pretty sure the client developers will not have access to server
> metrics
> at all.
>
> Gwen
>
> On Tue, Apr 7, 2015 at 7:41 AM, Jay Kreps  wrote:
>
> > Totally. But is that the only use? What I wanted to flesh out was
> whether
> > the goal was:
> > 1. Expose throttling in the client metrics
> > 2. Enable programmatic response (i.e. stop sending stuff or something
> like
> > that)
> >
> > I think I kind of understand (1) but let's get specific on the metric we
> > would be adding and what exactly you would expose in a dashboard. For
> > example if the goal is just monitoring do I really want a boolean flag
> for
> > is_throttled or do I want to know how much I am being throttled (i.e.
> > throttle_pct might indicate the percent of your request time that was
> due
> > to throttling or something like that)? If I am 1% throttled that may be
> > irrelevant but 99% throttled would be quite relevant? Not sure I agree,
> > just throwing that out there...
> >
> > For (2) the prior discussion seemed to kind of allude to this but I
> can't
> > really come up with a use case. Is there one?
> >
> > If it is just (1) I think the question is whether it really helps much
> to
> > have the metric on the client vs the server. I suppose this is a bit
> > environment specific. If you have a central metrics system it shouldn't
> > make any difference, but if you don't I suppose it does.
> >
> > -Jay
> >
> > On Mon, Apr 6, 2015 at 7:57 PM, Gwen Shapira 
> > wrote:
> >
> > > Here's a wild guess:
> > >
> > > An app developer included a Kafka Producer in his app, and is not
> happy
> > > with the throughput. He doesn't have visibility into the brokers since
> > they
> > > are owned by a different team. Obviously the first instinct of a
> > developer
> > > who knows that throttling exists is to blame throttling for any
> slowdown
> > in
> > > the app.
> > > If he doesn't have a way to know from the responses whether or not his
> > app
> > > is throttled, he may end up calling Aditya at 4am asked "Hey, is my
> app
> > > throttled?".
> > >
> > > I assume Aditya is trying to avoid this scenario.
> > >
> > > On Mon, Apr 6, 2015 at 7:47 PM, Jay Kreps 
> wrote:
> > >
> > > > Hey Aditya,
> > > >
> > > > 2. I kind of buy it, but I really like to understand the details of
> the
> > > use
> > > > case before we make protocol changes. What changes are you 

[VOTE] KIP-13 Quotas

[Aditya Auradkar]

Please vote.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas

Thanks,
Aditya

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Thanks for the detailed review. I've addressed your comments.

For rejected alternatives, we've rejected per-partition distribution because we 
choose client based quotas where there is no notion of partitions. I've 
explained in a bit more detail in that section.

Aditya


From: Joel Koshy [jjkosh...@gmail.com]
Sent: Wednesday, April 08, 2015 6:30 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Thanks for updating the wiki. Looks great overall. Just a couple
more comments:

Client status code:
- v0 requests -> current version (0) of those requests.
- Fetch response has a throttled flag instead of throttle time -  I
  think you intended the latter.
- Can you make it clear that the quota status is a new field
  called throttleTimeMs (or equivalent). It would help if some of
  that is moved (or repeated) in compatibility/migration plan.
- So you would need to upgrade brokers first, then the clients.
  While upgrading the brokers (via a rolling bounce) the brokers
  cannot start using the latest fetch-request version immediately
  (for replica fetches). Since there will be older brokers in the mix
  those brokers would not be able to read v1 fetch requests. So all
  the brokers should be upgraded before switching to the latest
  fetch request version. This is similar to what Gwen proposed in
  KIP-2/KAFKA-1809 and I think we will need to use the
  inter-broker protocol version config.

Rejected alternatives-quota-distribution.B: notes that this is the
most elegant model, but does not explain why it was rejected. I
think this was because we would then need some sort of gossip
between brokers since partitions are across the cluster. Can you
confirm?

Thanks,

Joel

On Wed, Apr 08, 2015 at 05:45:34AM +0000, Aditya Auradkar wrote:
> Hey everyone,
>
> Following up after today's hangout. After discussing the client side metrics 
> piece internally, we've incorporated that section into the KIP.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
>
> Since there appears to be sufficient consensus, I'm going to start a voting 
> thread.
>
> Thanks,
> Aditya
> 
> From: Gwen Shapira [gshap...@cloudera.com]
> Sent: Tuesday, April 07, 2015 11:31 AM
> To: Sriharsha Chintalapani
> Cc: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Yeah, I was not suggesting adding auth to metrics - I think this needlessly
> complicates everything.
> But we need to assume that client developers will not have access to the
> broker metrics (because in secure environment they probably won't).
>
> Gwen
>
> On Tue, Apr 7, 2015 at 11:20 AM, Sriharsha Chintalapani 
> wrote:
>
> > Having auth  on top of metrics is going to be lot more difficult. How are
> > we going to restrict metrics reporter which run as part of kafka server
> > they will have access to all the metrics and they can publish to ganglia
> > etc..  I look at the metrics as a read-only info. As you said metrics for
> > all the topics can be visible but what actions are we looking that can be
> > non-secure based on metrics alone? . This probably can be part of KIP-11
> > discussion.
> >  Having said that it will be great if the throttling details can be
> > exposed as part of the response to the client. Instead of looking at
> > metrics , client can depend on the response to slow down if its being
> > throttled.  This allows us the clients can be self-reliant based on the
> > response .
> >
> > --
> > Harsha
> >
> >
> > On April 7, 2015 at 9:55:41 AM, Gwen Shapira (gshap...@cloudera.com)
> > wrote:
> >
> > Re (1):
> > We have no authorization story on the metrics collected by brokers, so I
> > assume that access to broker metrics means knowing exactly which topics
> > exist and their throughputs. (Prath and Don, correct me if I got it
> > wrong...)
> > Secure environments will strictly control access to this information, so I
> > am pretty sure the client developers will not have access to server
> > metrics
> > at all.
> >
> > Gwen
> >
> > On Tue, Apr 7, 2015 at 7:41 AM, Jay Kreps  wrote:
> >
> > > Totally. But is that the only use? What I wanted to flesh out was
> > whether
> > > the goal was:
> > > 1. Expose throttling in the client metrics
> > > 2. Enable programmatic response (i.e. stop sending stuff or something
> > like
> > > that)
> > >
> > > I think I kind of understand (1) but let's get specific on the metric we
> > > would be adding and what exactly you would expose in a dashboard. For
> > > examp

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated April 11, 2015, 12:24 a.m.)


Review request for kafka.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 
b3d3d7c56acb445be16a3fbe00f05eaba659be46 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
69b772c1941865fbe15b34bb2784c511f8ce519a 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c63f4ba9d622817ea8636d4e6135fba917ce085a 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated April 11, 2015, 12:25 a.m.)


Review request for kafka.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

WIP: First patch for quotas. Changes are 
1. Adding per-client throttle time and quota metrics in 
ClientQuotaMetrics.scala 
2. Making changes in QuotaViolationException and Sensor to return delay time 
changes. 
3. Added configuration needed so far for quotas in KafkaConfig. 
4. Unit tests

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 
b3d3d7c56acb445be16a3fbe00f05eaba659be46 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
69b772c1941865fbe15b34bb2784c511f8ce519a 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c63f4ba9d622817ea8636d4e6135fba917ce085a 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

RE: [VOTE] KIP-13 Quotas

[Aditya Auradkar]

Thanks! I've clarified the request protocol change on the KIP.

Aditya


From: Joel Koshy [jjkosh...@gmail.com]
Sent: Wednesday, April 08, 2015 5:49 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-13 Quotas

Agreed on that - it is understood from the text of the KIP "Clients
that send current version (0) of those requests will not receive the
quota status flag in the response" => you have to send version (1) to
get the new repsonse, but it should probably be clarified on the page.

I'm +1 as well on the proposal.

Thanks,

Joel

On Wed, Apr 08, 2015 at 05:41:39PM -0700, Jun Rao wrote:
> Aditya,
>
> To evolve the response protocol for produce/fetch response, we need to have
> new versions of the requests as well. Version 1 of the requests will have
> the same protocol as that of version 0, and will get the new response. The
> rest of the stuff looks good to me.
>
> Thanks,
>
> Jun
>
> On Tue, Apr 7, 2015 at 10:47 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Please vote.
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> >
> > Thanks,
> > Aditya
> >

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

This is an implementation proposal for delaying requests in quotas using the 
current purgatory. I'll discuss the usage for produce and fetch requests 
separately.

1. Delayed Produce Requests - Here, the proposal is basically to reuse 
DelayedProduce objects and insert them into the purgatory with no watcher keys 
if the request is being throttled. The timeout used in the request should be 
the Max(quota_delay_time, replication_timeout).
In most cases, the quota timeout should be greater than the existing timeout 
but in order to be safe, we can use the maximum of these values. Having no 
watch keys will allow the operation to be enqueued directly into the timer and 
will not add any overhead in terms of watching keys (which was a concern). In 
this case, having watch keys is not beneficial since the operation must be 
delayed for a fixed amount of time and there is no possibility for the 
operation to complete before the timeout i.e. tryComplete() can never return 
true before the timeout. On timeout, since the operation is a TimerTask, the 
timer will call run() which calls onComplete().
In onComplete, the DelayedProduce can repeat the check in tryComplete() (only 
if acks=-1 whether all replicas fetched upto a certain offset) and return the 
response immediately.

Code will be structured as follows in ReplicaManager:appendMessages()

if(isThrottled) {
  fetch = new DelayedProduce(timeout)
  purgatory.tryCompleteElseWatch(fetch, Seq())
}
else if(delayedRequestRequired()) {
 // Insert into purgatory with watched keys for unthrottled requests
}

In this proposal, we avoid adding unnecessary watches because there is no 
possibility of early completion and this avoids any potential performance 
penalties we were concerned about earlier.

2. Delayed Fetch Requests - Similarly, the proposal here is to reuse the 
DelayedFetch objects and insert them into the purgatory with no watcher keys if 
the request is throttled. Timeout used is the Max(quota_delay_time, 
max_wait_timeout). Having no watch keys provides the same benefits as described 
above. Upon timeout, the onComplete() is called and the operation proceeds 
normally i.e. perform a readFromLocalLog and return a response.
The caveat here is that if the request is throttled but the throttle time is 
less than the max_wait timeout on the fetch request, the request will be 
delayed to a Max(quota_delay_time, max_wait_timeout). This may be more than 
strictly necessary (since we are not watching for
satisfaction on any keys).

I added some testcases to DelayedOperationTest to verify that it is possible to 
schedule operations with no watcher keys. By inserting elements with no watch 
keys, the purgatory simply becomes a delay queue. It may also make sense to add 
a new API to the purgatory called
delayFor() that basically accepts an operation without any watch keys (Thanks 
for the suggestion Joel).

Thoughts? 

Thanks,
Aditya


From: Guozhang Wang [wangg...@gmail.com]
Sent: Monday, April 13, 2015 7:27 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

I think KAFKA-2063 (bounding fetch response) is still under discussion, and
may not be got it in time with KAFKA-1927.

On Thu, Apr 9, 2015 at 4:49 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> I think it's reasonable to batch the protocol changes together. In
> addition to the protocol changes, is someone actively driving the server
> side changes/KIP process for KAFKA-2063?
>
> Thanks,
> Aditya
>
> 
> From: Jun Rao [j...@confluent.io]
> Sent: Thursday, April 09, 2015 8:59 AM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Since we are also thinking about evolving the fetch request protocol in
> KAFKA-2063 (bound fetch response size), perhaps it's worth thinking through
> if we can just evolve the protocol once.
>
> Thanks,
>
> Jun
>
> On Wed, Apr 8, 2015 at 10:43 AM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Thanks for the detailed review. I've addressed your comments.
> >
> > For rejected alternatives, we've rejected per-partition distribution
> > because we choose client based quotas where there is no notion of
> > partitions. I've explained in a bit more detail in that section.
> >
> > Aditya
> >
> > 
> > From: Joel Koshy [jjkosh...@gmail.com]
> > Sent: Wednesday, April 08, 2015 6:30 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> >
> > Thanks for updating the wiki. Looks great overall. Just a couple
> > more comments:
> >
> > Client status code:
> > - v0 requests -> current version (0) of those requests.
> > - Fetch response has a

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Thanks for the review Guozhang.

1. Agreed. 

2. This proposal actually waits for the maximum of the 2 timeouts. This reduces 
implementation complexity at the cost of waiting longer than strictly needed 
for quotas. Note that this is only for the case where acks=-1. 

However we can solve this if it is a significant concern by adding watcher keys 
for all partitions (only if acks=-1). These are the keys we would normally add 
while waiting for acknowledgements. We can change the tryComplete() function to 
return false until 'quota_timeout' time has elapsed AND all the 
acknowledgements have been received.

Thanks,
Aditya

From: Guozhang Wang [wangg...@gmail.com]
Sent: Wednesday, April 15, 2015 3:42 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Thanks for the summary. A few comments below:

1. Say a produce request has replication timeout X, and upon finishing the
local append it is determined to be throttled Y ms where Y > X, then after
it has timed out in the purgatory after Y ms we should still check if the
#.acks has fulfilled in order to set the correct error codes in the
response.

2. I think it is actually common that the calculated throttle time Y is
less than the replication timeout X, which will be a tricky case since we
need to make sure 1) at least the request it held in the purgatory for Y
ms, 2) after Y ms elapsed, if the #.acks has fulfilled within X ms then set
no-error-code and return immediately, 3) after X ms elapsed, set
timeout-error-code and return.

Guozhang

On Tue, Apr 14, 2015 at 5:01 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> This is an implementation proposal for delaying requests in quotas using
> the current purgatory. I'll discuss the usage for produce and fetch
> requests separately.
>
> 1. Delayed Produce Requests - Here, the proposal is basically to reuse
> DelayedProduce objects and insert them into the purgatory with no watcher
> keys if the request is being throttled. The timeout used in the request
> should be the Max(quota_delay_time, replication_timeout).
> In most cases, the quota timeout should be greater than the existing
> timeout but in order to be safe, we can use the maximum of these values.
> Having no watch keys will allow the operation to be enqueued directly into
> the timer and will not add any overhead in terms of watching keys (which
> was a concern). In this case, having watch keys is not beneficial since the
> operation must be delayed for a fixed amount of time and there is no
> possibility for the operation to complete before the timeout i.e.
> tryComplete() can never return true before the timeout. On timeout, since
> the operation is a TimerTask, the timer will call run() which calls
> onComplete().
> In onComplete, the DelayedProduce can repeat the check in tryComplete()
> (only if acks=-1 whether all replicas fetched upto a certain offset) and
> return the response immediately.
>
> Code will be structured as follows in ReplicaManager:appendMessages()
>
> if(isThrottled) {
>   fetch = new DelayedProduce(timeout)
>   purgatory.tryCompleteElseWatch(fetch, Seq())
> }
> else if(delayedRequestRequired()) {
>  // Insert into purgatory with watched keys for unthrottled requests
> }
>
> In this proposal, we avoid adding unnecessary watches because there is no
> possibility of early completion and this avoids any potential performance
> penalties we were concerned about earlier.
>
> 2. Delayed Fetch Requests - Similarly, the proposal here is to reuse the
> DelayedFetch objects and insert them into the purgatory with no watcher
> keys if the request is throttled. Timeout used is the Max(quota_delay_time,
> max_wait_timeout). Having no watch keys provides the same benefits as
> described above. Upon timeout, the onComplete() is called and the operation
> proceeds normally i.e. perform a readFromLocalLog and return a response.
> The caveat here is that if the request is throttled but the throttle time
> is less than the max_wait timeout on the fetch request, the request will be
> delayed to a Max(quota_delay_time, max_wait_timeout). This may be more than
> strictly necessary (since we are not watching for
> satisfaction on any keys).
>
> I added some testcases to DelayedOperationTest to verify that it is
> possible to schedule operations with no watcher keys. By inserting elements
> with no watch keys, the purgatory simply becomes a delay queue. It may also
> make sense to add a new API to the purgatory called
> delayFor() that basically accepts an operation without any watch keys
> (Thanks for the suggestion Joel).
>
> Thoughts?
>
> Thanks,
> Aditya
>
> 
> From: Guozhang Wang [wangg...@gmail.com]
> Sent: Monday, April 

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Hey Guozhang,

I don't think we should return an error if the request is satisfied after Y 
(throttling timeout) because it may cause the producer to think that the 
request was not ack'ed at all. 

Aditya


From: Guozhang Wang [wangg...@gmail.com]
Sent: Thursday, April 16, 2015 9:06 AM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Hi Adi,

2. I assume you were saying "than strictly needed for replications" here?

Also the concern I have is around error code: today if the replication is
not finished within in the replication timeout then the error code will be
set accordingly when it returns. Let's say if the produce request is not
satisfied after X (replication timeout) ms, but is satisfied after Y
(throttling timeout), should we still set the error code or not? I think it
is OK to just set NO_ERROR but we need to document such cases clearly for
quote actions mixed with ack = -1.

Guozhang

On Wed, Apr 15, 2015 at 4:23 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Thanks for the review Guozhang.
>
> 1. Agreed.
>
> 2. This proposal actually waits for the maximum of the 2 timeouts. This
> reduces implementation complexity at the cost of waiting longer than
> strictly needed for quotas. Note that this is only for the case where
> acks=-1.
>
> However we can solve this if it is a significant concern by adding watcher
> keys for all partitions (only if acks=-1). These are the keys we would
> normally add while waiting for acknowledgements. We can change the
> tryComplete() function to return false until 'quota_timeout' time has
> elapsed AND all the acknowledgements have been received.
>
> Thanks,
> Aditya
> 
> From: Guozhang Wang [wangg...@gmail.com]
> Sent: Wednesday, April 15, 2015 3:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Thanks for the summary. A few comments below:
>
> 1. Say a produce request has replication timeout X, and upon finishing the
> local append it is determined to be throttled Y ms where Y > X, then after
> it has timed out in the purgatory after Y ms we should still check if the
> #.acks has fulfilled in order to set the correct error codes in the
> response.
>
> 2. I think it is actually common that the calculated throttle time Y is
> less than the replication timeout X, which will be a tricky case since we
> need to make sure 1) at least the request it held in the purgatory for Y
> ms, 2) after Y ms elapsed, if the #.acks has fulfilled within X ms then set
> no-error-code and return immediately, 3) after X ms elapsed, set
> timeout-error-code and return.
>
> Guozhang
>
> On Tue, Apr 14, 2015 at 5:01 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > This is an implementation proposal for delaying requests in quotas using
> > the current purgatory. I'll discuss the usage for produce and fetch
> > requests separately.
> >
> > 1. Delayed Produce Requests - Here, the proposal is basically to reuse
> > DelayedProduce objects and insert them into the purgatory with no watcher
> > keys if the request is being throttled. The timeout used in the request
> > should be the Max(quota_delay_time, replication_timeout).
> > In most cases, the quota timeout should be greater than the existing
> > timeout but in order to be safe, we can use the maximum of these values.
> > Having no watch keys will allow the operation to be enqueued directly
> into
> > the timer and will not add any overhead in terms of watching keys (which
> > was a concern). In this case, having watch keys is not beneficial since
> the
> > operation must be delayed for a fixed amount of time and there is no
> > possibility for the operation to complete before the timeout i.e.
> > tryComplete() can never return true before the timeout. On timeout, since
> > the operation is a TimerTask, the timer will call run() which calls
> > onComplete().
> > In onComplete, the DelayedProduce can repeat the check in tryComplete()
> > (only if acks=-1 whether all replicas fetched upto a certain offset) and
> > return the response immediately.
> >
> > Code will be structured as follows in ReplicaManager:appendMessages()
> >
> > if(isThrottled) {
> >   fetch = new DelayedProduce(timeout)
> >   purgatory.tryCompleteElseWatch(fetch, Seq())
> > }
> > else if(delayedRequestRequired()) {
> >  // Insert into purgatory with watched keys for unthrottled requests
> > }
> >
> > In this proposal, we avoid adding unnecessary watches because there is no
> > possibility of early completion and thi

Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

Review request for kafka.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description
---

Changes are:
- protocol changes to the fetch reuqest and response to return the 
throttle_time_ms to clients
- New producer/consumer metrics to expose the avg and max delay time for a 
client
- Test cases

For now the patch will publish a zero delay and return a response

Added more tests


Diffs
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
ef9dd5238fbc771496029866ece1d85db6d7b7a5 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
70954cadb5c7e9a4c326afcf9d9a07db230e7db2 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
9c4518e840904c371a5816bfc52be1933cba0b96 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
b038c15186c0cbcc65b59479324052498361b717 
  core/src/main/scala/kafka/api/FetchResponse.scala 
75aaf57fb76ec01660d93701a57ae953d877d81c 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
570b2da1d865086f9830aa919a49063abbbe574d 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
b4004aa3a1456d337199aa1245fb0ae61f6add46 
  core/src/main/scala/kafka/server/OffsetManager.scala 
420e2c3535e722c503f13d093849469983f6f08d 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
8ddd325015de4245fd2cf500d8b0e8c1fd2bc7e8 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
566b5381665bb027a06e17c4fc27414943f85220 
  core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
9186c90de5a983a73b042fcb42987bfabae14fcf 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

(Updated April 21, 2015, 12:02 a.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description
---

Changes are:
- protocol changes to the fetch reuqest and response to return the 
throttle_time_ms to clients
- New producer/consumer metrics to expose the avg and max delay time for a 
client
- Test cases

For now the patch will publish a zero delay and return a response

Added more tests


Diffs
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
ef9dd5238fbc771496029866ece1d85db6d7b7a5 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
70954cadb5c7e9a4c326afcf9d9a07db230e7db2 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
9c4518e840904c371a5816bfc52be1933cba0b96 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
b038c15186c0cbcc65b59479324052498361b717 
  core/src/main/scala/kafka/api/FetchResponse.scala 
75aaf57fb76ec01660d93701a57ae953d877d81c 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
570b2da1d865086f9830aa919a49063abbbe574d 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
b4004aa3a1456d337199aa1245fb0ae61f6add46 
  core/src/main/scala/kafka/server/OffsetManager.scala 
420e2c3535e722c503f13d093849469983f6f08d 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
8ddd325015de4245fd2cf500d8b0e8c1fd2bc7e8 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
566b5381665bb027a06e17c4fc27414943f85220 
  core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
9186c90de5a983a73b042fcb42987bfabae14fcf 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing (updated)
---

New tests added


Thanks,

Aditya Auradkar

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

(Updated April 21, 2015, 12:02 a.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description (updated)
---

Changes are:
- protocol changes to the fetch request and response to return the 
throttle_time_ms to clients
- New producer/consumer metrics to expose the avg and max delay time for a 
client
- Test cases

For now the patch will publish a zero delay and return a response

Added more tests


Diffs
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
ef9dd5238fbc771496029866ece1d85db6d7b7a5 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
70954cadb5c7e9a4c326afcf9d9a07db230e7db2 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
9c4518e840904c371a5816bfc52be1933cba0b96 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
b038c15186c0cbcc65b59479324052498361b717 
  core/src/main/scala/kafka/api/FetchResponse.scala 
75aaf57fb76ec01660d93701a57ae953d877d81c 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
570b2da1d865086f9830aa919a49063abbbe574d 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
b4004aa3a1456d337199aa1245fb0ae61f6add46 
  core/src/main/scala/kafka/server/OffsetManager.scala 
420e2c3535e722c503f13d093849469983f6f08d 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
8ddd325015de4245fd2cf500d8b0e8c1fd2bc7e8 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
566b5381665bb027a06e17c4fc27414943f85220 
  core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
9186c90de5a983a73b042fcb42987bfabae14fcf 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing
---

New tests added


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated April 21, 2015, 7:21 p.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 
b3d3d7c56acb445be16a3fbe00f05eaba659be46 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
69b772c1941865fbe15b34bb2784c511f8ce519a 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c63f4ba9d622817ea8636d4e6135fba917ce085a 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
8ddd325015de4245fd2cf500d8b0e8c1fd2bc7e8 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

> On April 17, 2015, 11:21 p.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/server/ClientQuotaMetrics.scala, line 33
> > <https://reviews.apache.org/r/33049/diff/3/?file=924194#file924194line33>
> >
> > Is this necessary?

Not strictly but I felt it was a nice to have. I can make the default 1 if it 
helps. I'm envisioning a scenario where we discover in production that our 
throttle time computation isn't agressive enough and this can help in such a 
case.


> On April 17, 2015, 11:21 p.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/server/ClientQuotaMetrics.scala, line 81
> > <https://reviews.apache.org/r/33049/diff/3/?file=924194#file924194line81>
> >
> > Overall, I think this should work, but it seems slightly high touch no?
> > 
> > i.e., do we really need to wrap everything? i.e., you definitely need a 
> > clientQuotaMetricsConfig, but it seems many of the wrapper routines here 
> > can be folded into the core metrics package.
> > 
> > E.g., otherwise for every metric that we want to quota on, we are 
> > forced to add new record* methods;
> > 
> > If I'm reading this right, a motivation for having the wrappers is to 
> > getOrCreate the sensors. Can we just pre-emptively (at the beginning) 
> > create the per-client sensors and then avoid the wrapper routines? This 
> > would also help avoid the need for the extra quota map and the 
> > synchronization logic in creating the sensors.

It isn't possible to pro-actively create all the sensors since we don't know in 
advance what clients will connect to the system. Eventually, when we have 
dynamic configs we can't even proactively create sensors for the overridden 
ones. 

I've published a new patch that will make this easier to use i.e. not have 
typed methods per metric we want to throttle.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review80130
---


On April 21, 2015, 7:21 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated April 21, 2015, 7:21 p.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> This is currently not being used anywhere in the code because I haven't yet 
> figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
> better idea once I look at the new purgatory implementation. Hopefully, this 
> smaller patch is easier to review.
> 
> Added more testcases
> 
> 
> Some locking changes for reading/creating the sensors
> 
> 
> WIP patch
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
> dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
>   clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 
> b3d3d7c56acb445be16a3fbe00f05eaba659be46 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
>  a451e5385c9eca76b38b425e8ac856b2715fcffe 
>   clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
> ca823fd4639523018311b814fde69b6177e73b97 
>   core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 69b772c1941865fbe15b34bb2784c511f8ce519a 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c63f4ba9d622817ea8636d4e6135fba917ce085a 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 8ddd325015de4245fd2cf500d8b0e8c1fd2bc7e8 
>   core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33049/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated April 21, 2015, 7:28 p.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 
b3d3d7c56acb445be16a3fbe00f05eaba659be46 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/ClientQuotaMetrics2.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
69b772c1941865fbe15b34bb2784c511f8ce519a 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c63f4ba9d622817ea8636d4e6135fba917ce085a 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
8ddd325015de4245fd2cf500d8b0e8c1fd2bc7e8 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest2.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated April 21, 2015, 7:33 p.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

WIP: First patch for quotas. 
Changes are 
1. Adding per-client throttle time and quota metrics in 
ClientQuotaMetrics.scala 
2. Making changes in QuotaViolationException and Sensor to return delay time 
changes. 
3. Added configuration needed so far for quotas in KafkaConfig. 
4. Unit tests This is currently not being used anywhere in the code because I 
haven't yet figured out how to enforce delays i.e. purgatory vs delay queue. 
I'll have a better idea once I look at the new purgatory implementation.

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. Hopefully, 
this smaller patch is easier to review.


Please read: This patch has 2 approaches for managing quotas in 
ClientQuotaMetrics and CLientQuotaMetrics2 along with some example usage in 
ReplicaManager. This code will have to be cleaned up significantly in order to 
commit but I'm looking for feedback on which approach to use.

Approach 1: ClientQuotaMetrics wraps everything into a single class. Adding new 
metrics is much clumsier.
Approach 2: ClientQuotaMetrics2 only maintains per-client metrics for a single 
entity (producer, consumer) etc.. This makes the code easier to use. For 
throttling on a new dimention i.e. request per second, we only need to create 
this object with a new quota and will just work.


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 
b3d3d7c56acb445be16a3fbe00f05eaba659be46 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/ClientQuotaMetrics2.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
69b772c1941865fbe15b34bb2784c511f8ce519a 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c63f4ba9d622817ea8636d4e6135fba917ce085a 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
8ddd325015de4245fd2cf500d8b0e8c1fd2bc7e8 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest2.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

> > small request, large response: delay queue before handling
> > large request, small response: delay queue after handling, before
> response
> > small request, small response: either is fine
> > large request, large resopnse: we really cannot do anything here but we
> > don't really have this scenario yet
> >
> > So the design would look like this:
> >
> > - parse request
> > - before handling request check if quota violated; if so compute two
> delay
> > numbers:
> >   - before handling delay
> >   - before response delay
> > - if before-handling delay > 0 insert into before-handling delay queue
> > - handle the request
> > - if before-response delay > 0 insert into before-response delay queue
> > - respond
> >
> > Just throwing this out there for discussion.
> >
> > Thanks,
> >
> > Joel
> >
> > On Thu, Apr 16, 2015 at 02:56:23PM -0700, Jun Rao wrote:
> > > The quota check for the fetch request is a bit different from the
> produce
> > > request. I assume that for the fetch request, we will first get an
> > > estimated fetch response size to do the quota check. There are two
> things
> > > to think about. First, when we actually send the response, we probably
> > > don't want to record the metric again since it will double count.
> Second,
> > > the bytes that the fetch response actually sends could be more than the
> > > estimate. This means that the metric may not be 100% accurate. We may
> be
> > > able to limit the fetch size of each partition to what's in the
> original
> > > estimate.
> > >
> > > For the produce request, I was thinking that another way to do this is
> to
> > > first figure out the quota_timeout. Then wait in Purgatory for
> > > quota_timeout with no key. If the request is not satisfied in
> > quota_timeout
> > > and (request_timeout > quota_timeout), wait in Purgatory for
> > > (request_timeout - quota_timeout) with the original keys.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Apr 14, 2015 at 5:01 PM, Aditya Auradkar <
> > > aaurad...@linkedin.com.invalid> wrote:
> > >
> > > > This is an implementation proposal for delaying requests in quotas
> > using
> > > > the current purgatory. I'll discuss the usage for produce and fetch
> > > > requests separately.
> > > >
> > > > 1. Delayed Produce Requests - Here, the proposal is basically to
> reuse
> > > > DelayedProduce objects and insert them into the purgatory with no
> > watcher
> > > > keys if the request is being throttled. The timeout used in the
> request
> > > > should be the Max(quota_delay_time, replication_timeout).
> > > > In most cases, the quota timeout should be greater than the existing
> > > > timeout but in order to be safe, we can use the maximum of these
> > values.
> > > > Having no watch keys will allow the operation to be enqueued directly
> > into
> > > > the timer and will not add any overhead in terms of watching keys
> > (which
> > > > was a concern). In this case, having watch keys is not beneficial
> > since the
> > > > operation must be delayed for a fixed amount of time and there is no
> > > > possibility for the operation to complete before the timeout i.e.
> > > > tryComplete() can never return true before the timeout. On timeout,
> > since
> > > > the operation is a TimerTask, the timer will call run() which calls
> > > > onComplete().
> > > > In onComplete, the DelayedProduce can repeat the check in
> tryComplete()
> > > > (only if acks=-1 whether all replicas fetched upto a certain offset)
> > and
> > > > return the response immediately.
> > > >
> > > > Code will be structured as follows in ReplicaManager:appendMessages()
> > > >
> > > > if(isThrottled) {
> > > >   fetch = new DelayedProduce(timeout)
> > > >   purgatory.tryCompleteElseWatch(fetch, Seq())
> > > > }
> > > > else if(delayedRequestRequired()) {
> > > >  // Insert into purgatory with watched keys for unthrottled requests
> > > > }
> > > >
> > > > In this proposal, we avoid adding unnecessary watches because there
> is
> > no
> > > > possibility of early completion and this avoids any potential
> > performance
> > > > penalties we were

Re: Review Request 33551: Patch for KAFKA-2150

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33551/#review81593
---


Minor comment.


core/src/main/scala/kafka/server/AbstractFetcherThread.scala
<https://reviews.apache.org/r/33551/#comment132027>

The way this function is implemented, it potentially acquires the same lock 
twice when it's not really necessary.

Can you simply merge this check into the synchronized block above? It's a 
fairly cheap check I guess. The fetch request can still be performed outside 
the locked segment.


- Aditya Auradkar


On April 25, 2015, 2:48 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33551/
> ---
> 
> (Updated April 25, 2015, 2:48 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2150
> https://issues.apache.org/jira/browse/KAFKA-2150
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2150. FetcherThread backoff need to grab lock before wait on condition.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> a439046e118b6efcc3a5a9d9e8acb79f85e40398 
> 
> Diff: https://reviews.apache.org/r/33551/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

Re: Review Request 33551: Patch for KAFKA-2150

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33551/#review81596
---



core/src/main/scala/kafka/server/AbstractFetcherThread.scala
<https://reviews.apache.org/r/33551/#comment132029>

I was thinking that you could still process the fetchRequest outside the 
locked segment. Is there a reason a flag wouldn't work?


- Aditya Auradkar


On April 25, 2015, 2:48 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33551/
> ---
> 
> (Updated April 25, 2015, 2:48 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2150
> https://issues.apache.org/jira/browse/KAFKA-2150
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2150. FetcherThread backoff need to grab lock before wait on condition.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> a439046e118b6efcc3a5a9d9e8acb79f85e40398 
> 
> Diff: https://reviews.apache.org/r/33551/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

Re: Review Request 33551: Patch for KAFKA-2150

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33551/#review81693
---

Ship it!


Ship It!

- Aditya Auradkar


On April 25, 2015, 8:35 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33551/
> ---
> 
> (Updated April 25, 2015, 8:35 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2150
> https://issues.apache.org/jira/browse/KAFKA-2150
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2150. FetcherThread backoff need to grab lock before wait on condition.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> a439046e118b6efcc3a5a9d9e8acb79f85e40398 
> 
> Diff: https://reviews.apache.org/r/33551/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

So, are we all in agreement on this approach? If so, I can start working on a 
patch for it.

Also, there are a couple of patches for quotas in need of review. I think we 
need a couple of reviewers, so I've added Jun as a reviewer in addition to 
Joel. 

https://reviews.apache.org/r/33049/
https://reviews.apache.org/r/33378/

Thanks,
Aditya

From: Joel Koshy [jjkosh...@gmail.com]
Sent: Friday, April 24, 2015 5:34 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

I think Jay meant a catch-all request/sec limit on all requests
per-client. That makes sense.

On Fri, Apr 24, 2015 at 11:02:29PM +0000, Aditya Auradkar wrote:
> I think Joel's suggestion is quite good. It's still possible to throttle 
> other types of requests using purgatory but we will need a separate purgatory 
> and DelayedOperation variants of different request types or perhaps add a 
> ThrottledOperation type. It also addresses a couple of special case 
> situations wrt delay time and replication timeouts.
>
> Jay, if we have a general mechanism of delaying requests then it should be 
> possible to throttle any type of request as long as we have metrics on a 
> per-client basis. For offset commit requests, we would simply need a request 
> rate metric per-client and a good default quota.
>
> Thanks,
> Aditya
>
> 
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Friday, April 24, 2015 3:20 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Hey Jun/Joel,
>
> Yeah we will definitely want to quota non-produce/consume requests.
> Especially offset commit and any other requests the consumer can trigger
> could easily get invoked in a tight loop by accident. We haven't talked
> about this a ton, but presumably the mechanism for all these would just be
> a general requests/sec limit that covers all requests?
>
> -Jay
>
>
> On Fri, Apr 24, 2015 at 2:18 PM, Jun Rao  wrote:
>
> > Joel,
> >
> > What you suggested makes sense. Not sure if there is a strong need to
> > throttle TMR though since it should be infrequent.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Apr 21, 2015 at 12:21 PM, Joel Koshy  wrote:
> >
> > > Given the caveats, it may be worth doing further investigation on the
> > > alternate approach which is to use a dedicated DelayQueue for requests
> > > that violate quota and compare pros/cons.
> > >
> > > So the approach is the following: all request handling occurs normally
> > > (i.e., unchanged from what we do today). i.e., purgatories will be
> > > unchanged.  After handling a request and before sending the response,
> > > check if the request has violated a quota. If so, then enqueue the
> > > response into a DelayQueue. All responses can share the same
> > > DelayQueue. Send those responses out after the delay has been met.
> > >
> > > There are some benefits to doing this:
> > >
> > > - We will eventually want to quota other requests as well. The above
> > >   seems to be a clean staged approach that should work uniformly for
> > >   all requests. i.e., parse request -> handle request normally ->
> > >   check quota -> hold in delay queue if quota violated -> respond .
> > >   All requests can share the same DelayQueue. (In contrast with the
> > >   current proposal we could end up with a bunch of purgatories, or a
> > >   combination of purgatories and delay queues.)
> > > - Since this approach does not need any fundamental modifications to
> > >   the current request handling, it addresses the caveats that Adi
> > >   noted (which is holding producer requests/fetch requests longer than
> > >   strictly necessary if quota is violated since the proposal was to
> > >   not watch on keys in that case). Likewise it addresses the caveat
> > >   that Guozhang noted (we may return no error if the request is held
> > >   long enough due to quota violation and satisfy a producer request
> > >   that may have in fact exceeded the ack timeout) although it is
> > >   probably reasonable to hide this case from the user.
> > > - By avoiding the caveats it also avoids the suggested work-around to
> > >   the caveats which is effectively to add a min-hold-time to the
> > >   purgatory. Although this is not a lot of code, I think it adds a
> > >   quota-driven feature to the purgatory which is already non-trivial
> > >   and should ideally remain unassociated with quota enforcement.
> > >
> > > For this t

Re: Review Request 30196: Patch for KAFKA-1886

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30196/
---

(Updated April 28, 2015, 5:27 p.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-1886
https://issues.apache.org/jira/browse/KAFKA-1886


Repository: kafka


Description (updated)
---

Fixing KAFKA-1886


Diffs (updated)
-

  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
cbef84ac76e62768981f74e71d451f2bda995275 

Diff: https://reviews.apache.org/r/30196/diff/


Testing
---

Added an integration test to PrimitiveAPITest.scala.


Thanks,

Aditya Auradkar

Re: Review Request 30196: Patch for KAFKA-1886

[Aditya Auradkar]

> On Feb. 7, 2015, 4:22 p.m., Neha Narkhede wrote:
> > core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala, line 295
> > <https://reviews.apache.org/r/30196/diff/2/?file=844610#file844610line295>
> >
> > Why do you need the sleep here? We try to avoid blindly sleeping in 
> > Kafka tests since it almost always leads to transient test failures. 
> > Consider using TestUtils.waitUntilTrue().
> 
> Aditya Auradkar wrote:
> Thanks Neha. I missed this review comment.
> 
> I agree sleeping isn't ideal here but I don't think there is a condition 
> I can wait on to trigger this specific exception. The client has to be 
> waiting on a response from the server. I'm not even sure that this testcase 
> needs to exist in PrimitiveApiTest since it isn't testing an API. Can you 
> suggest a better place to put it, if it makes sense to keep it at all?
> 
> Neha Narkhede wrote:
> Yeah, if we can't make a fullproof unit test, let's remove it. We are 
> really trying to reduce the number of randomly failing unit tests.

Hey Neha,
I've removed the test.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30196/#review71556
---


On April 28, 2015, 5:28 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30196/
> ---
> 
> (Updated April 28, 2015, 5:28 p.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-1886
> https://issues.apache.org/jira/browse/KAFKA-1886
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Fixing KAFKA-1886
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
> cbef84ac76e62768981f74e71d451f2bda995275 
> 
> Diff: https://reviews.apache.org/r/30196/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: Review Request 30196: Patch for KAFKA-1886

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30196/
---

(Updated April 28, 2015, 5:28 p.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-1886
https://issues.apache.org/jira/browse/KAFKA-1886


Repository: kafka


Description
---

Fixing KAFKA-1886


Diffs
-

  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
cbef84ac76e62768981f74e71d451f2bda995275 

Diff: https://reviews.apache.org/r/30196/diff/


Testing (updated)
---


Thanks,

Aditya Auradkar

[DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

Hey everyone,

Wrote up a KIP to update topic, client and broker configs dynamically via 
Zookeeper.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration

Please read and provide feedback.

Thanks,
Aditya

PS: I've intentionally kept this discussion separate from KIP-5 since I'm not 
sure if that is actively being worked on and I wanted to start with a clean 
slate.

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

Hey Gwen,

Thanks for the feedback. As Joel said, these client configs do not introduce a 
producer/consumer zk dependency. It is configuration that is needed by the 
broker.

>From your comments, I gather that you are more worried about managing broker 
>internal configs via Zookeeper since we already have a file. So why have two 
>mechanisms? Given that we already manage topic specific configuration in ZK, 
>it seems a good fit to at least have client configs there since these config 
>parameters aren't really driven through a file anyway. It also maintains 
>consistency. 

Even for broker configs, it seems very consistent to have all the overridden 
configs in one place which is easy to view and change. As you mentioned user's 
shouldn't ever have to fiddle with Zookeeper directly, our tooling should 
provide the ability to view and modify configs on a per-broker basis. I do like 
your suggestion of reloading config files but I'm not sure this works easily 
for everyone. For example, often these per-host overrides in config files are 
managed by hostname but what we really want are broker level overrides which 
means that it should ideally be tied to a broker-id which is a Kafka detail. In 
addition, sometimes these configs pushed to individual hosts aren't the 
properties files themselves.. rather some company specific stuff that also 
contains the Kafka configs. I guess the point I'm trying to make is that people 
may not be able to reload configs directly from file without doing some 
additional work in many cases.

As far as propogating configuration changes, perhaps I can clarify this section 
a bit more. Also, we can also do a pass over all the configs in KafkaConfig and 
have a list of properties that can be converted slowly.

Thanks,
Aditya


From: Joel Koshy [jjkosh...@gmail.com]
Sent: Thursday, April 30, 2015 5:14 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management

>1. I have deep concerns about managing configuration in ZooKeeper.
>First, Producers and Consumers shouldn't depend on ZK at all, this seems
>to add back a dependency we are trying to get away from.

The KIP probably needs to be clarified here - I don't think Aditya was
referring to client (producer/consumer) configs. These are global
client-id-specific configs that need to be managed centrally.
(Specifically, quota overrides on a per-client basis).

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

Hey everyone,

Thanks for the comments. I'll respond to each one-by-one. In the meantime, can 
we put this on the agenda for the KIP hangout for next week?

Thanks,
Aditya


From: Neha Narkhede [n...@confluent.io]
Sent: Sunday, May 03, 2015 9:48 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management

Thanks for starting this discussion, Aditya. Few questions/comments

1. If you change the default values like it's mentioned in the KIP, do you
also overwrite the local config file as part of updating the default value?
If not, where does the admin look to find the default values, ZK or local
Kafka config file? What if a config value is different in both places?

2. I share Gwen's concern around making sure that popular config management
tools continue to work with this change. Would love to see how each of
those would work with the proposal in the KIP. I don't know enough about
each of the tools but seems like in some of the tools, you have to define
some sort of class with parameter names as config names. How will such
tools find out about the config values? In Puppet, if this means that each
Puppet agent has to read it from ZK, this means the ZK port has to be open
to pretty much every machine in the DC. This is a bummer and a very
confusing requirement. Not sure if this is really a problem or not (each of
those tools might behave differently), though pointing out that this is
something worth paying attention to.

3. The wrapper tools that let users read/change config tools should not
depend on ZK for the reason mentioned above. It's a pain to assume that the
ZK port is open from any machine that needs to run this tool. Ideally what
users want is a REST API to the brokers to change or read the config (ala
Elasticsearch), but in the absence of the REST API, we should think if we
can write the tool such that it just requires talking to the Kafka broker
port. This will require a config RPC.

4. Not sure if KIP is the right place to discuss the design of propagating
the config changes to the brokers, but have you thought about just letting
the controller oversee the config changes and propagate via RPC to the
brokers? That way, there is an easier way to express config changes that
require all brokers to change it for it to be called complete. Maybe this
is not required, but it is hard to say if we don't discuss the full set of
configs that need to be dynamic.

Thanks,
Neha

On Fri, May 1, 2015 at 12:53 PM, Jay Kreps  wrote:

> Hey Aditya,
>
> This is a great! A couple of comments:
>
> 1. Leaving the file config in place is definitely the least disturbance.
> But let's really think about getting rid of the files and just have one
> config mechanism. There is always a tendency to make everything pluggable
> which so often just leads to two mediocre solutions. Can we do the exercise
> of trying to consider fully getting rid of file config and seeing what goes
> wrong?
>
> 2. Do we need to model defaults? The current approach is that if you have a
> global config x it is overridden for a topic xyz by /topics/xyz/x, and I
> think this could be extended to /brokers/0/x. I think this is simpler. We
> need to specify the precedence for these overrides, e.g. if you override at
> the broker and topic level I think the topic level takes precedence.
>
> 3. I recommend we have the producer and consumer config just be an override
> under client.id. The override is by client id and we can have separate
> properties for controlling quotas for producers and consumers.
>
> 4. Some configs can be changed just by updating the reference, others may
> require some action. An example of this is if you want to disable log
> compaction (assuming we wanted to make that dynamic) we need to call
> shutdown() on the cleaner. I think it may be required to register a
> listener callback that gets called when the config changes.
>
> 5. For handling the reference can you explain your plan a bit? Currently we
> have an immutable KafkaConfig object with a bunch of vals. That or
> individual values in there get injected all over the code base. I was
> thinking something like this:
> a. We retain the KafkaConfig object as an immutable object just as today.
> b. It is no longer legit to grab values out fo that config if they are
> changeable.
> c. Instead of making KafkaConfig itself mutable we make KafkaConfiguration
> which has a single volatile reference to the current KafkaConfig.
> KafkaConfiguration is what gets passed into various components. So to
> access a config you do something like config.instance.myValue. When the
> config changes the config manager updates this reference.
> d. The KafkaConfiguration is the thing that allows doing the
> configuration.onChange("my.config", callback)
>
> -Jay

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

Hey Joe,

Can you elaborate what you mean by a stop the world change? In this protocol, 
we can target notifications to a subset of brokers in the cluster (controller 
if we need to). Is the AdminChangeNotification a ZK notification or a request 
type exposed by each broker? 

Thanks,
Aditya


From: Joe Stein [joe.st...@stealth.ly]
Sent: Friday, May 01, 2015 5:25 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management

Hi Aditya, thanks for the write up and focusing on this piece.

Agreed we need something that we can do broker changes dynamically without
rolling restarts.

I think though if every broker is getting changes it with notifications it
is going to limit which configs can be dynamic.

We could never deliver a "stop the world" configuration change because then
that would happen on the entire cluster to every broker on the same time.

Can maybe just the controller get the notification?

And we provide a layer for brokers to work with the controller to-do the
config change operations at is discretion (so it can stop things if needs).

controller gets notification, sends AdminChangeNotification to broker [X ..
N] then brokers can do their things, even send a response for heartbeating
while it takes the few milliseconds it needs or crashes. We need to go
through both scenarios.

I am worried we put this change in like this and it works for quotas and
maybe a few other things but nothing else gets dynamic and we don't get far
enough for almost no more rolling restarts.

~ Joe Stein
- - - - - - - - - - - - - - - - -

  http://www.stealth.ly
- - - - - - - - - - - - - - - - -

On Thu, Apr 30, 2015 at 8:14 PM, Joel Koshy  wrote:

> >1. I have deep concerns about managing configuration in ZooKeeper.
> >First, Producers and Consumers shouldn't depend on ZK at all, this
> seems
> >to add back a dependency we are trying to get away from.
>
> The KIP probably needs to be clarified here - I don't think Aditya was
> referring to client (producer/consumer) configs. These are global
> client-id-specific configs that need to be managed centrally.
> (Specifically, quota overrides on a per-client basis).
>
>

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

 that config if they are
changeable.
c. Instead of making KafkaConfig itself mutable we make KafkaConfiguration
which has a single volatile reference to the current KafkaConfig.
KafkaConfiguration is what gets passed into various components. So to
access a config you do something like config.instance.myValue. When the
config changes the config manager updates this reference.
d. The KafkaConfiguration is the thing that allows doing the
configuration.onChange("my.config", callback)

-Jay

On Tue, Apr 28, 2015 at 3:57 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey everyone,
>
> Wrote up a KIP to update topic, client and broker configs dynamically via
> Zookeeper.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
>
> Please read and provide feedback.
>
> Thanks,
> Aditya
>
> PS: I've intentionally kept this discussion separate from KIP-5 since I'm
> not sure if that is actively being worked on and I wanted to start with a
> clean slate.
>

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

1. Essentially, I think removing the option to configure properties via file 
may be a big change for everyone. Having said that, your points are very valid. 
I guess we can discuss this a bit more during the KIP hangout. 

4. Yes, we will need to make some changes to update the MetricConfig for any 
metric. I left it out because I felt it wasn't strictly related to the KIP. 

Thanks for the nice summary on the implementation breakdown. Basically, the KIP 
should provide a uniform mechanism to change any type of config dynamically but 
the work to actually convert configs can be out of scope.  

Thanks,
Aditya


From: Jay Kreps [jay.kr...@gmail.com]
Sent: Monday, May 04, 2015 2:00 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management

Hey Aditya,

1. I would argue for either staying with what we have or else moving to a
better solution, but not doing both. A solution that uses both is going to
be quite complex to figure out what is configured and where it comes from.
If you think this is needed let's try to construct the argument for why it
is needed. Like in the workflow you described think how confusing that will
be--the first time the broker starts it uses the file, but then after that
if you change the file nothing happens because it has copied the file into
ZK. Instead let's do the research to figure out why people would object to
a pure-zk solution and then see if we can't address all those concerns.

4. I think it is fine to implement this in a second phase, but I think you
will need it even to be able to update the MetricConfigs to execute the
quota change, right?

Not sure if I follow your question on implementation. I think what you are
saying might be doing something like this as a first pass
a. Generalize the TopicConfigManager to handle any type of config override
as described in the doc (broker, topic, client, etc)
b. Implement the concept of mutable configs in ConfigDef
c. Add the client-level overrides for quotas and make sure the topic
overrides all still work
d. Not actually do the work to make most of the broker configs mutable
since that will involve passing around the KafkaConfiguration object much
more broadly in places where we have static wiring now. This would be done
as-needed as we made more configs dynamic.

Is that right?

Personally I think that makes sense. The only concern is just that we get
to a good stopping point so that people aren't left in some half-way state
that we end up redoing in the next release. So I think getting to a final
state with the configuration "infrastructure" is important, but actually
making all the configs mutable can be done gradually.

-Jay

On Mon, May 4, 2015 at 1:31 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey Jay,
>
> Thanks for the feedback.
>
> 1. We can certainly discuss what it means to remove the file configuration
> as a thought exercise. However, is this something we want to do for real?
> IMO, we can remove file configuration by having all configs stored in
> zookeeper. The flow can be:
> - Broker starts and reads all the configs from ZK. (overrides)
> - Apply them on top of the defaults that are hardcoded within the broker.
> This should simulate file based config behavior as it is currently.
> - Potentially, we can write back all the merged configs to zookeeper
> (defaults + overrides). This means that the entire config of that broker is
> in ZK.
>
> Thoughts?
>
> 2. Good point. All overridden configs (topic, client level) will have a
> corresponding broker config that serves as a default. It should be
> sufficient to change that broker config dynamically and that effectively
> means that the default has been changed. The overrides on a per
> topic/client basis still take precedence. So yeah, I don't think we need to
> model defaults explicitly. Using an example to be sure we are on the same
> page, lets say we wanted to increase the log retention time for all topics
> without having to create a  separate override for each topic, we could
> simply change the "log.retention.time" under "/brokers/" to the
> desired value and that should change the default log retention for everyone
> (apart from the explicitly overridden ones on a per-topic basis).
>
> 3. I thought it was cleaner to simply separate the producer and consumer
> configs but I guess if they present the same clientId, they are essentially
> the "same" client. I'll follow your suggestion.
>
> 4. Interesting that you mention this. I actually thought about having
> callbacks but I left it out of the initial proposal since I wanted to keep
> it relatively simple. The only configs we can change by checking references
> are the ones we check frequently while processing requests or (or something
>

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

d all over the code base. I was
> thinking something like this:
> a. We retain the KafkaConfig object as an immutable object just as today.
> b. It is no longer legit to grab values out fo that config if they are
> changeable.
> c. Instead of making KafkaConfig itself mutable we make KafkaConfiguration
> which has a single volatile reference to the current KafkaConfig.
> KafkaConfiguration is what gets passed into various components. So to
> access a config you do something like config.instance.myValue. When the
> config changes the config manager updates this reference.
> d. The KafkaConfiguration is the thing that allows doing the
> configuration.onChange("my.config", callback)
>
> -Jay
>
> On Tue, Apr 28, 2015 at 3:57 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Hey everyone,
> >
> > Wrote up a KIP to update topic, client and broker configs dynamically via
> > Zookeeper.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> >
> > Please read and provide feedback.
> >
> > Thanks,
> > Aditya
> >
> > PS: I've intentionally kept this discussion separate from KIP-5 since I'm
> > not sure if that is actively being worked on and I wanted to start with a
> > clean slate.
> >
>



--
Thanks,
Neha

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

> On May 4, 2015, 3:46 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/ClientQuotaMetrics2.scala, lines 153-162
> > <https://reviews.apache.org/r/33049/diff/5/?file=938427#file938427line153>
> >
> > For measuring the amount of throtting, would it be better to measure it 
> > as a percentage of the total time? We can just measure this as a rate and 
> > keep recording the amount of delayed time. The rate then tells us the 
> > fraction of the total time when throtting happens. This is probably simpler 
> > and more intuitive than measuring a separate max and avg.

Good suggestion. I'll incorporate into the review


> On May 4, 2015, 3:46 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/ClientQuotaMetrics2.scala, line 115
> > <https://reviews.apache.org/r/33049/diff/5/?file=938427#file938427line115>
> >
> > Instead of putting both the apiKey and clientId in a single string as 
> > the name, we probably should make use of tags in MetricName to be 
> > consistent with the rest of the usage.

I'm using this as a sensor name which must be a unique string (doesn't have any 
tags). The MetricName does have tags and I'm them while creating the metrics 
within a sensor.


> On May 4, 2015, 3:46 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/ClientQuotaMetrics2.scala, lines 82-95
> > <https://reviews.apache.org/r/33049/diff/5/?file=938427#file938427line82>
> >
> > Perhaps we should handle the delaying logic here too. I was thinking 
> > that we can pass in a callback in record(). If the quota is violated, we 
> > will register the callback in a delayed queue.

Interesting thought. So basically, something like this? 
try {
  // record metric
  // trigger callback
}
catch {
  case QuotaViolationException =>
  // insert in delayed queue
}


I'm currently managing the callbacks inside KafkaApis in my current patch. Take 
a look at that(once I publish it) and let me know which one you prefer. I open 
to doing it either way.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review82382
---


On April 28, 2015, 12:38 a.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated April 28, 2015, 12:38 a.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> WIP: First patch for quotas. 
> Changes are 
> 1. Adding per-client throttle time and quota metrics in 
> ClientQuotaMetrics.scala 
> 2. Making changes in QuotaViolationException and Sensor to return delay time 
> changes. 
> 3. Added configuration needed so far for quotas in KafkaConfig. 
> 4. Unit tests This is currently not being used anywhere in the code because I 
> haven't yet figured out how to enforce delays i.e. purgatory vs delay queue. 
> I'll have a better idea once I look at the new purgatory implementation.
> 
> This is currently not being used anywhere in the code because I haven't yet 
> figured out how to enforce delays i.e. purgatory vs delay queue. Hopefully, 
> this smaller patch is easier to review.
> 
> 
> Please read: This patch has 2 approaches for managing quotas in 
> ClientQuotaMetrics and CLientQuotaMetrics2 along with some example usage in 
> ReplicaManager. This code will have to be cleaned up significantly in order 
> to commit but I'm looking for feedback on which approach to use.
> 
> Approach 1: ClientQuotaMetrics wraps everything into a single class. Adding 
> new metrics is much clumsier.
> Approach 2: ClientQuotaMetrics2 only maintains per-client metrics for a 
> single entity (producer, consumer) etc.. This makes the code easier to use. 
> For throttling on a new dimention i.e. request per second, we only need to 
> create this object with a new quota and will just work.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
> dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
>   clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 
> b3d3d7c56acb445be16a3fbe00f05eaba659be46 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/commo

Re: Review Request 33049: Patch for KAFKA-1984

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 5, 2015, 2:52 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Summary (updated)
-

Patch for KAFKA-1984


Bugs: KAFKA-1984 and KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-1984
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-1984

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 5, 2015, 4:33 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 5, 2015, 4:33 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Summary (updated)
-

Patch for KAFKA-2084


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 5, 2015, 4:36 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Updated patch for quotas. This patch does the following: 
1. Add per-client metrics for both producer and consumers 
2. Add configuration for quotas 
3. Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 
4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
request. Implemented request throttling for produce and fetch requests. 
5. Added unit and integration test cases. I've not yet added integration 
testcases testing the consumer delays.. will update the patch once those are 
ready


Incorporated Jun's comments


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 5, 2015, 10:27 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Adding javadoc


KAFKA-2084 - Moved the callbacks to ClientQuotaMetrics


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 5, 2015, 10:29 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Updated patch for quotas. This patch does the following:
1. Add per-client metrics for both producer and consumers 
2. Add configuration for quotas 
3. Compute delay times in the metrics package and return the delay times in 
QuotaViolationException
4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
request. Implemented request throttling for produce and fetch requests. 
5. Added unit and integration test cases. I've not yet added integration 
testcases testing the consumer delays.. will update the patch once those are 
ready

Incorporated Jun's comments
KAFKA-2084 - Moved the callbacks to ClientQuotaMetrics


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 6, 2015, 12:52 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Adding javadoc


KAFKA-2084 - Moved the callbacks to ClientQuotaMetrics


Adding more configs


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 6, 2015, 12:52 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Updated patch for quotas. This patch does the following:
1. Add per-client metrics for both producer and consumers 
2. Add configuration for quotas 
3. Compute delay times in the metrics package and return the delay times in 
QuotaViolationException
4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
request. Implemented request throttling for produce and fetch requests. 
5. Added unit and integration test cases. I've not yet added integration 
testcases testing the consumer delays.. will update the patch once those are 
ready

Incorporated Jun and Joel's comments
KAFKA-2084 - Moved the callbacks to ClientQuotaMetrics


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

(Updated May 7, 2015, 1:32 a.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description (updated)
---

Changes are:
- protocol changes to the fetch reuqest and response to return the 
throttle_time_ms to clients
- New producer/consumer metrics to expose the avg and max delay time for a 
client
- Test cases

For now the patch will publish a zero delay and return a response

Added more tests


Addressing Jun's comments


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
ef9dd5238fbc771496029866ece1d85db6d7b7a5 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
b2db91ca14bbd17fef5ce85839679144fff3f689 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
b038c15186c0cbcc65b59479324052498361b717 
  core/src/main/scala/kafka/api/FetchResponse.scala 
75aaf57fb76ec01660d93701a57ae953d877d81c 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
570b2da1d865086f9830aa919a49063abbbe574d 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
31a2639477bf66f9a05d2b9b07794572d7ec393b 
  core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
a439046e118b6efcc3a5a9d9e8acb79f85e40398 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/OffsetManager.scala 
18680ce100f10035175cc0263ba7787ab0f6a17a 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
5717165f2344823fabe8f7cfafae4bb8af2d949a 
  core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
f3ab3f4ff8eb1aa6b2ab87ba75f72eceb6649620 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing
---

New tests added


Thanks,

Aditya Auradkar

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

(Updated May 7, 2015, 1:36 a.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description (updated)
---

Changes are:
- protocol changes to the fetch reuqest and response to return the 
throttle_time_ms to clients
- New producer/consumer metrics to expose the avg and max delay time for a 
client
- Test cases

For now the patch will publish a zero delay and return a response

Added more tests


Addressing Jun's comments


Formatting changes


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
ef9dd5238fbc771496029866ece1d85db6d7b7a5 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
b2db91ca14bbd17fef5ce85839679144fff3f689 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
b038c15186c0cbcc65b59479324052498361b717 
  core/src/main/scala/kafka/api/FetchResponse.scala 
75aaf57fb76ec01660d93701a57ae953d877d81c 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
570b2da1d865086f9830aa919a49063abbbe574d 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
31a2639477bf66f9a05d2b9b07794572d7ec393b 
  core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
a439046e118b6efcc3a5a9d9e8acb79f85e40398 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/OffsetManager.scala 
18680ce100f10035175cc0263ba7787ab0f6a17a 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
5717165f2344823fabe8f7cfafae4bb8af2d949a 
  core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
f3ab3f4ff8eb1aa6b2ab87ba75f72eceb6649620 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing
---

New tests added


Thanks,

Aditya Auradkar

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

> On May 4, 2015, 4:51 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/api/FetchResponse.scala, lines 152-153
> > <https://reviews.apache.org/r/33378/diff/1/?file=937083#file937083line152>
> >
> > This is tricky since FetchRequest is used in the follower as well. When 
> > doing a rolling upgrade of the broker to 0.8.3, we have to follow the 
> > following steps.
> > 1. Configure intra.cluster.protocol to 0.8.2 and rolling upgrade each 
> > broker to 0.8.3. After this step, each broker understands version 1 of the 
> > fetch request, but still sends fetch request in version 0.
> > 2. Configure intra.cluster.protocol to 0.8.3 and restart each broker. 
> > After this step, every broker will start sending fetch request in version 1.
> > 
> > So, we need the logic in ReplicaFetcherThread to issue the right 
> > version of the FetchRequest according to intra.cluster.protocol. We also 
> > need to read the response according to the request version (i.e., can't 
> > just assume the response is always on the latest version).

Good point. I'm passing in the protocol version to use on the fetchRequest to 
the AbstractFetcherThread. Also, not reading the response based on the request 
version.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/#review82389
---


On May 7, 2015, 1:36 a.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33378/
> ---
> 
> (Updated May 7, 2015, 1:36 a.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2136
> https://issues.apache.org/jira/browse/KAFKA-2136
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Changes are:
> - protocol changes to the fetch reuqest and response to return the 
> throttle_time_ms to clients
> - New producer/consumer metrics to expose the avg and max delay time for a 
> client
> - Test cases
> 
> For now the patch will publish a zero delay and return a response
> 
> Added more tests
> 
> 
> Addressing Jun's comments
> 
> 
> Formatting changes
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java
>  ef9dd5238fbc771496029866ece1d85db6d7b7a5 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> b2db91ca14bbd17fef5ce85839679144fff3f689 
>   clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
> 3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
>   clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
> 8686d83aa52e435c6adafbe9ff4bd1602281072a 
>   clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
> eb8951fba48c335095cc43fc3672de1c733e07ff 
>   clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
> fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
>   clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
> 37ec0b79beafcf5735c386b066eb319fb697eff5 
>   
> clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
>  419541011d652becf0cda7a5e62ce813cddb1732 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
>   
> clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java
>  e3cc1967e407b64cc734548c19e30de700b64ba8 
>   core/src/main/scala/kafka/api/FetchRequest.scala 
> b038c15186c0cbcc65b59479324052498361b717 
>   core/src/main/scala/kafka/api/FetchResponse.scala 
> 75aaf57fb76ec01660d93701a57ae953d877d81c 
>   core/src/main/scala/kafka/api/ProducerRequest.scala 
> 570b2da1d865086f9830aa919a49063abbbe574d 
>   core/src/main/scala/kafka/api/ProducerResponse.scala 
> 5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
>   core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
> 31a2639477bf66f9a05d2b9b07794572d7ec393b 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> a439046e118b6efcc3a5a9d9e8acb79f85e40398 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> de6cf5bdaa0e70394162febc63b50b55ca0a92db 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 05078b24ef28f2f4e099afa943e43f1d00359fda 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 417960dd1ab407ebebad8fdb0e97415db3e91a2f 
>   core/s

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

ed that Chef integration could still
> > work
> > > if
> > > > > all configs are moved to ZK. My rough understanding of how Chef
> works
> > > is
> > > > > that a user first registers a service host with a Chef server.
> After
> > > > that,
> > > > > a Chef client will be run on the service host. The user can then
> push
> > > > > config changes intended for a service/host to the Chef server. The
> > > server
> > > > > is then responsible for pushing the changes to Chef clients. Chef
> > > clients
> > > > > support pluggable logic. For example, it can generate a config file
> > > that
> > > > > Kafka broker will take. If we move all configs to ZK, we can
> > customize
> > > > the
> > > > > Chef client to use our config CLI to make the config changes in
> > Kafka.
> > > In
> > > > > this model, one probably doesn't need to register every broker in
> > Chef
> > > > for
> > > > > the config push. Not sure if Puppet works in a similar way.
> > > > >
> > > > > Also for storing the configs, we probably can't store the
> > broker/global
> > > > > level configs in Kafka itself (e.g. in a special topic). The reason
> > is
> > > > that
> > > > > in order to start a broker, we likely need to make some broker
> level
> > > > config
> > > > > changes (e.g., the default log.dir may not be present, the default
> > port
> > > > may
> > > > > not be available, etc). If we need a broker to be up to make those
> > > > changes,
> > > > > we get into this chicken and egg problem.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, May 5, 2015 at 4:14 PM, Gwen Shapira <
> gshap...@cloudera.com 
> > > >
> > > > > wrote:
> > > > >
> > > > > > Sorry I missed the call today :)
> > > > > >
> > > > > > I think an additional requirement would be:
> > > > > > Make sure that traditional deployment tools (Puppet, Chef, etc)
> are
> > > > still
> > > > > > capable of managing Kafka configuration.
> > > > > >
> > > > > > For this reason, I'd like the configuration refresh to be pretty
> > > close
> > > > to
> > > > > > what most Linux services are doing to force a reload of
> > > configuration.
> > > > > > AFAIK, this involves handling HUP signal in the main thread to
> > reload
> > > > > > configuration. Then packaging scripts can add something nice like
> > > > > "service
> > > > > > kafka reload".
> > > > > >
> > > > > > (See Apache web server:
> > > > > >
> > https://github.com/apache/httpd/blob/trunk/build/rpm/httpd.init#L101
> > > )
> > > > > >
> > > > > > Gwen
> > > > > >
> > > > > >
> > > > > > On Tue, May 5, 2015 at 8:54 AM, Joel Koshy  
> > > >
> > > > wrote:
> > > > > >
> > > > > > > Good discussion. Since we will be talking about this at 11am, I
> > > > wanted
> > > > > > > to organize these comments into requirements to see if we are
> all
> > > on
> > > > > > > the same page.
> > > > > > >
> > > > > > > REQUIREMENT 1: Needs to accept dynamic config changes. This
> needs
> > > to
> > > > > > > be general enough to work for all configs that we envision may
> > need
> > > > to
> > > > > > > accept changes at runtime. e.g., log (topic), broker, client
> > > > (quotas),
> > > > > > > etc.. possible options include:
> > > > > > > - ZooKeeper watcher
> > > > > > > - Kafka topic
> > > > > > > - Direct RPC to controller (or config coordinator)
> > > > > > >
> > > > > > > The current KIP is really focused on REQUIREMENT 1 and I think
> > that
> > > > is
> > > > > > > reasonable as long as we don't come up with something that
> > requires
> > > > > > > significa

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

(Updated May 11, 2015, 9:51 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description (updated)
---

Fixing bug


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
ef9dd5238fbc771496029866ece1d85db6d7b7a5 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
b2db91ca14bbd17fef5ce85839679144fff3f689 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
b038c15186c0cbcc65b59479324052498361b717 
  core/src/main/scala/kafka/api/FetchResponse.scala 
75aaf57fb76ec01660d93701a57ae953d877d81c 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
570b2da1d865086f9830aa919a49063abbbe574d 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
31a2639477bf66f9a05d2b9b07794572d7ec393b 
  core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
a439046e118b6efcc3a5a9d9e8acb79f85e40398 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/OffsetManager.scala 
18680ce100f10035175cc0263ba7787ab0f6a17a 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
5717165f2344823fabe8f7cfafae4bb8af2d949a 
  core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
f3ab3f4ff8eb1aa6b2ab87ba75f72eceb6649620 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing
---

New tests added


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 11, 2015, 11:16 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Adding javadoc


KAFKA-2084 - Moved the callbacks to ClientQuotaMetrics


Adding more configs


Don't quota replica traffic


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 11, 2015, 11:17 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Updated patch for quotas. This patch does the following: 
1. Add per-client metrics for both producer and consumers 
2. Add configuration for quotas 
3. Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 
4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
request. Implemented request throttling for produce and fetch requests. 
5. Added unit and integration test cases.
6. This doesn't include a system test. There is a separate ticket for that


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b7d2a2842e17411a823b93bdedc84657cbd62be1 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

fka broker will take. If we move all configs to ZK, we can
> > > customize
> > > > > the
> > > > > > Chef client to use our config CLI to make the config changes in
> > > Kafka.
> > > > In
> > > > > > this model, one probably doesn't need to register every broker in
> > > Chef
> > > > > for
> > > > > > the config push. Not sure if Puppet works in a similar way.
> > > > > >
> > > > > > Also for storing the configs, we probably can't store the
> > > broker/global
> > > > > > level configs in Kafka itself (e.g. in a special topic). The reason
> > > is
> > > > > that
> > > > > > in order to start a broker, we likely need to make some broker
> > level
> > > > > config
> > > > > > changes (e.g., the default log.dir may not be present, the default
> > > port
> > > > > may
> > > > > > not be available, etc). If we need a broker to be up to make those
> > > > > changes,
> > > > > > we get into this chicken and egg problem.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Tue, May 5, 2015 at 4:14 PM, Gwen Shapira <
> > gshap...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Sorry I missed the call today :)
> > > > > > >
> > > > > > > I think an additional requirement would be:
> > > > > > > Make sure that traditional deployment tools (Puppet, Chef, etc)
> > are
> > > > > still
> > > > > > > capable of managing Kafka configuration.
> > > > > > >
> > > > > > > For this reason, I'd like the configuration refresh to be pretty
> > > > close
> > > > > to
> > > > > > > what most Linux services are doing to force a reload of
> > > > configuration.
> > > > > > > AFAIK, this involves handling HUP signal in the main thread to
> > > reload
> > > > > > > configuration. Then packaging scripts can add something nice like
> > > > > > "service
> > > > > > > kafka reload".
> > > > > > >
> > > > > > > (See Apache web server:
> > > > > > >
> > > https://github.com/apache/httpd/blob/trunk/build/rpm/httpd.init#L101
> > > > )
> > > > > > >
> > > > > > > Gwen
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 5, 2015 at 8:54 AM, Joel Koshy 
> > > > > wrote:
> > > > > > >
> > > > > > > > Good discussion. Since we will be talking about this at 11am, I
> > > > > wanted
> > > > > > > > to organize these comments into requirements to see if we are
> > all
> > > > on
> > > > > > > > the same page.
> > > > > > > >
> > > > > > > > REQUIREMENT 1: Needs to accept dynamic config changes. This
> > needs
> > > > to
> > > > > > > > be general enough to work for all configs that we envision may
> > > need
> > > > > to
> > > > > > > > accept changes at runtime. e.g., log (topic), broker, client
> > > > > (quotas),
> > > > > > > > etc.. possible options include:
> > > > > > > > - ZooKeeper watcher
> > > > > > > > - Kafka topic
> > > > > > > > - Direct RPC to controller (or config coordinator)
> > > > > > > >
> > > > > > > > The current KIP is really focused on REQUIREMENT 1 and I think
> > > that
> > > > > is
> > > > > > > > reasonable as long as we don't come up with something that
> > > requires
> > > > > > > > significant re-engineering to support the other requirements.
> > > > > > > >
> > > > > > > > REQUIREMENT 2: Provide consistency of configs across brokers
> > > > (modulo
> > > > > > > > per-broker overrides) or at least be able to verify
> > consistency.
> > > > > What

Re: Review Request 34070: Patch for KAFKA-2186

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34070/#review83343
---

Ship it!



core/src/main/scala/kafka/javaapi/consumer/ConsumerConnector.java
<https://reviews.apache.org/r/34070/#comment134315>

How does this work if the consumer doesn't own these partitions? Is it 
possible to commit offsets for any topic? Just curious..


- Aditya Auradkar


On May 12, 2015, 1:39 a.m., Jiangjie Qin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34070/
> ---
> 
> (Updated May 12, 2015, 1:39 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2186
> https://issues.apache.org/jira/browse/KAFKA-2186
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Patch for KAFKA-2186 follow-up patch of KAFKA-1650, add the missing offset 
> commit with map in java api
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/javaapi/consumer/ConsumerConnector.java 
> cc3400ff81fc0db69b5129ad7b440f20a211a79d 
> 
> Diff: https://reviews.apache.org/r/34070/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jiangjie Qin
> 
>

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

> On May 12, 2015, 7:38 p.m., Dong Lin wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java, line 117
> > <https://reviews.apache.org/r/33049/diff/9/?file=955824#file955824line117>
> >
> > metric.value(timeMs), which translates to Rate.measure(config, timeMs), 
> > may return Infinity. This may introduce infinite delay.
> > 
> > I think this bug is rooted either in class Rate or  class SampledStat. 
> > In short, SampledStat.purgeObsoleteSamples will remove all Samples, such 
> > that SampledStat.oldest(now) == now.
> > 
> > Should I open a ticket and submit a patch for it?

Hey dong, yeah you should submit a patch for it.


> On May 12, 2015, 7:38 p.m., Dong Lin wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java, lines 
> > 122-127
> > <https://reviews.apache.org/r/33049/diff/9/?file=955824#file955824line122>
> >
> > The function quits on the first quota violaation, and calculate 
> > delayTime based on this violation. 
> > 
> > Should we enumerate all metric in this.metrics() and set delayTime of 
> > the QuotaViolationException to be the largest delayTime of all violatoins?

Also a good point. Perhaps you can fix it in the patch you will submit? For 
quotas, we effectively have only one quota per sensor, so it's ok (for now).


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review83441
---


On May 11, 2015, 11:17 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated May 11, 2015, 11:17 p.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Updated patch for quotas. This patch does the following: 
> 1. Add per-client metrics for both producer and consumers 
> 2. Add configuration for quotas 
> 3. Compute delay times in the metrics package and return the delay times in 
> QuotaViolationException 
> 4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
> request. Implemented request throttling for produce and fetch requests. 
> 5. Added unit and integration test cases.
> 6. This doesn't include a system test. There is a separate ticket for that
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
> dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
>  a451e5385c9eca76b38b425e8ac856b2715fcffe 
>   clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
> ca823fd4639523018311b814fde69b6177e73b97 
>   clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
>   core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 417960dd1ab407ebebad8fdb0e97415db3e91a2f 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> b7d2a2842e17411a823b93bdedc84657cbd62be1 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
>   core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> fc226c863095b7761290292cd8755cd7ad0f155c 
>   core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
> 8014a5a6c362785539f24eb03d77278434614fe6 
>   core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33049/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

(Updated May 12, 2015, 9:40 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description (updated)
---

Minor bug fix


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
ef9dd5238fbc771496029866ece1d85db6d7b7a5 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
b2db91ca14bbd17fef5ce85839679144fff3f689 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
b038c15186c0cbcc65b59479324052498361b717 
  core/src/main/scala/kafka/api/FetchResponse.scala 
75aaf57fb76ec01660d93701a57ae953d877d81c 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
570b2da1d865086f9830aa919a49063abbbe574d 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
31a2639477bf66f9a05d2b9b07794572d7ec393b 
  core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
83fc47417dd7fe4edf030217fa7fd69d99b170b0 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/OffsetManager.scala 
18680ce100f10035175cc0263ba7787ab0f6a17a 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
5717165f2344823fabe8f7cfafae4bb8af2d949a 
  core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
f3ab3f4ff8eb1aa6b2ab87ba75f72eceb6649620 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing
---

New tests added


Thanks,

Aditya Auradkar

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

> On May 12, 2015, 7:39 p.m., Dong Lin wrote:
> > core/src/main/scala/kafka/server/ReplicaManager.scala, line 428
> > <https://reviews.apache.org/r/33378/diff/4/?file=955709#file955709line428>
> >
> > Will readFromLocalLog() read data into memory before the cilent's quota 
> > is checked?

It shouldn't. I believe only FileMessageSet is passed around which only 
contains start and end pointers of file segments. The actual read is done later.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/#review83439
-------


On May 12, 2015, 9:40 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33378/
> ---
> 
> (Updated May 12, 2015, 9:40 p.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2136
> https://issues.apache.org/jira/browse/KAFKA-2136
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Minor bug fix
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java
>  ef9dd5238fbc771496029866ece1d85db6d7b7a5 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> b2db91ca14bbd17fef5ce85839679144fff3f689 
>   clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
> 3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
>   clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
> 8686d83aa52e435c6adafbe9ff4bd1602281072a 
>   clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
> eb8951fba48c335095cc43fc3672de1c733e07ff 
>   clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
> fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
>   clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
> 37ec0b79beafcf5735c386b066eb319fb697eff5 
>   
> clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
>  419541011d652becf0cda7a5e62ce813cddb1732 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
>   
> clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java
>  e3cc1967e407b64cc734548c19e30de700b64ba8 
>   core/src/main/scala/kafka/api/FetchRequest.scala 
> b038c15186c0cbcc65b59479324052498361b717 
>   core/src/main/scala/kafka/api/FetchResponse.scala 
> 75aaf57fb76ec01660d93701a57ae953d877d81c 
>   core/src/main/scala/kafka/api/ProducerRequest.scala 
> 570b2da1d865086f9830aa919a49063abbbe574d 
>   core/src/main/scala/kafka/api/ProducerResponse.scala 
> 5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
>   core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
> 31a2639477bf66f9a05d2b9b07794572d7ec393b 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> 83fc47417dd7fe4edf030217fa7fd69d99b170b0 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> de6cf5bdaa0e70394162febc63b50b55ca0a92db 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 05078b24ef28f2f4e099afa943e43f1d00359fda 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 417960dd1ab407ebebad8fdb0e97415db3e91a2f 
>   core/src/main/scala/kafka/server/OffsetManager.scala 
> 18680ce100f10035175cc0263ba7787ab0f6a17a 
>   core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
> b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
>   core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
> 5717165f2344823fabe8f7cfafae4bb8af2d949a 
>   core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
> f3ab3f4ff8eb1aa6b2ab87ba75f72eceb6649620 
>   core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
> 00d59337a99ac135e8689bd1ecd928f7b1423d79 
> 
> Diff: https://reviews.apache.org/r/33378/diff/
> 
> 
> Testing
> ---
> 
> New tests added
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

(Updated May 12, 2015, 9:42 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description (updated)
---

Changes are
- protocol changes to the fetch request and response to return the 
throttle_time_ms to clients
- New producer/consumer metrics to expose the avg and max delay time for a 
client
- Test cases.

For now the patch will publish a zero delay and return a response


Diffs
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
ef9dd5238fbc771496029866ece1d85db6d7b7a5 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
b2db91ca14bbd17fef5ce85839679144fff3f689 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
b038c15186c0cbcc65b59479324052498361b717 
  core/src/main/scala/kafka/api/FetchResponse.scala 
75aaf57fb76ec01660d93701a57ae953d877d81c 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
570b2da1d865086f9830aa919a49063abbbe574d 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
31a2639477bf66f9a05d2b9b07794572d7ec393b 
  core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
83fc47417dd7fe4edf030217fa7fd69d99b170b0 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
417960dd1ab407ebebad8fdb0e97415db3e91a2f 
  core/src/main/scala/kafka/server/OffsetManager.scala 
18680ce100f10035175cc0263ba7787ab0f6a17a 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
5717165f2344823fabe8f7cfafae4bb8af2d949a 
  core/src/test/scala/unit/kafka/server/DelayedOperationTest.scala 
f3ab3f4ff8eb1aa6b2ab87ba75f72eceb6649620 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing
---

New tests added


Thanks,

Aditya Auradkar

Re: Review Request 34170: Patch for KAFKA-2191

[Aditya Auradkar]

> On May 13, 2015, 5:14 p.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java, line 
> > 62
> > 
> >
> > Is this actually right? I agree you'll get discontinuities as the 
> > measured time shrinks to zero but why is giving back 0 the right answer? In 
> > the case you guys were testing 0 was safe, but imagine a case where the 
> > monitoring was checking that the value didn't fall below some threshold.
> 
> Dong Lin wrote:
> The question is, when Rate.measure() is called right after 
> Rate.record(n), what should be the return value? I think there are two 
> possibilities: 0 and n/config.timeWindowMs(). I didn't find any use case 
> where these two values make a difference. 
> 
> Which value do you think is the best?
> 
> Thank you.

I think returning 0 is reasonable if no time has elapsed (technically).
As an alternate solution, what if we assumed that "elapsed" is always 1 (at 
least). For example:

double elapsed = convert(now - stat.oldest(now).lastWindowMs) + 1

In case of seconds, this basically means that you assume the current second is 
always complete. This is only a problem (for a couple of seconds) when all 
previous samples have zero activity or when the server is just starting up.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34170/#review83629
---


On May 13, 2015, 10:32 p.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34170/
> ---
> 
> (Updated May 13, 2015, 10:32 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2191
> https://issues.apache.org/jira/browse/KAFKA-2191
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2191; Measured rate should not be infinite
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
> 98429da34418f7f1deba1b5e44e2e6025212edb3 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/stats/SampledStat.java 
> b341b7daaa10204906d78b812fb05fd27bc69373 
> 
> Diff: https://reviews.apache.org/r/34170/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>

Re: Review Request 34170: Patch for KAFKA-2191

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34170/#review83687
---



clients/src/main/java/org/apache/kafka/common/metrics/stats/SampledStat.java
<https://reviews.apache.org/r/34170/#comment134723>

I think this is a good catch.

Just so I understand, this is adding new samples for blank intervals of 
time when record was not called. As an optimization, I think you should only 
add "MetricConfig.samples" number of samples. In the rare case, there is no 
activity for 10 mins (say), this will add 10sec*6*10 = 600 samples which will 
be purged immediately on the next record call.


- Aditya Auradkar


On May 13, 2015, 10:32 p.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34170/
> ---
> 
> (Updated May 13, 2015, 10:32 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2191
> https://issues.apache.org/jira/browse/KAFKA-2191
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2191; Measured rate should not be infinite
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
> 98429da34418f7f1deba1b5e44e2e6025212edb3 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/stats/SampledStat.java 
> b341b7daaa10204906d78b812fb05fd27bc69373 
> 
> Diff: https://reviews.apache.org/r/34170/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>

Re: Review Request 34170: Patch for KAFKA-2191

[Aditya Auradkar]

> On May 13, 2015, 5:14 p.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java, line 
> > 62
> > <https://reviews.apache.org/r/34170/diff/1/?file=958215#file958215line62>
> >
> > Is this actually right? I agree you'll get discontinuities as the 
> > measured time shrinks to zero but why is giving back 0 the right answer? In 
> > the case you guys were testing 0 was safe, but imagine a case where the 
> > monitoring was checking that the value didn't fall below some threshold.
> 
> Dong Lin wrote:
> The question is, when Rate.measure() is called right after 
> Rate.record(n), what should be the return value? I think there are two 
> possibilities: 0 and n/config.timeWindowMs(). I didn't find any use case 
> where these two values make a difference. 
> 
> Which value do you think is the best?
> 
> Thank you.
> 
> Aditya Auradkar wrote:
> I think returning 0 is reasonable if no time has elapsed (technically).
> As an alternate solution, what if we assumed that "elapsed" is always 1 
> (at least). For example:
> 
> double elapsed = convert(now - stat.oldest(now).lastWindowMs) + 1
> 
> In case of seconds, this basically means that you assume the current 
> second is always complete. This is only a problem (for a couple of seconds) 
> when all previous samples have zero activity or when the server is just 
> starting up.
> 
> Jay Kreps wrote:
> Actually there is a fundamental issue with the computation that patching 
> around the 0 case doesn't fix. That is the instability of the estimate. This 
> is the "taking a poll with sample size one" problem.
> 
> Even if you patch the 0 case you still get a bad answer 1 ms later. That 
> is, let's say you get a single 50k request and your quota is 1MB/sec. 
> Currently at 0ms we estimate infinity which is in fact the measured rate but 
> obviously not a good estimate. But even 1 ms later the estimate is bad. 
> 50k*1000ms = ~50MB/sec.
> 
> This is somewhat rare because it only happens when there is just one 
> sample.
> 
> They key observation is that if a sample is missing, nothing happened in 
> that time period. But the calculation should still use that time period.
> 
> So the right way to compute it, I think, is actually
>   ellapsed = (num_samples-1)*window_size + (now - current_sample.begin)
>   
> For safety I think we should also require the number of samples to be >= 
> 2 and default it to 3.
> 
> Dong Lin wrote:
> Hi Jay: thanks for comments!
> 
> I think the problem here is that, when event number is very small (e.g. 
> 0, 1), what value should rate.measure() return, right? If I understand your 
> solution formula right, when there is only one sample, your measured rate is 
> n/ellapsed = n/(now - current_sample.begin), which is exactly same as the 
> current code. I agree that requiring number of samples to be >= 2 solve the 
> problem. But what happens when user call rate.measure() when there is only 1 
> sample?
> 
> I agree with you that we still get a bad answer if we patch the 0 case. 
> How about we patch the 0 to timeWindowMs case: if (ellapsed < timeWindowMs) 
> then ellapsed = timeWindowMs. Does this solve the problem here?
> 
> Dong Lin wrote:
> I think we all agree that there is problem with rate measure when 
> ellapsed time is too small. What we need to properly define the rate measure 
> to handle such case, which is currently missing.
> 
> If we want to require number of samples to be >= 2 as Jay suggested, we 
> can also let ellapsed = max(ellapsed, 2*timeWindowMs) and keep the rest of 
> the code, right?

If we require num samples greater than 2, we shoud simply change this condition 
in MetricConfig:
if (samples < 1)
throw new IllegalArgumentException("The number of samples must be 
at least 1.");

Jay's equation is now similar to Dongs.

If num samples = 3,
ellapsed = (num_samples-1)*window_size + (now - current_sample.begin)
ellapsed = 2*window_size + (now - current_sample.begin)


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34170/#review83629
---


On May 14, 2015, 7:34 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34170/
> ---
> 
&

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

t; > > 1. In line with pretty much any Linux service that exists,
> > so
> > > > > admins
> > > > > > > > > have a
> > > > > > > > > > lot of related experience.
> > > > > > > > > > 2. Much smaller change to our code-base, so easier to
> > patch,
> > > > > review
> > > > > > > and
> > > > > > > > > > test. Lower risk overall.
> > > > > > > > > >
> > > > > > > > > > Can you walk me over the benefits of using Zookeeper?
> > > > Especially
> > > > > > > since
> > > > > > > > it
> > > > > > > > > > looks like we can't get rid of the file entirely?
> > > > > > > > > >
> > > > > > > > > > Gwen
> > > > > > > > > >
> > > > > > > > > > On Thu, May 7, 2015 at 3:33 AM, Jun Rao  > > > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > One of the Chef users confirmed that Chef integration
> > could
> > > > > still
> > > > > > > > work
> > > > > > > > > if
> > > > > > > > > > > all configs are moved to ZK. My rough understanding of
> > how
> > > > Chef
> > > > > > > works
> > > > > > > > > is
> > > > > > > > > > > that a user first registers a service host with a Chef
> > > > server.
> > > > > > > After
> > > > > > > > > > that,
> > > > > > > > > > > a Chef client will be run on the service host. The user
> > can
> > > > > then
> > > > > > > push
> > > > > > > > > > > config changes intended for a service/host to the Chef
> > > > server.
> > > > > The
> > > > > > > > > server
> > > > > > > > > > > is then responsible for pushing the changes to Chef
> > > clients.
> > > > > Chef
> > > > > > > > > clients
> > > > > > > > > > > support pluggable logic. For example, it can generate a
> > > > config
> > > > > file
> > > > > > > > > that
> > > > > > > > > > > Kafka broker will take. If we move all configs to ZK, we
> > > can
> > > > > > > > customize
> > > > > > > > > > the
> > > > > > > > > > > Chef client to use our config CLI to make the config
> > > changes
> > > > in
> > > > > > > > Kafka.
> > > > > > > > > In
> > > > > > > > > > > this model, one probably doesn't need to register every
> > > > broker
> > > > > in
> > > > > > > > Chef
> > > > > > > > > > for
> > > > > > > > > > > the config push. Not sure if Puppet works in a similar
> > way.
> > > > > > > > > > >
> > > > > > > > > > > Also for storing the configs, we probably can't store the
> > > > > > > > broker/global
> > > > > > > > > > > level configs in Kafka itself (e.g. in a special topic).
> > > The
> > > > > reason
> > > > > > > > is
> > > > > > > > > > that
> > > > > > > > > > > in order to start a broker, we likely need to make some
> > > > broker
> > > > > > > level
> > > > > > > > > > config
> > > > > > > > > > > changes (e.g., the default log.dir may not be present,
> > the
> > > > > default
> > > > > > > > port
> > > > > > > > > > may
> > > > > > > > > > > not be available, etc). If we need a broker to be up to
> > > make
> > > > > those
> > > > > > > > > > changes,
> > > > > > > > > > > we get into this chicken and egg problem.
> > > > > > &g

RE: [DISCUSS] KIP-21 Configuration Management

[Aditya Auradkar]

Yes we did. I just overlooked that line.. cleaning it up now.

Aditya


From: Gwen Shapira [gshap...@cloudera.com]
Sent: Friday, May 15, 2015 12:55 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management

The wiki says:
"There will be 3 paths within config
/config/clients/
/config/topics/
/config/brokers/
Didn't we decide that brokers will not be configured dynamically, rather we
will keep the config in the file?

On Fri, May 15, 2015 at 10:46 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Updated the wiki to capture our recent discussions. Please read.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
>
> Thanks,
> Aditya
>
> 
> From: Joel Koshy [jjkosh...@gmail.com]
> Sent: Tuesday, May 12, 2015 1:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-21 Configuration Management
>
> The lack of audit could be addressed to some degree by an internal
> __config_changes topic which can have very long retention. Also, per
> the hangout summary that Gwen sent out it appears that we decided
> against supporting SIGHUP/dynamic configs for the broker.
>
> Thanks,
>
> Joel
>
> On Tue, May 12, 2015 at 11:05:06AM -0700, Neha Narkhede wrote:
> > Thanks for chiming in, Todd!
> >
> > Agree that the lack of audit and rollback is a major downside of moving
> all
> > configs to ZooKeeper. Being able to configure dynamically created
> entities
> > in Kafka is required though. So I think what Todd suggested is a good
> > solution to managing all configs - catching SIGHUP for broker configs and
> > storing dynamic configs in ZK like we do today.
> >
> > On Tue, May 12, 2015 at 10:30 AM, Jay Kreps  wrote:
> >
> > > Hmm, here is how I think we can change the split brain proposal to
> make it
> > > a bit better:
> > > 1. Get rid of broker overrides, this is just done in the config file.
> This
> > > makes the precedence chain a lot clearer (e.g. zk always overrides
> file on
> > > a per-entity basis).
> > > 2. Get rid of the notion of dynamic configs in ConfigDef and in the
> broker.
> > > All overrides are dynamic and all server configs are static.
> > > 3. Create an equivalent of LogConfig for ClientConfig and any future
> config
> > > type we make.
> > > 4. Generalize the TopicConfigManager to handle multiple types of
> overrides.
> > >
> > > What we haven't done is try to think through how the pure zk approach
> would
> > > work.
> > >
> > > -Jay
> > >
> > >
> > >
> > > On Mon, May 11, 2015 at 10:53 PM, Ashish Singh 
> > > wrote:
> > >
> > > > I agree with the Joel's suggestion on keeping broker's configs in
> > > > config file and clients/topics config in ZK. Few other projects,
> Apache
> > > > Solr for one, also does something similar for its configurations.
> > > >
> > > > On Monday, May 11, 2015, Gwen Shapira  wrote:
> > > >
> > > > > I like this approach (obviously).
> > > > > I am also OK with supporting broker re-read of config file based
> on ZK
> > > > > watch instead of SIGHUP, if we see this as more consistent with the
> > > rest
> > > > of
> > > > > our code base.
> > > > >
> > > > > Either is fine by me as long as brokers keep the file and just do
> > > refresh
> > > > > :)
> > > > >
> > > > > On Tue, May 12, 2015 at 2:54 AM, Joel Koshy  > > > > > wrote:
> > > > >
> > > > > > So the general concern here is the dichotomy of configs (which we
> > > > > > already have - i.e., in the form of broker config files vs topic
> > > > > > configs in zookeeper). We (at LinkedIn) had some discussions on
> this
> > > > > > last week and had this very question for the operations team
> whose
> > > > > > opinion is I think to a large degree a touchstone for this
> decision:
> > > > > > "Has the operations team at LinkedIn experienced any pain so far
> with
> > > > > > managing topic configs in ZooKeeper (while broker configs are
> > > > > > file-based)?" It turns out that ops overwhelmingly favors the
> current
> > > > > > approach. i.e., service configs as file-based configs and
> > > client/topic
> > > > > > c

[VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration

Aditya

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Yeah, that was just a typo. I've fixed it. Thanks for calling it out.

In KIP-4, I believe we have 3 types of requests: CreateTopic, AlterTopic and 
DeleteTopic. The topic configs are a sub-type of the Create and Alter commands. 
I think it would be nice to simply have a AlterConfig command that can alter 
any type of config rather than having a specific ClientConfig.

AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]]
ConfigType => string
AddedConfigEntry => ConfigKey ConfigValue
ConfigKey => string
ConfigValue => string
DeletedConfig => string

The downside of this approach is that we will have 2 separate ways of changing 
topic configs (AlterTopic and AlterConfig). While a general AlterConfig only 
makes sense if we plan to have more than two types of entity configs.. it's 
definitely more future proof. Thoughts?

Aditya


From: Todd Palino [tpal...@gmail.com]
Sent: Monday, May 18, 2015 12:39 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

Agree with Jun here on the JSON format. I think your intention was likely
to have actual JSON here and it was just a typo in the wiki?

-Todd

On Mon, May 18, 2015 at 12:07 PM, Jun Rao  wrote:

> Aditya,
>
> Another thing to consider. In KIP-4, we are adding a new RPC request to
> change and retrieve topic configs. Do we want to add a similar RPC request
> to change configs per client id? If so, do we want to introduce a separate
> new request or have a combined new request for both topic and client id
> level config changes?
>
> A minor point in the wiki, for the json format in ZK, we should change
> {X1=Y1,
> X2=Y2..} to a json map, right?
>
> Thanks,
>
> Jun
>
>
> On Mon, May 18, 2015 at 9:48 AM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> >
> > Aditya
> >
>

Re: Review Request 34418: Patch for KAFKA-2191

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34418/#review84344
---


Looks good. minor comment


clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java
<https://reviews.apache.org/r/34418/#comment135563>

It's still possible to configure the samples() in MetricConfig to 1. Should 
we change the assert to ensure we have at least 2 samples always.



clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java
<https://reviews.apache.org/r/34418/#comment135560>

can you remove this?


- Aditya Auradkar


On May 19, 2015, 5:12 p.m., Jay Kreps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34418/
> ---
> 
> (Updated May 19, 2015, 5:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2191
> https://issues.apache.org/jira/browse/KAFKA-2191
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Fix 0 ellapsed time rate bug.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
> 98429da34418f7f1deba1b5e44e2e6025212edb3 
>   clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
> 544e120594de78c43581a980b1e4087b4fb98ccb 
> 
> Diff: https://reviews.apache.org/r/34418/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Updating the discussion with the latest comments.

1. We discussed adding 2 new API's (AlterConfig and DescribeConfig). I'll 
update KIP-21 with details on these.
2. Discussed during the KIP hangout. We are in agreement.

(1) has a dependency on KIP-4 being completed. Rest of the work in the KIP can 
be implemented independently. Any concerns if we tackle it as two separate work 
items implementation wise?
 
We also discussed changing the AlterTopic command in KIP-4 to not include 
config changes. Instead, all config changes will pass through the newly 
proposed AlterConfig. If no-one objects, I can make some changes to KIP-4 to 
reflect this.

Aditya


From: Jay Kreps [jay.kr...@gmail.com]
Sent: Tuesday, May 19, 2015 10:51 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

Hey Aditya,

Two comments:

1. Yeah we need to reconcile this with the APIs in KIP-4. I think it does
make sense to allow setting config during topic creation. I agree with your
summary that having alter topic and alter config may be confusing, but
there are also some non-config changes such as replication factor and
partition count that alter topic can carry out. What is the final state you
are proposing?

2. This is implementation related so probably can be removed from the KIP
entirely, but you seem to be proposing a separate config manager for each
config override type. Should we just generalize TopicConfigManager to be
ConfigOverrideManager and have it handle all the override types we will
have? I think I may just be unclear on what you are proposing...

-Jay

On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Yeah, that was just a typo. I've fixed it. Thanks for calling it out.
>
> In KIP-4, I believe we have 3 types of requests: CreateTopic, AlterTopic
> and DeleteTopic. The topic configs are a sub-type of the Create and Alter
> commands. I think it would be nice to simply have a AlterConfig command
> that can alter any type of config rather than having a specific
> ClientConfig.
>
> AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]]
> ConfigType => string
> AddedConfigEntry => ConfigKey ConfigValue
> ConfigKey => string
> ConfigValue => string
> DeletedConfig => string
>
> The downside of this approach is that we will have 2 separate ways of
> changing topic configs (AlterTopic and AlterConfig). While a general
> AlterConfig only makes sense if we plan to have more than two types of
> entity configs.. it's definitely more future proof. Thoughts?
>
> Aditya
>
> 
> From: Todd Palino [tpal...@gmail.com]
> Sent: Monday, May 18, 2015 12:39 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Agree with Jun here on the JSON format. I think your intention was likely
> to have actual JSON here and it was just a typo in the wiki?
>
> -Todd
>
> On Mon, May 18, 2015 at 12:07 PM, Jun Rao  wrote:
>
> > Aditya,
> >
> > Another thing to consider. In KIP-4, we are adding a new RPC request to
> > change and retrieve topic configs. Do we want to add a similar RPC
> request
> > to change configs per client id? If so, do we want to introduce a
> separate
> > new request or have a combined new request for both topic and client id
> > level config changes?
> >
> > A minor point in the wiki, for the json format in ZK, we should change
> > {X1=Y1,
> > X2=Y2..} to a json map, right?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, May 18, 2015 at 9:48 AM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> > >
> > > Aditya
> > >
> >
>

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Thanks Andrii. I'll make the changes.

I've also updated KIP-21 to include the new config requests. Take a look and 
vote.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration

Aditya

From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
Sent: Tuesday, May 19, 2015 2:26 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

Hi,

Sorry I wasn't able to participate. I don't have objections about removing
config changes from AlterTopic (as I understand both AddedConfig and
DeletedConfig) - you are welcome to update the KIP page.

Thanks,
Andrii Biletskyi

On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Updating the discussion with the latest comments.
>
> 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig). I'll
> update KIP-21 with details on these.
> 2. Discussed during the KIP hangout. We are in agreement.
>
> (1) has a dependency on KIP-4 being completed. Rest of the work in the KIP
> can be implemented independently. Any concerns if we tackle it as two
> separate work items implementation wise?
>
> We also discussed changing the AlterTopic command in KIP-4 to not include
> config changes. Instead, all config changes will pass through the newly
> proposed AlterConfig. If no-one objects, I can make some changes to KIP-4
> to reflect this.
>
> Aditya
>
> 
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Tuesday, May 19, 2015 10:51 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Hey Aditya,
>
> Two comments:
>
> 1. Yeah we need to reconcile this with the APIs in KIP-4. I think it does
> make sense to allow setting config during topic creation. I agree with your
> summary that having alter topic and alter config may be confusing, but
> there are also some non-config changes such as replication factor and
> partition count that alter topic can carry out. What is the final state you
> are proposing?
>
> 2. This is implementation related so probably can be removed from the KIP
> entirely, but you seem to be proposing a separate config manager for each
> config override type. Should we just generalize TopicConfigManager to be
> ConfigOverrideManager and have it handle all the override types we will
> have? I think I may just be unclear on what you are proposing...
>
> -Jay
>
> On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Yeah, that was just a typo. I've fixed it. Thanks for calling it out.
> >
> > In KIP-4, I believe we have 3 types of requests: CreateTopic, AlterTopic
> > and DeleteTopic. The topic configs are a sub-type of the Create and Alter
> > commands. I think it would be nice to simply have a AlterConfig command
> > that can alter any type of config rather than having a specific
> > ClientConfig.
> >
> > AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]]
> > ConfigType => string
> > AddedConfigEntry => ConfigKey ConfigValue
> > ConfigKey => string
> > ConfigValue => string
> > DeletedConfig => string
> >
> > The downside of this approach is that we will have 2 separate ways of
> > changing topic configs (AlterTopic and AlterConfig). While a general
> > AlterConfig only makes sense if we plan to have more than two types of
> > entity configs.. it's definitely more future proof. Thoughts?
> >
> > Aditya
> >
> > 
> > From: Todd Palino [tpal...@gmail.com]
> > Sent: Monday, May 18, 2015 12:39 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Agree with Jun here on the JSON format. I think your intention was likely
> > to have actual JSON here and it was just a typo in the wiki?
> >
> > -Todd
> >
> > On Mon, May 18, 2015 at 12:07 PM, Jun Rao  wrote:
> >
> > > Aditya,
> > >
> > > Another thing to consider. In KIP-4, we are adding a new RPC request to
> > > change and retrieve topic configs. Do we want to add a similar RPC
> > request
> > > to change configs per client id? If so, do we want to introduce a
> > separate
> > > new request or have a combined new request for both topic and client id
> > > level config changes?
> > >
> > > A minor point in the wiki, for the json format in ZK, we should change
> > > {X1=Y1,
> > > X2=Y2..} to a json map, right?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, May 18, 2015 at 9:48 AM, Aditya Auradkar <
> > > aaurad...@linkedin.com.invalid> wrote:
> > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> > > >
> > > > Aditya
> > > >
> > >
> >
>

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Hey Jun,

I've added a section on error codes on the KIP-21 wiki.

Here are the proposed changes to KIP-4. I'll update the wiki shortly.
- Change AlterTopic to not allow setting configs. Config changes will flow 
through AlterConfig. CreateTopic will still allow setting configs as it is nice 
to be able to specify configs while creating the topic.
- Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig" as 
proposed in KIP-21. 


Thanks,
Aditya


From: Jun Rao [j...@confluent.io]
Sent: Thursday, May 21, 2015 8:41 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

Aditya,

For completeness, could you list the set of error codes in the wiki? Also,
could you summarize the changes that are needed for the requests listed in
KIP-4 and update the wiki accordingly?

Thanks,

Jun

On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Thanks Andrii. I'll make the changes.
>
> I've also updated KIP-21 to include the new config requests. Take a look
> and vote.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
>
> Aditya
> 
> From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> Sent: Tuesday, May 19, 2015 2:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Hi,
>
> Sorry I wasn't able to participate. I don't have objections about removing
> config changes from AlterTopic (as I understand both AddedConfig and
> DeletedConfig) - you are welcome to update the KIP page.
>
> Thanks,
> Andrii Biletskyi
>
> On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Updating the discussion with the latest comments.
> >
> > 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig). I'll
> > update KIP-21 with details on these.
> > 2. Discussed during the KIP hangout. We are in agreement.
> >
> > (1) has a dependency on KIP-4 being completed. Rest of the work in the
> KIP
> > can be implemented independently. Any concerns if we tackle it as two
> > separate work items implementation wise?
> >
> > We also discussed changing the AlterTopic command in KIP-4 to not include
> > config changes. Instead, all config changes will pass through the newly
> > proposed AlterConfig. If no-one objects, I can make some changes to KIP-4
> > to reflect this.
> >
> > Aditya
> >
> > 
> > From: Jay Kreps [jay.kr...@gmail.com]
> > Sent: Tuesday, May 19, 2015 10:51 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Hey Aditya,
> >
> > Two comments:
> >
> > 1. Yeah we need to reconcile this with the APIs in KIP-4. I think it does
> > make sense to allow setting config during topic creation. I agree with
> your
> > summary that having alter topic and alter config may be confusing, but
> > there are also some non-config changes such as replication factor and
> > partition count that alter topic can carry out. What is the final state
> you
> > are proposing?
> >
> > 2. This is implementation related so probably can be removed from the KIP
> > entirely, but you seem to be proposing a separate config manager for each
> > config override type. Should we just generalize TopicConfigManager to be
> > ConfigOverrideManager and have it handle all the override types we will
> > have? I think I may just be unclear on what you are proposing...
> >
> > -Jay
> >
> > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Yeah, that was just a typo. I've fixed it. Thanks for calling it out.
> > >
> > > In KIP-4, I believe we have 3 types of requests: CreateTopic,
> AlterTopic
> > > and DeleteTopic. The topic configs are a sub-type of the Create and
> Alter
> > > commands. I think it would be nice to simply have a AlterConfig command
> > > that can alter any type of config rather than having a specific
> > > ClientConfig.
> > >
> > > AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]]
> > > ConfigType => string
> > > AddedConfigEntry => ConfigKey ConfigValue
> > > ConfigKey => string
> > > ConfigValue => string
> > > DeletedConfig => string
> > >
> > > The downside of this approach is that we will have 2 separate ways of
> >

Review Request 34554: Patch for KAFKA-2205

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34554/
---

Review request for kafka.


Bugs: KAFKA-2205
https://issues.apache.org/jira/browse/KAFKA-2205


Repository: kafka


Description
---

Some fixes


KAFKA-2205


KAFKA-2205


Diffs
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
f06edf41c732a7b794e496d0048b0ce6f897e72b 
  core/src/main/scala/kafka/admin/ConfigCommand.scala PRE-CREATION 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
8e6f18633b25bf1beee3f813b28ef7aa7d779d7b 
  core/src/main/scala/kafka/cluster/Partition.scala 
730a232482fdf77be5704cdf5941cfab3828db88 
  core/src/main/scala/kafka/controller/KafkaController.scala 
69bba243a9a511cc5292b43da0cc48e421a428b0 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
3b15ab4eef22c6f50a7483e99a6af40fb55aca9f 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
64ecb499f24bc801d48f86e1612d927cc08e006d 
  core/src/main/scala/kafka/server/ConfigHandler.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
ea6d165d8e5c3146d2c65e8ad1a513308334bf6f 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
b675a7e45ea4f4179f8b15fe221fd988aff13aa0 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
2618dd39b925b979ad6e4c0abd5c6eaafb3db5d5 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 
efb2f8e79b3faef78722774b951fea828cd50374 
  core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala 
c7136f20972614ac47aa57ab13e3c94ef775a4b7 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
7877f6ca1845c2edbf96d4a9783a07a552db8f07 

Diff: https://reviews.apache.org/r/34554/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 34554: Patch for KAFKA-2205

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34554/
---

(Updated May 21, 2015, 5:55 p.m.)


Review request for kafka.


Bugs: KAFKA-2205
https://issues.apache.org/jira/browse/KAFKA-2205


Repository: kafka


Description (updated)
---

KAFKA-2205. Summary of changes:

1. Generalized TopicConfigManager to DynamicConfigManager. It is now able to 
handle multiple types of entities.
2. Changed format of the notification znode as described in KIP-21
3. Replaced TopicConfigManager with DynamicConfigManager.
4. Added new testcases. Existing testcases all pass
5. Added ConfigCommand to handle all config changes. Eventually this will make 
calls to the broker once the new API's are built for now it speaks to ZK 
directly


Diffs
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
f06edf41c732a7b794e496d0048b0ce6f897e72b 
  core/src/main/scala/kafka/admin/ConfigCommand.scala PRE-CREATION 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
8e6f18633b25bf1beee3f813b28ef7aa7d779d7b 
  core/src/main/scala/kafka/cluster/Partition.scala 
730a232482fdf77be5704cdf5941cfab3828db88 
  core/src/main/scala/kafka/controller/KafkaController.scala 
69bba243a9a511cc5292b43da0cc48e421a428b0 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
3b15ab4eef22c6f50a7483e99a6af40fb55aca9f 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
64ecb499f24bc801d48f86e1612d927cc08e006d 
  core/src/main/scala/kafka/server/ConfigHandler.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
ea6d165d8e5c3146d2c65e8ad1a513308334bf6f 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
b675a7e45ea4f4179f8b15fe221fd988aff13aa0 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
2618dd39b925b979ad6e4c0abd5c6eaafb3db5d5 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 
efb2f8e79b3faef78722774b951fea828cd50374 
  core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala 
c7136f20972614ac47aa57ab13e3c94ef775a4b7 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
7877f6ca1845c2edbf96d4a9783a07a552db8f07 

Diff: https://reviews.apache.org/r/34554/diff/


Testing (updated)
---

1. Added new testcases for new code.
2. Verified that both topic and client configs can be changed dynamically by 
starting a local cluster


Thanks,

Aditya Auradkar

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

I think we should remove the config part in TopicMetadataResponse. It's 
probably cleaner if Alter and Describe are the only way to view and modify 
configs but I don't feel very strongly about it. 

Re-summarizing the proposed changes to KIP-4:
- Change AlterTopic to not allow setting configs. Config changes will flow 
through AlterConfig. CreateTopic will still allow setting configs as it is nice 
to be able to specify configs while creating the topic.
- TopicMetadataResponse shoudn't return config for the topic. DescribeConfig is 
the way to go.
- Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig" as 
proposed in KIP-21.

Aditya


From: Jun Rao [j...@confluent.io]
Sent: Thursday, May 21, 2015 10:50 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

What about TopicMetadataResponse in KIP-4? Do we remove the config part in
it?

Thanks,

Jun

On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey Jun,
>
> I've added a section on error codes on the KIP-21 wiki.
>
> Here are the proposed changes to KIP-4. I'll update the wiki shortly.
> - Change AlterTopic to not allow setting configs. Config changes will flow
> through AlterConfig. CreateTopic will still allow setting configs as it is
> nice to be able to specify configs while creating the topic.
> - Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig"
> as proposed in KIP-21.
>
>
> Thanks,
> Aditya
>
> 
> From: Jun Rao [j...@confluent.io]
> Sent: Thursday, May 21, 2015 8:41 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Aditya,
>
> For completeness, could you list the set of error codes in the wiki? Also,
> could you summarize the changes that are needed for the requests listed in
> KIP-4 and update the wiki accordingly?
>
> Thanks,
>
> Jun
>
> On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Thanks Andrii. I'll make the changes.
> >
> > I've also updated KIP-21 to include the new config requests. Take a look
> > and vote.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> >
> > Aditya
> > 
> > From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> > Sent: Tuesday, May 19, 2015 2:26 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Hi,
> >
> > Sorry I wasn't able to participate. I don't have objections about
> removing
> > config changes from AlterTopic (as I understand both AddedConfig and
> > DeletedConfig) - you are welcome to update the KIP page.
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Updating the discussion with the latest comments.
> > >
> > > 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig).
> I'll
> > > update KIP-21 with details on these.
> > > 2. Discussed during the KIP hangout. We are in agreement.
> > >
> > > (1) has a dependency on KIP-4 being completed. Rest of the work in the
> > KIP
> > > can be implemented independently. Any concerns if we tackle it as two
> > > separate work items implementation wise?
> > >
> > > We also discussed changing the AlterTopic command in KIP-4 to not
> include
> > > config changes. Instead, all config changes will pass through the newly
> > > proposed AlterConfig. If no-one objects, I can make some changes to
> KIP-4
> > > to reflect this.
> > >
> > > Aditya
> > >
> > > 
> > > From: Jay Kreps [jay.kr...@gmail.com]
> > > Sent: Tuesday, May 19, 2015 10:51 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > Hey Aditya,
> > >
> > > Two comments:
> > >
> > > 1. Yeah we need to reconcile this with the APIs in KIP-4. I think it
> does
> > > make sense to allow setting config during topic creation. I agree with
> > your
> > > summary that having alter topic and alter config may be confusing, but
> > > there are also some non-config changes such as replication factor and
> > > partition count that alter topic can carry out. What is 

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 26, 2015, 6:50 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Adding javadoc


KAFKA-2084 - Moved the callbacks to ClientQuotaMetrics


Adding more configs


Don't quota replica traffic


KAFKA-2084


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
387e387998fc3a6c9cb585dab02b5f77b0381fbf 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
e66710d2368334ece66f70d55f57b3f888262620 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated May 26, 2015, 6:53 p.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Updated patch for quotas. This patch does the following: 
1. Add per-client metrics for both producer and consumers 
2. Add configuration for quotas 
3. Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 
4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
request. Implemented request throttling for produce and fetch requests. 
5. Added unit and integration test cases.
6. This doesn't include a system test. There is a separate ticket for that


Diffs
-

  clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
387e387998fc3a6c9cb585dab02b5f77b0381fbf 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
e66710d2368334ece66f70d55f57b3f888262620 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

> On May 12, 2015, 7:38 p.m., Dong Lin wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java, line 117
> > <https://reviews.apache.org/r/33049/diff/9/?file=955824#file955824line117>
> >
> > metric.value(timeMs), which translates to Rate.measure(config, timeMs), 
> > may return Infinity. This may introduce infinite delay.
> > 
> > I think this bug is rooted either in class Rate or  class SampledStat. 
> > In short, SampledStat.purgeObsoleteSamples will remove all Samples, such 
> > that SampledStat.oldest(now) == now.
> > 
> > Should I open a ticket and submit a patch for it?
> 
> Aditya Auradkar wrote:
> Hey dong, yeah you should submit a patch for it.
> 
> Dong Lin wrote:
> Sure! I will do it.

Dropping since this is being tracked in a separate ticket.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review83441
-------


On May 26, 2015, 6:53 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated May 26, 2015, 6:53 p.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Updated patch for quotas. This patch does the following: 
> 1. Add per-client metrics for both producer and consumers 
> 2. Add configuration for quotas 
> 3. Compute delay times in the metrics package and return the delay times in 
> QuotaViolationException 
> 4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
> request. Implemented request throttling for produce and fetch requests. 
> 5. Added unit and integration test cases.
> 6. This doesn't include a system test. There is a separate ticket for that
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
> dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
>  a451e5385c9eca76b38b425e8ac856b2715fcffe 
>   clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
> ca823fd4639523018311b814fde69b6177e73b97 
>   clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
>   core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 387e387998fc3a6c9cb585dab02b5f77b0381fbf 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> e66710d2368334ece66f70d55f57b3f888262620 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
>   core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> fc226c863095b7761290292cd8755cd7ad0f155c 
>   core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
> 8014a5a6c362785539f24eb03d77278434614fe6 
>   core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33049/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

[Aditya Auradkar]

Andryii,

I made a few edits to this document as discussed in the KIP-21 thread. 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations

With these changes. the only difference between TopicMetadataResponse_V1 and V0 
is the removal of the ISR field. I've altered the KIP with the assumption that 
this is a good enough reason by itself to evolve the request/response protocol. 
Any concerns there?

Thanks,
Aditya


From: Mayuresh Gharat [gharatmayures...@gmail.com]
Sent: Thursday, May 21, 2015 8:29 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative 
operations (Thread 2)

Hi Jun,

Thanks a lot. I get it now.
 Point 4) will actually enable clients to who don't want to create a topic
with default partitions, if it does not exist and then can manually create
the topic with their own configs(#partitions).

Thanks,

Mayuresh

On Thu, May 21, 2015 at 6:16 PM, Jun Rao  wrote:

> Mayuresh,
>
> The current plan is the following.
>
> 1. Add TMR v1, which still triggers auto topic creation.
> 2. Change the consumer client to TMR v1. Change the producer client to use
> TMR v1 and on UnknownTopicException, issue TopicCreateRequest to explicitly
> create the topic with the default server side partitions and replicas.
> 3. At some later time after the new clients are released and deployed,
> disable auto topic creation in TMR v1. This will make sure consumers never
> create new topics.
> 4. If needed, we can add a new config in the producer to control whether
> TopicCreateRequest should be issued or not on UnknownTopicException. If
> this is disabled and the topic doesn't exist, send will fail and the user
> is expected to create the topic manually.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi,
> > I had a question about TopicMetadata Request.
> > Currently the way it works is :
> >
> > 1) Suppose a topic T1 does not exist.
> > 2) Client wants to produce data to T1 using producer P1.
> > 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka.
> > This in turn creates the default number of partition. The number of
> > partitions is a cluster wide config.
> > 4) Same goes for a consumer. If the topic does not exist and new topic
> will
> > be created when the consumer issues TopicMetadata request.
> >
> > Here are 2 use cases where it might not be suited :
> >
> > The auto creation flag for topics  is turned  ON.
> >
> > a) Some clients might not want to create topic with default number of
> > partitions but with lower number of partitions. Currently in a
> multi-tenant
> > environment this is not possible without changing the cluster wide
> default
> > config.
> >
> > b) Some clients might want to just check if the topic exist or not but
> > currently the topic gets created automatically using default number of
> > partitions.
> >
> > Here are some ideas to address this :
> >
> > 1) The way this can be  addressed is that TopicMetadata request should
> have
> > a way to specify whether it should only check if the topic exist or check
> > and create a topic with given number of partitions. If the number of
> > partitions is not specified use the default cluster wide config.
> >
> > OR
> >
> > 2) We should only allow TopicMetadata Request to get the metadata
> > explicitly and not allow it to create a new topic. We should have another
> > Request that takes in config parameters from the user regarding how
> he/she
> > wants the topic to be created. This request can be used if we get an
> empty
> > TopicMetadata Response.
> >
> >
> > Thanks,
> >
> > Mayuresh
> >
> >
> > On Thu, May 14, 2015 at 10:22 AM, Jun Rao  wrote:
> >
> > > For ListTopics, we decided not to add a ListTopics request for now and
> > just
> > > rely on passing in an empty list to TMR. We can revisit this in the
> > future
> > > if it becomes an issue.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy 
> wrote:
> > >
> > > > Just had a few minor questions before I join the vote thread.
> > > > Apologies if these have been discussed:
> > > >
> > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just return
> > > >   InvalidPartitions instead?
> > > > - AdminClient.listTopics: should we allow listing all partitions? Or
> > > >   do you intend for the client to issue listTopics followed by
> > > >   describeTopics?
> > > > - On returning future for partition reassignments: do we need
> to
> > > >   return any future especially since you have the
> > > >   verifyReassignPartitions method? For e.g., what happens if the
> > > >   controller moves? The get should fail right? The client will then
> > > >   need to connect to the new controller and reissue the request but
> > > >   will then get ReassignPartitionsInProgress. So in that case the
> > > >   client any way needs

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Hey everyone,

Completed the changes to KIP-4. After today's hangout, there doesn't appear to 
be anything remaining to discuss on this KIP. 
Please vote so we can formally close this.

Thanks,
Aditya

____
From: Aditya Auradkar
Sent: Thursday, May 21, 2015 11:26 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-21 Dynamic Configuration

I think we should remove the config part in TopicMetadataResponse. It's 
probably cleaner if Alter and Describe are the only way to view and modify 
configs but I don't feel very strongly about it.

Re-summarizing the proposed changes to KIP-4:
- Change AlterTopic to not allow setting configs. Config changes will flow 
through AlterConfig. CreateTopic will still allow setting configs as it is nice 
to be able to specify configs while creating the topic.
- TopicMetadataResponse shoudn't return config for the topic. DescribeConfig is 
the way to go.
- Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig" as 
proposed in KIP-21.

Aditya


From: Jun Rao [j...@confluent.io]
Sent: Thursday, May 21, 2015 10:50 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

What about TopicMetadataResponse in KIP-4? Do we remove the config part in
it?

Thanks,

Jun

On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey Jun,
>
> I've added a section on error codes on the KIP-21 wiki.
>
> Here are the proposed changes to KIP-4. I'll update the wiki shortly.
> - Change AlterTopic to not allow setting configs. Config changes will flow
> through AlterConfig. CreateTopic will still allow setting configs as it is
> nice to be able to specify configs while creating the topic.
> - Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig"
> as proposed in KIP-21.
>
>
> Thanks,
> Aditya
>
> 
> From: Jun Rao [j...@confluent.io]
> Sent: Thursday, May 21, 2015 8:41 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Aditya,
>
> For completeness, could you list the set of error codes in the wiki? Also,
> could you summarize the changes that are needed for the requests listed in
> KIP-4 and update the wiki accordingly?
>
> Thanks,
>
> Jun
>
> On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Thanks Andrii. I'll make the changes.
> >
> > I've also updated KIP-21 to include the new config requests. Take a look
> > and vote.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> >
> > Aditya
> > 
> > From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> > Sent: Tuesday, May 19, 2015 2:26 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Hi,
> >
> > Sorry I wasn't able to participate. I don't have objections about
> removing
> > config changes from AlterTopic (as I understand both AddedConfig and
> > DeletedConfig) - you are welcome to update the KIP page.
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Updating the discussion with the latest comments.
> > >
> > > 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig).
> I'll
> > > update KIP-21 with details on these.
> > > 2. Discussed during the KIP hangout. We are in agreement.
> > >
> > > (1) has a dependency on KIP-4 being completed. Rest of the work in the
> > KIP
> > > can be implemented independently. Any concerns if we tackle it as two
> > > separate work items implementation wise?
> > >
> > > We also discussed changing the AlterTopic command in KIP-4 to not
> include
> > > config changes. Instead, all config changes will pass through the newly
> > > proposed AlterConfig. If no-one objects, I can make some changes to
> KIP-4
> > > to reflect this.
> > >
> > > Aditya
> > >
> > > 
> > > From: Jay Kreps [jay.kr...@gmail.com]
> > > Sent: Tuesday, May 19, 2015 10:51 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > Hey Aditya,
> > >
> > > Two comments:
> > >
> > > 1. Yeah we need to

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

bump


From: Aditya Auradkar
Sent: Tuesday, May 26, 2015 1:16 PM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-21 Dynamic Configuration

Hey everyone,

Completed the changes to KIP-4. After today's hangout, there doesn't appear to 
be anything remaining to discuss on this KIP.
Please vote so we can formally close this.

Thanks,
Aditya

____
From: Aditya Auradkar
Sent: Thursday, May 21, 2015 11:26 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-21 Dynamic Configuration

I think we should remove the config part in TopicMetadataResponse. It's 
probably cleaner if Alter and Describe are the only way to view and modify 
configs but I don't feel very strongly about it.

Re-summarizing the proposed changes to KIP-4:
- Change AlterTopic to not allow setting configs. Config changes will flow 
through AlterConfig. CreateTopic will still allow setting configs as it is nice 
to be able to specify configs while creating the topic.
- TopicMetadataResponse shoudn't return config for the topic. DescribeConfig is 
the way to go.
- Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig" as 
proposed in KIP-21.

Aditya


From: Jun Rao [j...@confluent.io]
Sent: Thursday, May 21, 2015 10:50 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

What about TopicMetadataResponse in KIP-4? Do we remove the config part in
it?

Thanks,

Jun

On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey Jun,
>
> I've added a section on error codes on the KIP-21 wiki.
>
> Here are the proposed changes to KIP-4. I'll update the wiki shortly.
> - Change AlterTopic to not allow setting configs. Config changes will flow
> through AlterConfig. CreateTopic will still allow setting configs as it is
> nice to be able to specify configs while creating the topic.
> - Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig"
> as proposed in KIP-21.
>
>
> Thanks,
> Aditya
>
> 
> From: Jun Rao [j...@confluent.io]
> Sent: Thursday, May 21, 2015 8:41 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Aditya,
>
> For completeness, could you list the set of error codes in the wiki? Also,
> could you summarize the changes that are needed for the requests listed in
> KIP-4 and update the wiki accordingly?
>
> Thanks,
>
> Jun
>
> On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Thanks Andrii. I'll make the changes.
> >
> > I've also updated KIP-21 to include the new config requests. Take a look
> > and vote.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> >
> > Aditya
> > 
> > From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> > Sent: Tuesday, May 19, 2015 2:26 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Hi,
> >
> > Sorry I wasn't able to participate. I don't have objections about
> removing
> > config changes from AlterTopic (as I understand both AddedConfig and
> > DeletedConfig) - you are welcome to update the KIP page.
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Updating the discussion with the latest comments.
> > >
> > > 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig).
> I'll
> > > update KIP-21 with details on these.
> > > 2. Discussed during the KIP hangout. We are in agreement.
> > >
> > > (1) has a dependency on KIP-4 being completed. Rest of the work in the
> > KIP
> > > can be implemented independently. Any concerns if we tackle it as two
> > > separate work items implementation wise?
> > >
> > > We also discussed changing the AlterTopic command in KIP-4 to not
> include
> > > config changes. Instead, all config changes will pass through the newly
> > > proposed AlterConfig. If no-one objects, I can make some changes to
> KIP-4
> > > to reflect this.
> > >
> > > Aditya
> > >
> > > 
> > > From: Jay Kreps [jay.kr...@gmail.com]
> > > Sent: Tuesday, May 19, 2015 10:51 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] 

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

[Aditya Auradkar]

Thanks. Perhaps we should leave TMR unchanged for now. Should we discuss this 
during the next hangout?

Aditya


From: Jun Rao [j...@confluent.io]
Sent: Thursday, May 28, 2015 5:32 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative 
operations (Thread 2)

There is a reasonable use case of ISR in KAFKA-2225. Basically, for
economical reasons, we may want to let a consumer fetch from a replica in
ISR that's in the same zone. In order to support that, it will be
convenient to have TMR return the correct ISR for the consumer to choose.

So, perhaps it's worth fixing the ISR inconsistency issue in KAFKA-1367
(there is some new discussion there on what it takes to fix this). If we do
that, we can leave TMR unchanged.

Thanks,

Jun

On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Andryii,
>
> I made a few edits to this document as discussed in the KIP-21 thread.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
>
> With these changes. the only difference between TopicMetadataResponse_V1
> and V0 is the removal of the ISR field. I've altered the KIP with the
> assumption that this is a good enough reason by itself to evolve the
> request/response protocol. Any concerns there?
>
> Thanks,
> Aditya
>
> 
> From: Mayuresh Gharat [gharatmayures...@gmail.com]
> Sent: Thursday, May 21, 2015 8:29 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
> operations (Thread 2)
>
> Hi Jun,
>
> Thanks a lot. I get it now.
>  Point 4) will actually enable clients to who don't want to create a topic
> with default partitions, if it does not exist and then can manually create
> the topic with their own configs(#partitions).
>
> Thanks,
>
> Mayuresh
>
> On Thu, May 21, 2015 at 6:16 PM, Jun Rao  wrote:
>
> > Mayuresh,
> >
> > The current plan is the following.
> >
> > 1. Add TMR v1, which still triggers auto topic creation.
> > 2. Change the consumer client to TMR v1. Change the producer client to
> use
> > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to
> explicitly
> > create the topic with the default server side partitions and replicas.
> > 3. At some later time after the new clients are released and deployed,
> > disable auto topic creation in TMR v1. This will make sure consumers
> never
> > create new topics.
> > 4. If needed, we can add a new config in the producer to control whether
> > TopicCreateRequest should be issued or not on UnknownTopicException. If
> > this is disabled and the topic doesn't exist, send will fail and the user
> > is expected to create the topic manually.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Hi,
> > > I had a question about TopicMetadata Request.
> > > Currently the way it works is :
> > >
> > > 1) Suppose a topic T1 does not exist.
> > > 2) Client wants to produce data to T1 using producer P1.
> > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka.
> > > This in turn creates the default number of partition. The number of
> > > partitions is a cluster wide config.
> > > 4) Same goes for a consumer. If the topic does not exist and new topic
> > will
> > > be created when the consumer issues TopicMetadata request.
> > >
> > > Here are 2 use cases where it might not be suited :
> > >
> > > The auto creation flag for topics  is turned  ON.
> > >
> > > a) Some clients might not want to create topic with default number of
> > > partitions but with lower number of partitions. Currently in a
> > multi-tenant
> > > environment this is not possible without changing the cluster wide
> > default
> > > config.
> > >
> > > b) Some clients might want to just check if the topic exist or not but
> > > currently the topic gets created automatically using default number of
> > > partitions.
> > >
> > > Here are some ideas to address this :
> > >
> > > 1) The way this can be  addressed is that TopicMetadata request should
> > have
> > > a way to specify whether it should only check if the topic exist or
> check
> > > and create a topic with given number of partitions. If the number of
> > >

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Yeah, the same cleaning mechanism will be carried over.

> 1. Are we introducing a new Java API for the config change protocol and if
> so where will that appear? Is that going to be part of the java api in the
> admin api kip? Let's document that.
Yeah, we need to introduce a new Java API for the config change protocol. It 
should be a part of the AdminClient API. I'll alter KIP-4 to reflect that since 
the API is being introduced there.

> 2. The proposed JSON format uses camel case for field names, is that what
> we've used for other JSON in zookeeper?
I think camel case is more appropriate for the JSON format. For example, under 
the "brokers" znode, I found "jmx_port".

> 3. This changes the format of the notifications, right? How will we
> grandfather in the old format? Clusters will have existing change
> notifications in the old format so I think the new code will need to be
> able to read those?
Interesting, I figured the existing notifications were purged by a cleaner 
thread frequently. In that case, we wouldn't need to grandfather any 
notifications since we would only need to not make any config changes for X 
minutes for there to be no changes in ZK. But the old notifications are 
actually removed when a new notification is received or the broker is bounced. 
So we do need to handle notifications in the old format. Should we simply 
ignore older changes since they are only valid for a short period of time?

Thanks,
Aditya

From: Jay Kreps [jay.kr...@gmail.com]
Sent: Thursday, May 28, 2015 5:25 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

That is handled now so I am assuming the same mechanism carries over?

-Jay

On Thu, May 28, 2015 at 5:12 PM, Guozhang Wang  wrote:

> For the sequential config/changes/config_change_XX znode, do we have any
> manners to do cleaning in order to avoid the change-log from growing
> indefinitely?
>
> Guozhang
>
> On Thu, May 28, 2015 at 5:02 PM, Jay Kreps  wrote:
>
> > I still have a couple of questions:
> > 1. Are we introducing a new Java API for the config change protocol and
> if
> > so where will that appear? Is that going to be part of the java api in
> the
> > admin api kip? Let's document that.
> > 2. The proposed JSON format uses camel case for field names, is that what
> > we've used for other JSON in zookeeper?
> > 3. This changes the format of the notifications, right? How will we
> > grandfather in the old format? Clusters will have existing change
> > notifications in the old format so I think the new code will need to be
> > able to read those?
> >
> > -Jay
> >
> > On Thu, May 28, 2015 at 11:41 AM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > bump
> > >
> > > 
> > > From: Aditya Auradkar
> > > Sent: Tuesday, May 26, 2015 1:16 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > Hey everyone,
> > >
> > > Completed the changes to KIP-4. After today's hangout, there doesn't
> > > appear to be anything remaining to discuss on this KIP.
> > > Please vote so we can formally close this.
> > >
> > > Thanks,
> > > Aditya
> > >
> > > 
> > > From: Aditya Auradkar
> > > Sent: Thursday, May 21, 2015 11:26 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > I think we should remove the config part in TopicMetadataResponse. It's
> > > probably cleaner if Alter and Describe are the only way to view and
> > modify
> > > configs but I don't feel very strongly about it.
> > >
> > > Re-summarizing the proposed changes to KIP-4:
> > > - Change AlterTopic to not allow setting configs. Config changes will
> > flow
> > > through AlterConfig. CreateTopic will still allow setting configs as it
> > is
> > > nice to be able to specify configs while creating the topic.
> > > - TopicMetadataResponse shoudn't return config for the topic.
> > > DescribeConfig is the way to go.
> > > - Change "InvalidTopicConfiguration" error code to
> "InvalidEntityConfig"
> > > as proposed in KIP-21.
> > >
> > > Aditya
> > >
> > > 
> > > From: Jun Rao [j...@confluent.io]
> > > Sent: Thursday, May 21, 2015 10:50 AM
> > > To: dev@kafka.apach

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Minor edit: I meant that we should expect change notifications in the old 
format made earlier, but should perhaps ignore them. After the upgrade is done, 
older versions of AdminTools can no longer be used to make config changes.

Aditya


From: Aditya Auradkar
Sent: Thursday, May 28, 2015 11:22 PM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-21 Dynamic Configuration

Yeah, the same cleaning mechanism will be carried over.

> 1. Are we introducing a new Java API for the config change protocol and if
> so where will that appear? Is that going to be part of the java api in the
> admin api kip? Let's document that.
Yeah, we need to introduce a new Java API for the config change protocol. It 
should be a part of the AdminClient API. I'll alter KIP-4 to reflect that since 
the API is being introduced there.

> 2. The proposed JSON format uses camel case for field names, is that what
> we've used for other JSON in zookeeper?
I think camel case is more appropriate for the JSON format. For example, under 
the "brokers" znode, I found "jmx_port".

> 3. This changes the format of the notifications, right? How will we
> grandfather in the old format? Clusters will have existing change
> notifications in the old format so I think the new code will need to be
> able to read those?
Interesting, I figured the existing notifications were purged by a cleaner 
thread frequently. In that case, we wouldn't need to grandfather any 
notifications since we would only need to not make any config changes for X 
minutes for there to be no changes in ZK. But the old notifications are 
actually removed when a new notification is received or the broker is bounced. 
So we do need to handle notifications in the old format. Should we simply 
ignore older changes since they are only valid for a short period of time?

Thanks,
Aditya

From: Jay Kreps [jay.kr...@gmail.com]
Sent: Thursday, May 28, 2015 5:25 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

That is handled now so I am assuming the same mechanism carries over?

-Jay

On Thu, May 28, 2015 at 5:12 PM, Guozhang Wang  wrote:

> For the sequential config/changes/config_change_XX znode, do we have any
> manners to do cleaning in order to avoid the change-log from growing
> indefinitely?
>
> Guozhang
>
> On Thu, May 28, 2015 at 5:02 PM, Jay Kreps  wrote:
>
> > I still have a couple of questions:
> > 1. Are we introducing a new Java API for the config change protocol and
> if
> > so where will that appear? Is that going to be part of the java api in
> the
> > admin api kip? Let's document that.
> > 2. The proposed JSON format uses camel case for field names, is that what
> > we've used for other JSON in zookeeper?
> > 3. This changes the format of the notifications, right? How will we
> > grandfather in the old format? Clusters will have existing change
> > notifications in the old format so I think the new code will need to be
> > able to read those?
> >
> > -Jay
> >
> > On Thu, May 28, 2015 at 11:41 AM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > bump
> > >
> > > 
> > > From: Aditya Auradkar
> > > Sent: Tuesday, May 26, 2015 1:16 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > Hey everyone,
> > >
> > > Completed the changes to KIP-4. After today's hangout, there doesn't
> > > appear to be anything remaining to discuss on this KIP.
> > > Please vote so we can formally close this.
> > >
> > > Thanks,
> > > Aditya
> > >
> > > 
> > > From: Aditya Auradkar
> > > Sent: Thursday, May 21, 2015 11:26 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > I think we should remove the config part in TopicMetadataResponse. It's
> > > probably cleaner if Alter and Describe are the only way to view and
> > modify
> > > configs but I don't feel very strongly about it.
> > >
> > > Re-summarizing the proposed changes to KIP-4:
> > > - Change AlterTopic to not allow setting configs. Config changes will
> > flow
> > > through AlterConfig. CreateTopic will still allow setting configs as it
> > is
> > > nice to be able to specify configs while creating the topic.
> > > - TopicMetadataResponse shoudn't return config for the topic.
> > > DescribeConfig

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

2. There was a typo in my previous email. I meant to say that we should use 
snake case because it's more consistent. I couldn't find any examples of camel 
case but did find some snake case (jmx_port). Other than that, most other 
entries are single word keys.

3. The purge frequency is short (15 minutes). So it should be safe to ignore 
older notifications. Add a story here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration#KIP-21-DynamicConfiguration-Migrationplanfornotifications

Thanks,
Aditya

From: Jay Kreps [jay.kr...@gmail.com]
Sent: Saturday, May 30, 2015 9:51 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

1. Great.
2. I don't have a preference as to the casing, but I really appreciate
consistency. Is everything using underscores today? If so let's stick with
that. If we are already inconsistent then I guess it's too late and we can
do whatever. Let me know and I'll update the coding standard.
3. Not sure what the default purge frequency is. I don't think we need to
work the details of this out in the KIP, but we need a story for the
upgrade path so people don't get bitten.

-Jay

On Thu, May 28, 2015 at 11:22 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Yeah, the same cleaning mechanism will be carried over.
>
> > 1. Are we introducing a new Java API for the config change protocol and
> if
> > so where will that appear? Is that going to be part of the java api in
> the
> > admin api kip? Let's document that.
> Yeah, we need to introduce a new Java API for the config change protocol.
> It should be a part of the AdminClient API. I'll alter KIP-4 to reflect
> that since the API is being introduced there.
>
> > 2. The proposed JSON format uses camel case for field names, is that what
> > we've used for other JSON in zookeeper?
> I think camel case is more appropriate for the JSON format. For example,
> under the "brokers" znode, I found "jmx_port".
>
> > 3. This changes the format of the notifications, right? How will we
> > grandfather in the old format? Clusters will have existing change
> > notifications in the old format so I think the new code will need to be
> > able to read those?
> Interesting, I figured the existing notifications were purged by a cleaner
> thread frequently. In that case, we wouldn't need to grandfather any
> notifications since we would only need to not make any config changes for X
> minutes for there to be no changes in ZK. But the old notifications are
> actually removed when a new notification is received or the broker is
> bounced. So we do need to handle notifications in the old format. Should we
> simply ignore older changes since they are only valid for a short period of
> time?
>
> Thanks,
> Aditya
> 
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Thursday, May 28, 2015 5:25 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> That is handled now so I am assuming the same mechanism carries over?
>
> -Jay
>
> On Thu, May 28, 2015 at 5:12 PM, Guozhang Wang  wrote:
>
> > For the sequential config/changes/config_change_XX znode, do we have any
> > manners to do cleaning in order to avoid the change-log from growing
> > indefinitely?
> >
> > Guozhang
> >
> > On Thu, May 28, 2015 at 5:02 PM, Jay Kreps  wrote:
> >
> > > I still have a couple of questions:
> > > 1. Are we introducing a new Java API for the config change protocol and
> > if
> > > so where will that appear? Is that going to be part of the java api in
> > the
> > > admin api kip? Let's document that.
> > > 2. The proposed JSON format uses camel case for field names, is that
> what
> > > we've used for other JSON in zookeeper?
> > > 3. This changes the format of the notifications, right? How will we
> > > grandfather in the old format? Clusters will have existing change
> > > notifications in the old format so I think the new code will need to be
> > > able to read those?
> > >
> > > -Jay
> > >
> > > On Thu, May 28, 2015 at 11:41 AM, Aditya Auradkar <
> > > aaurad...@linkedin.com.invalid> wrote:
> > >
> > > > bump
> > > >
> > > > 
> > > > From: Aditya Auradkar
> > > > Sent: Tuesday, May 26, 2015 1:16 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration
> > > >
> > &g

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Thanks Jay. Since we have enough votes now, I'll mark this as adopted.

Aditya


From: Jay Kreps [jay.kr...@gmail.com]
Sent: Monday, June 01, 2015 8:42 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

Awesome. +1

On Sunday, May 31, 2015, Aditya Auradkar 
wrote:

> 2. There was a typo in my previous email. I meant to say that we should
> use snake case because it's more consistent. I couldn't find any examples
> of camel case but did find some snake case (jmx_port). Other than that,
> most other entries are single word keys.
>
> 3. The purge frequency is short (15 minutes). So it should be safe to
> ignore older notifications. Add a story here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration#KIP-21-DynamicConfiguration-Migrationplanfornotifications
>
> Thanks,
> Aditya
> 
> From: Jay Kreps [jay.kr...@gmail.com ]
> Sent: Saturday, May 30, 2015 9:51 AM
> To: dev@kafka.apache.org 
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> 1. Great.
> 2. I don't have a preference as to the casing, but I really appreciate
> consistency. Is everything using underscores today? If so let's stick with
> that. If we are already inconsistent then I guess it's too late and we can
> do whatever. Let me know and I'll update the coding standard.
> 3. Not sure what the default purge frequency is. I don't think we need to
> work the details of this out in the KIP, but we need a story for the
> upgrade path so people don't get bitten.
>
> -Jay
>
> On Thu, May 28, 2015 at 11:22 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Yeah, the same cleaning mechanism will be carried over.
> >
> > > 1. Are we introducing a new Java API for the config change protocol and
> > if
> > > so where will that appear? Is that going to be part of the java api in
> > the
> > > admin api kip? Let's document that.
> > Yeah, we need to introduce a new Java API for the config change protocol.
> > It should be a part of the AdminClient API. I'll alter KIP-4 to reflect
> > that since the API is being introduced there.
> >
> > > 2. The proposed JSON format uses camel case for field names, is that
> what
> > > we've used for other JSON in zookeeper?
> > I think camel case is more appropriate for the JSON format. For example,
> > under the "brokers" znode, I found "jmx_port".
> >
> > > 3. This changes the format of the notifications, right? How will we
> > > grandfather in the old format? Clusters will have existing change
> > > notifications in the old format so I think the new code will need to be
> > > able to read those?
> > Interesting, I figured the existing notifications were purged by a
> cleaner
> > thread frequently. In that case, we wouldn't need to grandfather any
> > notifications since we would only need to not make any config changes
> for X
> > minutes for there to be no changes in ZK. But the old notifications are
> > actually removed when a new notification is received or the broker is
> > bounced. So we do need to handle notifications in the old format. Should
> we
> > simply ignore older changes since they are only valid for a short period
> of
> > time?
> >
> > Thanks,
> > Aditya
> > 
> > From: Jay Kreps [jay.kr...@gmail.com ]
> > Sent: Thursday, May 28, 2015 5:25 PM
> > To: dev@kafka.apache.org 
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > That is handled now so I am assuming the same mechanism carries over?
> >
> > -Jay
> >
> > On Thu, May 28, 2015 at 5:12 PM, Guozhang Wang  > wrote:
> >
> > > For the sequential config/changes/config_change_XX znode, do we have
> any
> > > manners to do cleaning in order to avoid the change-log from growing
> > > indefinitely?
> > >
> > > Guozhang
> > >
> > > On Thu, May 28, 2015 at 5:02 PM, Jay Kreps  > wrote:
> > >
> > > > I still have a couple of questions:
> > > > 1. Are we introducing a new Java API for the config change protocol
> and
> > > if
> > > > so where will that appear? Is that going to be part of the java api
> in
> > > the
> > > > admin api kip? Let's document that.
> > > > 2. The proposed JSON format uses camel case for field names, is that
> > what
> > > > we've used for other JSON in zookeep

KIP Wiki

[Aditya Auradkar]

Hey everyone,

We have enough KIP's now (25) that it's a bit hard to tell which ones are 
adopted or under discussion by glancing at the wiki. Any concerns if I split it 
into 3 tables (adopted, discarded and KIP's under discussion)?
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

Aditya

RE: [VOTE] KIP-19 Add a request timeout to NetworkClient

[Aditya Auradkar]

+1 (non-binding).

One minor comment. Should we rename "max.block.ms" to "max.send.block.ms"? 
Something that indicates that it is being applied to the "send" API methods 
exposed by KafkaProducer.

Aditya

From: Joel Koshy [jjkosh...@gmail.com]
Sent: Tuesday, June 02, 2015 4:36 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-19 Add a request timeout to NetworkClient

+1

On Tue, Jun 02, 2015 at 10:13:11PM +, Jiangjie Qin wrote:
> Ah, yes. Just changed it to ³request.timeout.ms².
>
> Thanks.
>
> Jiangjie (Becket) Qin
>
> On 6/2/15, 2:31 PM, "Jun Rao"  wrote:
>
> >The wiki references "network.request.timeout.ms" in ProducerConfig. It
> >should be "request.timeout.ms", right?
> >
> >Thanks,
> >
> >Jun
> >
> >On Tue, Jun 2, 2015 at 2:06 PM, Jiangjie Qin 
> >wrote:
> >
> >> Hi folks,
> >>
> >> Thanks a lot for all the input and help. It looks we do not have further
> >> concerns on KIP-19. I¹ve updated the wiki page, let¹s vote!
> >>
> >>
> >>
> >>https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+
> >>timeout+to+NetworkClient
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
>

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

ka/utils/timer/TimerTask.scala 
3407138115d579339ffb6b00e32e38c984ac5d6e 
  core/src/main/scala/kafka/utils/timer/TimerTaskList.scala 
e7a96570ddc2367583d6d5590628db7e7f6ba39b 
  core/src/main/scala/kafka/utils/timer/TimingWheel.scala 
e92aba3844dbf3372182e14536a5d98cf3366d73 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 
  gradle.properties 90b1945372e767b9c2d0a50df9eb7063e0629952 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated June 3, 2015, 12:09 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Adding javadoc


KAFKA-2084 - Moved the callbacks to ClientQuotaMetrics


Adding more configs


Don't quota replica traffic


KAFKA-2084


Fixing test failures


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
98429da34418f7f1deba1b5e44e2e6025212edb3 
  clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
544e120594de78c43581a980b1e4087b4fb98ccb 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
387e387998fc3a6c9cb585dab02b5f77b0381fbf 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
e66710d2368334ece66f70d55f57b3f888262620 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated June 3, 2015, 12:10 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

This is currently not being used anywhere in the code because I haven't yet 
figured out how to enforce delays i.e. purgatory vs delay queue. I'll have a 
better idea once I look at the new purgatory implementation. Hopefully, this 
smaller patch is easier to review.

Added more testcases


Some locking changes for reading/creating the sensors


WIP patch


Sample usage in ReplicaManager


Updated patch for quotas. This patch does the following: 1. Add per-client 
metrics for both producer and consumers 2. Add configuration for quotas 3. 
Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 4. Add a DelayQueue in KafkaApi's that can be used to 
throttle any type of request. Implemented request throttling for produce and 
fetch requests. 5. Added unit and integration test cases. I've not yet added 
integration testcases testing the consumer delays.. will update the patch once 
those are ready


Incorporated Jun's comments


Adding javadoc


KAFKA-2084 - Moved the callbacks to ClientQuotaMetrics


Adding more configs


Don't quota replica traffic


KAFKA-2084


Fixing test failures


Minor fix


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
98429da34418f7f1deba1b5e44e2e6025212edb3 
  clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
544e120594de78c43581a980b1e4087b4fb98ccb 
  clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
  core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
387e387998fc3a6c9cb585dab02b5f77b0381fbf 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
e66710d2368334ece66f70d55f57b3f888262620 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
8014a5a6c362785539f24eb03d77278434614fe6 
  core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar