This thread might be of some use, it discusses the memory footprint when using metrics: https://groups.google.com/forum/#!topic/metrics-user/T68H70ea1Og
On Sat, Jul 13, 2013 at 5:35 PM, Swapnil Ghike (JIRA) <j...@apache.org>wrote: > > [ > https://issues.apache.org/jira/browse/KAFKA-656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707836#comment-13707836] > > Swapnil Ghike commented on KAFKA-656: > ------------------------------------- > > Hey Prashanth/Jonathan, have you had the time to run with this? If you're > busy, I would be interested in taking this up. > > > Add Quotas to Kafka > > ------------------- > > > > Key: KAFKA-656 > > URL: https://issues.apache.org/jira/browse/KAFKA-656 > > Project: Kafka > > Issue Type: New Feature > > Components: core > > Affects Versions: 0.8.1 > > Reporter: Jay Kreps > > Labels: project > > > > It would be nice to implement a quota system in Kafka to improve our > support for highly multi-tenant usage. The goal of this system would be to > prevent one naughty user from accidently overloading the whole cluster. > > There are several quantities we would want to track: > > 1. Requests pers second > > 2. Bytes written per second > > 3. Bytes read per second > > There are two reasonable groupings we would want to aggregate and > enforce these thresholds at: > > 1. Topic level > > 2. Client level (e.g. by client id from the request) > > When a request hits one of these limits we will simply reject it with a > QUOTA_EXCEEDED exception. > > To avoid suddenly breaking things without warning, we should ideally > support two thresholds: a soft threshold at which we produce some kind of > warning and a hard threshold at which we give the error. The soft threshold > could just be defined as 80% (or whatever) of the hard threshold. > > There are nuances to getting this right. If you measure second-by-second > a single burst may exceed the threshold, so we need a sustained measurement > over a period of time. > > Likewise when do we stop giving this error? To make this work right we > likely need to charge against the quota for request *attempts* not just > successful requests. Otherwise a client that is overloading the server will > just flap on and off--i.e. we would disable them for a period of time but > when we re-enabled them they would likely still be abusing us. > > It would be good to a wiki design on how this would all work as a > starting point for discussion. > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira >
