Guozhang/Dong, Thank you for the feedback.
Guozhang : I have updated the section on co-existence of byte rate and request time quotas. Dong: I hadn't added much detail to the metrics and sensors since they are going to be very similar to the existing metrics and sensors. To avoid confusion, I have now added more detail. All metrics are in the group "quotaType" and all sensors have names starting with "quotaType" (where quotaType is Produce/Fetch/LeaderReplication/FollowerReplication/*IOThread*). So there will be no reuse of existing metrics/sensors. The new ones for request processing time based throttling will be completely independent of existing metrics/sensors, but will be consistent in format. The existing throttle_time_ms field in produce/fetch responses will not be impacted by this KIP. That will continue to return byte-rate based throttling times. In addition, a new field request_throttle_time_ms will be added to return request quota based throttling times. These will be exposed as new metrics on the client-side. Since all metrics and sensors are different for each type of quota, I believe there is already sufficient metrics to monitor throttling on both client and broker side for each type of throttling. Regards, Rajini On Thu, Feb 23, 2017 at 4:32 AM, Dong Lin <lindon...@gmail.com> wrote: > Hey Rajini, > > I think it makes a lot of sense to use io_thread_units as metric to quota > user's traffic here. LGTM overall. I have some questions regarding sensors. > > - Can you be more specific in the KIP what sensors will be added? For > example, it will be useful to specify the name and attributes of these new > sensors. > > - We currently have throttle-time and queue-size for byte-rate based quota. > Are you going to have separate throttle-time and queue-size for requests > throttled by io_thread_unit-based quota, or will they share the same > sensor? > > - Does the throttle-time in the ProduceResponse and FetchResponse contains > time due to io_thread_unit-based quota? > > - Currently kafka server doesn't not provide any log or metrics that tells > whether any given clientId (or user) is throttled. This is not too bad > because we can still check the client-side byte-rate metric to validate > whether a given client is throttled. But with this io_thread_unit, there > will be no way to validate whether a given client is slow because it has > exceeded its io_thread_unit limit. It is necessary for user to be able to > know this information to figure how whether they have reached there quota > limit. How about we add log4j log on the server side to periodically print > the (client_id, byte-rate-throttle-time, io-thread-unit-throttle-time) so > that kafka administrator can figure those users that have reached their > limit and act accordingly? > > Thanks, > Dong > > > > > > On Wed, Feb 22, 2017 at 4:46 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Made a pass over the doc, overall LGTM except a minor comment on the > > throttling implementation: > > > > Stated as "Request processing time throttling will be applied on top if > > necessary." I thought that it meant the request processing time > throttling > > is applied first, but continue reading I found it actually meant to apply > > produce / fetch byte rate throttling first. > > > > Also the last sentence "The remaining delay if any is applied to the > > response." is a bit confusing to me. Maybe rewording it a bit? > > > > > > Guozhang > > > > > > On Wed, Feb 22, 2017 at 3:24 PM, Jun Rao <j...@confluent.io> wrote: > > > > > Hi, Rajini, > > > > > > Thanks for the updated KIP. The latest proposal looks good to me. > > > > > > Jun > > > > > > On Wed, Feb 22, 2017 at 2:19 PM, Rajini Sivaram < > rajinisiva...@gmail.com > > > > > > wrote: > > > > > > > Jun/Roger, > > > > > > > > Thank you for the feedback. > > > > > > > > 1. I have updated the KIP to use absolute units instead of > percentage. > > > The > > > > property is called* io_thread_units* to align with the thread count > > > > property *num.io.threads*. When we implement network thread > utilization > > > > quotas, we can add another property *network_thread_units.* > > > > > > > > 2. ControlledShutdown is already listed under the exempt requests. > Jun, > > > did > > > > you mean a different request that needs to be added? The four > requests > > > > currently exempt in the KIP are StopReplica, ControlledShutdown, > > > > LeaderAndIsr and UpdateMetadata. These are controlled using > > ClusterAction > > > > ACL, so it is easy to exclude and only throttle if unauthorized. I > > wasn't > > > > sure if there are other requests used only for inter-broker that > needed > > > to > > > > be excluded. > > > > > > > > 3. I was thinking the smallest change would be to replace all > > references > > > to > > > > *requestChannel.sendResponse()* with a local method > > > > *sendResponseMaybeThrottle()* that does the throttling if any plus > send > > > > response. If we throttle first in *KafkaApis.handle()*, the time > spent > > > > within the method handling the request will not be recorded or used > in > > > > throttling. We can look into this again when the PR is ready for > > review. > > > > > > > > Regards, > > > > > > > > Rajini > > > > > > > > > > > > > > > > On Wed, Feb 22, 2017 at 5:55 PM, Roger Hoover < > roger.hoo...@gmail.com> > > > > wrote: > > > > > > > > > Great to see this KIP and the excellent discussion. > > > > > > > > > > To me, Jun's suggestion makes sense. If my application is > allocated > > 1 > > > > > request handler unit, then it's as if I have a Kafka broker with a > > > single > > > > > request handler thread dedicated to me. That's the most I can use, > > at > > > > > least. That allocation doesn't change even if an admin later > > increases > > > > the > > > > > size of the request thread pool on the broker. It's similar to the > > CPU > > > > > abstraction that VMs and containers get from hypervisors or OS > > > > schedulers. > > > > > While different client access patterns can use wildly different > > amounts > > > > of > > > > > request thread resources per request, a given application will > > > generally > > > > > have a stable access pattern and can figure out empirically how > many > > > > > "request thread units" it needs to meet it's throughput/latency > > goals. > > > > > > > > > > Cheers, > > > > > > > > > > Roger > > > > > > > > > > On Wed, Feb 22, 2017 at 8:53 AM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > > > Hi, Rajini, > > > > > > > > > > > > Thanks for the updated KIP. A few more comments. > > > > > > > > > > > > 1. A concern of request_time_percent is that it's not an absolute > > > > value. > > > > > > Let's say you give a user a 10% limit. If the admin doubles the > > > number > > > > of > > > > > > request handler threads, that user now actually has twice the > > > absolute > > > > > > capacity. This may confuse people a bit. So, perhaps setting the > > > quota > > > > > > based on an absolute request thread unit is better. > > > > > > > > > > > > 2. ControlledShutdownRequest is also an inter-broker request and > > > needs > > > > to > > > > > > be excluded from throttling. > > > > > > > > > > > > 3. Implementation wise, I am wondering if it's simpler to apply > the > > > > > request > > > > > > time throttling first in KafkaApis.handle(). Otherwise, we will > > need > > > to > > > > > add > > > > > > the throttling logic in each type of request. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Wed, Feb 22, 2017 at 5:58 AM, Rajini Sivaram < > > > > rajinisiva...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Jun, > > > > > > > > > > > > > > Thank you for the review. > > > > > > > > > > > > > > I have reverted to the original KIP that throttles based on > > request > > > > > > handler > > > > > > > utilization. At the moment, it uses percentage, but I am happy > to > > > > > change > > > > > > to > > > > > > > a fraction (out of 1 instead of 100) if required. I have added > > the > > > > > > examples > > > > > > > from this discussion to the KIP. Also added a "Future Work" > > section > > > > to > > > > > > > address network thread utilization. The configuration is named > > > > > > > "request_time_percent" with the expectation that it can also be > > > used > > > > as > > > > > > the > > > > > > > limit for network thread utilization when that is implemented, > so > > > > that > > > > > > > users have to set only one config for the two and not have to > > worry > > > > > about > > > > > > > the internal distribution of the work between the two thread > > pools > > > in > > > > > > > Kafka. > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 22, 2017 at 12:23 AM, Jun Rao <j...@confluent.io> > > > wrote: > > > > > > > > > > > > > > > Hi, Rajini, > > > > > > > > > > > > > > > > Thanks for the proposal. > > > > > > > > > > > > > > > > The benefit of using the request processing time over the > > request > > > > > rate > > > > > > is > > > > > > > > exactly what people have said. I will just expand that a bit. > > > > > Consider > > > > > > > the > > > > > > > > following case. The producer sends a produce request with a > > 10MB > > > > > > message > > > > > > > > but compressed to 100KB with gzip. The decompression of the > > > message > > > > > on > > > > > > > the > > > > > > > > broker could take 10-15 seconds, during which time, a request > > > > handler > > > > > > > > thread is completely blocked. In this case, neither the > byte-in > > > > quota > > > > > > nor > > > > > > > > the request rate quota may be effective in protecting the > > broker. > > > > > > > Consider > > > > > > > > another case. A consumer group starts with 10 instances and > > later > > > > on > > > > > > > > switches to 20 instances. The request rate will likely > double, > > > but > > > > > the > > > > > > > > actually load on the broker may not double since each fetch > > > request > > > > > > only > > > > > > > > contains half of the partitions. Request rate quota may not > be > > > easy > > > > > to > > > > > > > > configure in this case. > > > > > > > > > > > > > > > > What we really want is to be able to prevent a client from > > using > > > > too > > > > > > much > > > > > > > > of the server side resources. In this particular KIP, this > > > resource > > > > > is > > > > > > > the > > > > > > > > capacity of the request handler threads. I agree that it may > > not > > > be > > > > > > > > intuitive for the users to determine how to set the right > > limit. > > > > > > However, > > > > > > > > this is not completely new and has been done in the container > > > world > > > > > > > > already. For example, Linux cgroup ( > https://access.redhat.com/ > > > > > > > > documentation/en-US/Red_Hat_Enterprise_Linux/6/html/ > > > > > > > > Resource_Management_Guide/sec-cpu.html) has the concept of > > > > > > > > cpu.cfs_quota_us, > > > > > > > > which specifies the total amount of time in microseconds for > > > which > > > > > all > > > > > > > > tasks in a cgroup can run during a one second period. We can > > > > > > potentially > > > > > > > > model the request handler threads in a similar way. For > > example, > > > > each > > > > > > > > request handler thread can be 1 request handler unit and the > > > admin > > > > > can > > > > > > > > configure a limit on how many units (say 0.01) a client can > > have. > > > > > > > > > > > > > > > > Regarding not throttling the internal broker to broker > > requests. > > > We > > > > > > could > > > > > > > > do that. Alternatively, we could just let the admin > configure a > > > > high > > > > > > > limit > > > > > > > > for the kafka user (it may not be able to do that easily > based > > on > > > > > > > clientId > > > > > > > > though). > > > > > > > > > > > > > > > > Ideally we want to be able to protect the utilization of the > > > > network > > > > > > > thread > > > > > > > > pool too. The difficult is mostly what Rajini said: (1) The > > > > mechanism > > > > > > for > > > > > > > > throttling the requests is through Purgatory and we will have > > to > > > > > think > > > > > > > > through how to integrate that into the network layer. (2) In > > the > > > > > > network > > > > > > > > layer, currently we know the user, but not the clientId of > the > > > > > request. > > > > > > > So, > > > > > > > > it's a bit tricky to throttle based on clientId there. Plus, > > the > > > > > > byteOut > > > > > > > > quota can already protect the network thread utilization for > > > fetch > > > > > > > > requests. So, if we can't figure out this part right now, > just > > > > > focusing > > > > > > > on > > > > > > > > the request handling threads for this KIP is still a useful > > > > feature. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 21, 2017 at 4:27 AM, Rajini Sivaram < > > > > > > rajinisiva...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thank you all for the feedback. > > > > > > > > > > > > > > > > > > Jay: I have removed exemption for consumer heartbeat etc. > > Agree > > > > > that > > > > > > > > > protecting the cluster is more important than protecting > > > > individual > > > > > > > apps. > > > > > > > > > Have retained the exemption for StopReplicat/LeaderAndIsr > > etc, > > > > > these > > > > > > > are > > > > > > > > > throttled only if authorization fails (so can't be used for > > DoS > > > > > > attacks > > > > > > > > in > > > > > > > > > a secure cluster, but allows inter-broker requests to > > complete > > > > > > without > > > > > > > > > delays). > > > > > > > > > > > > > > > > > > I will wait another day to see if these is any objection to > > > > quotas > > > > > > > based > > > > > > > > on > > > > > > > > > request processing time (as opposed to request rate) and if > > > there > > > > > are > > > > > > > no > > > > > > > > > objections, I will revert to the original proposal with > some > > > > > changes. > > > > > > > > > > > > > > > > > > The original proposal was only including the time used by > the > > > > > request > > > > > > > > > handler threads (that made calculation easy). I think the > > > > > suggestion > > > > > > is > > > > > > > > to > > > > > > > > > include the time spent in the network threads as well since > > > that > > > > > may > > > > > > be > > > > > > > > > significant. As Jay pointed out, it is more complicated to > > > > > calculate > > > > > > > the > > > > > > > > > total available CPU time and convert to a ratio when there > > *m* > > > > I/O > > > > > > > > threads > > > > > > > > > and *n* network threads. ThreadMXBean#getThreadCPUTime() > may > > > > give > > > > > us > > > > > > > > what > > > > > > > > > we want, but it can be very expensive on some platforms. As > > > > Becket > > > > > > and > > > > > > > > > Guozhang have pointed out, we do have several time > > measurements > > > > > > already > > > > > > > > for > > > > > > > > > generating metrics that we could use, though we might want > to > > > > > switch > > > > > > to > > > > > > > > > nanoTime() instead of currentTimeMillis() since some of the > > > > values > > > > > > for > > > > > > > > > small requests may be < 1ms. But rather than add up the > time > > > > spent > > > > > in > > > > > > > I/O > > > > > > > > > thread and network thread, wouldn't it be better to convert > > the > > > > > time > > > > > > > > spent > > > > > > > > > on each thread into a separate ratio? UserA has a request > > quota > > > > of > > > > > > 5%. > > > > > > > > Can > > > > > > > > > we take that to mean that UserA can use 5% of the time on > > > network > > > > > > > threads > > > > > > > > > and 5% of the time on I/O threads? If either is exceeded, > the > > > > > > response > > > > > > > is > > > > > > > > > throttled - it would mean maintaining two sets of metrics > for > > > the > > > > > two > > > > > > > > > durations, but would result in more meaningful ratios. We > > could > > > > > > define > > > > > > > > two > > > > > > > > > quota limits (UserA has 5% of request threads and 10% of > > > network > > > > > > > > threads), > > > > > > > > > but that seems unnecessary and harder to explain to users. > > > > > > > > > > > > > > > > > > Back to why and how quotas are applied to network thread > > > > > utilization: > > > > > > > > > a) In the case of fetch, the time spent in the network > > thread > > > > may > > > > > be > > > > > > > > > significant and I can see the need to include this. Are > there > > > > other > > > > > > > > > requests where the network thread utilization is > significant? > > > In > > > > > the > > > > > > > case > > > > > > > > > of fetch, request handler thread utilization would throttle > > > > clients > > > > > > > with > > > > > > > > > high request rate, low data volume and fetch byte rate > quota > > > will > > > > > > > > throttle > > > > > > > > > clients with high data volume. Network thread utilization > is > > > > > perhaps > > > > > > > > > proportional to the data volume. I am wondering if we even > > need > > > > to > > > > > > > > throttle > > > > > > > > > based on network thread utilization or whether the data > > volume > > > > > quota > > > > > > > > covers > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > b) At the moment, we record and check for quota violation > at > > > the > > > > > same > > > > > > > > time. > > > > > > > > > If a quota is violated, the response is delayed. Using > Jay'e > > > > > example > > > > > > of > > > > > > > > > disk reads for fetches happening in the network thread, We > > > can't > > > > > > record > > > > > > > > and > > > > > > > > > delay a response after the disk reads. We could record the > > time > > > > > spent > > > > > > > on > > > > > > > > > the network thread when the response is complete and > > introduce > > > a > > > > > > delay > > > > > > > > for > > > > > > > > > handling a subsequent request (separate out recording and > > quota > > > > > > > violation > > > > > > > > > handling in the case of network thread overload). Does that > > > make > > > > > > sense? > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 21, 2017 at 2:58 AM, Becket Qin < > > > > becket....@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hey Jay, > > > > > > > > > > > > > > > > > > > > Yeah, I agree that enforcing the CPU time is a little > > > tricky. I > > > > > am > > > > > > > > > thinking > > > > > > > > > > that maybe we can use the existing request statistics. > They > > > are > > > > > > > already > > > > > > > > > > very detailed so we can probably see the approximate CPU > > time > > > > > from > > > > > > > it, > > > > > > > > > e.g. > > > > > > > > > > something like (total_time - request/response_queue_time > - > > > > > > > > remote_time). > > > > > > > > > > > > > > > > > > > > I agree with Guozhang that when a user is throttled it is > > > > likely > > > > > > that > > > > > > > > we > > > > > > > > > > need to see if anything has went wrong first, and if the > > > users > > > > > are > > > > > > > well > > > > > > > > > > behaving and just need more resources, we will have to > bump > > > up > > > > > the > > > > > > > > quota > > > > > > > > > > for them. It is true that pre-allocating CPU time quota > > > > precisely > > > > > > for > > > > > > > > the > > > > > > > > > > users is difficult. So in practice it would probably be > > more > > > > like > > > > > > > first > > > > > > > > > set > > > > > > > > > > a relative high protective CPU time quota for everyone > and > > > > > increase > > > > > > > > that > > > > > > > > > > for some individual clients on demand. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 20, 2017 at 5:48 PM, Guozhang Wang < > > > > > wangg...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > This is a great proposal, glad to see it happening. > > > > > > > > > > > > > > > > > > > > > > I am inclined to the CPU throttling, or more > specifically > > > > > > > processing > > > > > > > > > time > > > > > > > > > > > ratio instead of the request rate throttling as well. > > > Becket > > > > > has > > > > > > > very > > > > > > > > > > well > > > > > > > > > > > summed my rationales above, and one thing to add here > is > > > that > > > > > the > > > > > > > > > former > > > > > > > > > > > has a good support for both "protecting against rogue > > > > clients" > > > > > as > > > > > > > > well > > > > > > > > > as > > > > > > > > > > > "utilizing a cluster for multi-tenancy usage": when > > > thinking > > > > > > about > > > > > > > > how > > > > > > > > > to > > > > > > > > > > > explain this to the end users, I find it actually more > > > > natural > > > > > > than > > > > > > > > the > > > > > > > > > > > request rate since as mentioned above, different > requests > > > > will > > > > > > have > > > > > > > > > quite > > > > > > > > > > > different "cost", and Kafka today already have various > > > > request > > > > > > > types > > > > > > > > > > > (produce, fetch, admin, metadata, etc), because of that > > the > > > > > > request > > > > > > > > > rate > > > > > > > > > > > throttling may not be as effective unless it is set > very > > > > > > > > > conservatively. > > > > > > > > > > > > > > > > > > > > > > Regarding to user reactions when they are throttled, I > > > think > > > > it > > > > > > may > > > > > > > > > > differ > > > > > > > > > > > case-by-case, and need to be discovered / guided by > > looking > > > > at > > > > > > > > relative > > > > > > > > > > > metrics. So in other words users would not expect to > get > > > > > > additional > > > > > > > > > > > information by simply being told "hey, you are > > throttled", > > > > > which > > > > > > is > > > > > > > > all > > > > > > > > > > > what throttling does; they need to take a follow-up > step > > > and > > > > > see > > > > > > > > "hmm, > > > > > > > > > > I'm > > > > > > > > > > > throttled probably because of ..", which is by looking > at > > > > other > > > > > > > > metric > > > > > > > > > > > values: e.g. whether I'm bombarding the brokers with > > > metadata > > > > > > > > request, > > > > > > > > > > > which are usually cheap to handle but I'm sending > > thousands > > > > per > > > > > > > > second; > > > > > > > > > > or > > > > > > > > > > > is it because I'm catching up and hence sending very > > heavy > > > > > > fetching > > > > > > > > > > request > > > > > > > > > > > with large min.bytes, etc. > > > > > > > > > > > > > > > > > > > > > > Regarding to the implementation, as once discussed with > > > Jun, > > > > > this > > > > > > > > seems > > > > > > > > > > not > > > > > > > > > > > very difficult since today we are already collecting > the > > > > > "thread > > > > > > > pool > > > > > > > > > > > utilization" metrics, which is a single percentage > > > > > > > > "aggregateIdleMeter" > > > > > > > > > > > value; but we are already effectively aggregating it > for > > > each > > > > > > > > requests > > > > > > > > > in > > > > > > > > > > > KafkaRequestHandler, and we can just extend it by > > recording > > > > the > > > > > > > > source > > > > > > > > > > > client id when handling them and aggregating by > clientId > > as > > > > > well > > > > > > as > > > > > > > > the > > > > > > > > > > > total aggregate. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 20, 2017 at 4:27 PM, Jay Kreps < > > > j...@confluent.io > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hey Becket/Rajini, > > > > > > > > > > > > > > > > > > > > > > > > When I thought about it more deeply I came around to > > the > > > > > > "percent > > > > > > > > of > > > > > > > > > > > > processing time" metric too. It seems a lot closer to > > the > > > > > thing > > > > > > > we > > > > > > > > > > > actually > > > > > > > > > > > > care about and need to protect. I also think this > would > > > be > > > > a > > > > > > very > > > > > > > > > > useful > > > > > > > > > > > > metric even in the absence of throttling just to > debug > > > > whose > > > > > > > using > > > > > > > > > > > > capacity. > > > > > > > > > > > > > > > > > > > > > > > > Two problems to consider: > > > > > > > > > > > > > > > > > > > > > > > > 1. I agree that for the user it is understandable > > what > > > > > lead > > > > > > to > > > > > > > > > their > > > > > > > > > > > > being throttled, but it is a bit hard to figure > out > > > the > > > > > safe > > > > > > > > range > > > > > > > > > > for > > > > > > > > > > > > them. i.e. if I have a new app that will send 200 > > > > > > > messages/sec I > > > > > > > > > can > > > > > > > > > > > > probably reason that I'll be under the throttling > > > limit > > > > of > > > > > > 300 > > > > > > > > > > > req/sec. > > > > > > > > > > > > However if I need to be under a 10% CPU resources > > > limit > > > > it > > > > > > may > > > > > > > > be > > > > > > > > > a > > > > > > > > > > > bit > > > > > > > > > > > > harder for me to know a priori if i will or won't. > > > > > > > > > > > > 2. Calculating the available CPU time is a bit > > > difficult > > > > > > since > > > > > > > > > there > > > > > > > > > > > are > > > > > > > > > > > > actually two thread pools--the I/O threads and the > > > > network > > > > > > > > > threads. > > > > > > > > > > I > > > > > > > > > > > > think > > > > > > > > > > > > it might be workable to count just the I/O thread > > time > > > > as > > > > > in > > > > > > > the > > > > > > > > > > > > proposal, > > > > > > > > > > > > but the network thread work is actually > non-trivial > > > > (e.g. > > > > > > all > > > > > > > > the > > > > > > > > > > disk > > > > > > > > > > > > reads for fetches happen in that thread). If you > > count > > > > > both > > > > > > > the > > > > > > > > > > > network > > > > > > > > > > > > and > > > > > > > > > > > > I/O threads it can skew things a bit. E.g. say you > > > have > > > > 50 > > > > > > > > network > > > > > > > > > > > > threads, > > > > > > > > > > > > 10 I/O threads, and 8 cores, what is the available > > cpu > > > > > time > > > > > > > > > > available > > > > > > > > > > > > in a > > > > > > > > > > > > second? I suppose this is a problem whenever you > > have > > > a > > > > > > > > bottleneck > > > > > > > > > > > > between > > > > > > > > > > > > I/O and network threads or if you end up > > significantly > > > > > > > > > > > over-provisioning > > > > > > > > > > > > one pool (both of which are hard to avoid). > > > > > > > > > > > > > > > > > > > > > > > > An alternative for CPU throttling would be to use > this > > > api: > > > > > > > > > > > > http://docs.oracle.com/javase/ > > 1.5.0/docs/api/java/lang/ > > > > > > > > > > > > management/ThreadMXBean.html#getThreadCpuTime(long) > > > > > > > > > > > > > > > > > > > > > > > > That would let you track actual CPU usage across the > > > > network, > > > > > > I/O > > > > > > > > > > > threads, > > > > > > > > > > > > and purgatory threads and look at it as a percentage > of > > > > total > > > > > > > > cores. > > > > > > > > > I > > > > > > > > > > > > think this fixes many problems in the reliability of > > the > > > > > > metric. > > > > > > > > It's > > > > > > > > > > > > meaning is slightly different as it is just CPU (you > > > don't > > > > > get > > > > > > > > > charged > > > > > > > > > > > for > > > > > > > > > > > > time blocking on I/O) but that may be okay because we > > > > already > > > > > > > have > > > > > > > > a > > > > > > > > > > > > throttle on I/O. The downside is I think it is > possible > > > > this > > > > > > api > > > > > > > > can > > > > > > > > > be > > > > > > > > > > > > disabled or isn't always available and it may also be > > > > > expensive > > > > > > > > (also > > > > > > > > > > > I've > > > > > > > > > > > > never used it so not sure if it really works the way > i > > > > > think). > > > > > > > > > > > > > > > > > > > > > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 20, 2017 at 3:17 PM, Becket Qin < > > > > > > > becket....@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > If the purpose of the KIP is only to protect the > > > cluster > > > > > from > > > > > > > > being > > > > > > > > > > > > > overwhelmed by crazy clients and is not intended to > > > > address > > > > > > > > > resource > > > > > > > > > > > > > allocation problem among the clients, I am > wondering > > if > > > > > using > > > > > > > > > request > > > > > > > > > > > > > handling time quota (CPU time quota) is a better > > > option. > > > > > Here > > > > > > > are > > > > > > > > > the > > > > > > > > > > > > > reasons: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. request handling time quota has better > protection. > > > Say > > > > > we > > > > > > > have > > > > > > > > > > > request > > > > > > > > > > > > > rate quota and set that to some value like 100 > > > > > requests/sec, > > > > > > it > > > > > > > > is > > > > > > > > > > > > possible > > > > > > > > > > > > > that some of the requests are very expensive > actually > > > > take > > > > > a > > > > > > > lot > > > > > > > > of > > > > > > > > > > > time > > > > > > > > > > > > to > > > > > > > > > > > > > handle. In that case a few clients may still > occupy a > > > lot > > > > > of > > > > > > > CPU > > > > > > > > > time > > > > > > > > > > > > even > > > > > > > > > > > > > the request rate is low. Arguably we can carefully > > set > > > > > > request > > > > > > > > rate > > > > > > > > > > > quota > > > > > > > > > > > > > for each request and client id combination, but it > > > could > > > > > > still > > > > > > > be > > > > > > > > > > > tricky > > > > > > > > > > > > to > > > > > > > > > > > > > get it right for everyone. > > > > > > > > > > > > > > > > > > > > > > > > > > If we use the request time handling quota, we can > > > simply > > > > > say > > > > > > no > > > > > > > > > > clients > > > > > > > > > > > > can > > > > > > > > > > > > > take up to more than 30% of the total request > > handling > > > > > > capacity > > > > > > > > > > > (measured > > > > > > > > > > > > > by time), regardless of the difference among > > different > > > > > > requests > > > > > > > > or > > > > > > > > > > what > > > > > > > > > > > > is > > > > > > > > > > > > > the client doing. In this case maybe we can quota > all > > > the > > > > > > > > requests > > > > > > > > > if > > > > > > > > > > > we > > > > > > > > > > > > > want to. > > > > > > > > > > > > > > > > > > > > > > > > > > 2. The main benefit of using request rate limit is > > that > > > > it > > > > > > > seems > > > > > > > > > more > > > > > > > > > > > > > intuitive. It is true that it is probably easier to > > > > explain > > > > > > to > > > > > > > > the > > > > > > > > > > user > > > > > > > > > > > > > what does that mean. However, in practice it looks > > the > > > > > impact > > > > > > > of > > > > > > > > > > > request > > > > > > > > > > > > > rate quota is not more quantifiable than the > request > > > > > handling > > > > > > > > time > > > > > > > > > > > quota. > > > > > > > > > > > > > Unlike the byte rate quota, it is still difficult > to > > > > give a > > > > > > > > number > > > > > > > > > > > about > > > > > > > > > > > > > impact of throughput or latency when a request rate > > > quota > > > > > is > > > > > > > hit. > > > > > > > > > So > > > > > > > > > > it > > > > > > > > > > > > is > > > > > > > > > > > > > not better than the request handling time quota. In > > > fact > > > > I > > > > > > feel > > > > > > > > it > > > > > > > > > is > > > > > > > > > > > > > clearer to tell user that "you are limited because > > you > > > > have > > > > > > > taken > > > > > > > > > 30% > > > > > > > > > > > of > > > > > > > > > > > > > the CPU time on the broker" than otherwise > something > > > like > > > > > > "your > > > > > > > > > > request > > > > > > > > > > > > > rate quota on metadata request has reached". > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 20, 2017 at 2:23 PM, Jay Kreps < > > > > > j...@confluent.io > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I think this proposal makes a lot of sense > > > (especially > > > > > now > > > > > > > that > > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > > > > > oriented around request rate) and fills the > biggest > > > > > > remaining > > > > > > > > gap > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > > multi-tenancy story. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think for intra-cluster communication > > (StopReplica, > > > > > etc) > > > > > > we > > > > > > > > > could > > > > > > > > > > > > avoid > > > > > > > > > > > > > > throttling entirely. You can secure or otherwise > > > > > lock-down > > > > > > > the > > > > > > > > > > > cluster > > > > > > > > > > > > > > communication to avoid any unauthorized external > > > party > > > > > from > > > > > > > > > trying > > > > > > > > > > to > > > > > > > > > > > > > > initiate these requests. As a result we are as > > likely > > > > to > > > > > > > cause > > > > > > > > > > > problems > > > > > > > > > > > > > as > > > > > > > > > > > > > > solve them by throttling these, right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm not so sure that we should exempt the > consumer > > > > > requests > > > > > > > > such > > > > > > > > > as > > > > > > > > > > > > > > heartbeat. It's true that if we throttle an app's > > > > > heartbeat > > > > > > > > > > requests > > > > > > > > > > > it > > > > > > > > > > > > > may > > > > > > > > > > > > > > cause it to fall out of its consumer group. > However > > > if > > > > we > > > > > > > don't > > > > > > > > > > > > throttle > > > > > > > > > > > > > it > > > > > > > > > > > > > > it may DDOS the cluster if the heartbeat interval > > is > > > > set > > > > > > > > > > incorrectly > > > > > > > > > > > or > > > > > > > > > > > > > if > > > > > > > > > > > > > > some client in some language has a bug. I think > the > > > > > policy > > > > > > > with > > > > > > > > > > this > > > > > > > > > > > > kind > > > > > > > > > > > > > > of throttling is to protect the cluster above any > > > > > > individual > > > > > > > > app, > > > > > > > > > > > > right? > > > > > > > > > > > > > I > > > > > > > > > > > > > > think in general this should be okay since for > most > > > > > > > deployments > > > > > > > > > > this > > > > > > > > > > > > > > setting is meant as more of a safety valve---that > > is > > > > > rather > > > > > > > > than > > > > > > > > > > set > > > > > > > > > > > > > > something very close to what you expect to need > > (say > > > 2 > > > > > > > req/sec > > > > > > > > or > > > > > > > > > > > > > whatever) > > > > > > > > > > > > > > you would have something quite high (like 100 > > > req/sec) > > > > > with > > > > > > > > this > > > > > > > > > > > meant > > > > > > > > > > > > to > > > > > > > > > > > > > > prevent a client gone crazy. I think when used > this > > > way > > > > > > > > allowing > > > > > > > > > > > those > > > > > > > > > > > > to > > > > > > > > > > > > > > be throttled would actually provide meaningful > > > > > protection. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 17, 2017 at 9:05 AM, Rajini Sivaram < > > > > > > > > > > > > rajinisiva...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have just created KIP-124 to introduce > request > > > rate > > > > > > > quotas > > > > > > > > to > > > > > > > > > > > > Kafka: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/ > > > > confluence/display/KAFKA/KIP- > > > > > > > > > > > > > > > 124+-+Request+rate+quotas > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The proposal is for a simple percentage request > > > > > handling > > > > > > > time > > > > > > > > > > quota > > > > > > > > > > > > > that > > > > > > > > > > > > > > > can be allocated to *<client-id>*, *<user>* or > > > > *<user, > > > > > > > > > > client-id>*. > > > > > > > > > > > > > There > > > > > > > > > > > > > > > are a few other suggestions also under > "Rejected > > > > > > > > alternatives". > > > > > > > > > > > > > Feedback > > > > > > > > > > > > > > > and suggestions are welcome. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > -- Guozhang > > >
