My personal opinion on this is that control of memory was always the intent behind queued.max.requests and so this KIP could completely obsolete it. For now its probably safest to leave it as-is (making memory-bound "opt-in") and revisit this at a later date
On Mon, Nov 7, 2016 at 2:32 PM, Gwen Shapira <g...@confluent.io> wrote: > Hey Radai, > > Looking at the proposal, it looks like a major question is still > unresolved? > "This configuration parameter can either replace queued.max.requests > completely, or co-exist with it (by way of either-or or respecting > both bounds and not picking up new requests when either is hit)." > > On Mon, Nov 7, 2016 at 1:08 PM, radai <radai.rosenbl...@gmail.com> wrote: > > Hi, > > > > I would like to initiate a vote on KIP-72: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 72%3A+Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests > > > > The kip allows specifying a limit on the amount of memory allocated for > > reading incoming requests into. This is useful for "sizing" a broker and > > avoiding OOMEs under heavy load (as actually happens occasionally at > > linkedin). > > > > I believe I've addressed most (all?) concerns brought up during the > > discussion. > > > > To the best of my understanding this vote is about the goal and > > public-facing changes related to the new proposed behavior, but as for > > implementation, i have the code up here: > > > > https://github.com/radai-rosenblatt/kafka/tree/broker- > memory-pool-with-muting > > > > and I've stress-tested it to work properly (meaning it chugs along and > > throttles under loads that would DOS 10.0.1.0 code). > > > > I also believe that the primitives and "pattern"s introduced in this KIP > > (namely the notion of a buffer pool and retrieving from / releasing to > said > > pool instead of allocating memory) are generally useful beyond the scope > of > > this KIP for both performance issues (allocating lots of short-lived > large > > buffers is a performance bottleneck) and other areas where memory limits > > are a problem (KIP-81) > > > > Thank you, > > > > Radai. > > > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
