+1 (binding) On Tue, Nov 8, 2016 at 10:26 AM, radai <radai.rosenbl...@gmail.com> wrote: > I've updated the KIP page to specify the new config would co-exist with > "queued.max.request" to minimize the impact on compatibility. > > On Tue, Nov 8, 2016 at 7:02 AM, radai <radai.rosenbl...@gmail.com> wrote: > >> My personal opinion on this is that control of memory was always the >> intent behind queued.max.requests and so this KIP could completely obsolete >> it. >> For now its probably safest to leave it as-is (making memory-bound >> "opt-in") and revisit this at a later date >> >> On Mon, Nov 7, 2016 at 2:32 PM, Gwen Shapira <g...@confluent.io> wrote: >> >>> Hey Radai, >>> >>> Looking at the proposal, it looks like a major question is still >>> unresolved? >>> "This configuration parameter can either replace queued.max.requests >>> completely, or co-exist with it (by way of either-or or respecting >>> both bounds and not picking up new requests when either is hit)." >>> >>> On Mon, Nov 7, 2016 at 1:08 PM, radai <radai.rosenbl...@gmail.com> wrote: >>> > Hi, >>> > >>> > I would like to initiate a vote on KIP-72: >>> > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-72%3A+ >>> Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests >>> > >>> > The kip allows specifying a limit on the amount of memory allocated for >>> > reading incoming requests into. This is useful for "sizing" a broker and >>> > avoiding OOMEs under heavy load (as actually happens occasionally at >>> > linkedin). >>> > >>> > I believe I've addressed most (all?) concerns brought up during the >>> > discussion. >>> > >>> > To the best of my understanding this vote is about the goal and >>> > public-facing changes related to the new proposed behavior, but as for >>> > implementation, i have the code up here: >>> > >>> > https://github.com/radai-rosenblatt/kafka/tree/broker-memory >>> -pool-with-muting >>> > >>> > and I've stress-tested it to work properly (meaning it chugs along and >>> > throttles under loads that would DOS 10.0.1.0 code). >>> > >>> > I also believe that the primitives and "pattern"s introduced in this KIP >>> > (namely the notion of a buffer pool and retrieving from / releasing to >>> said >>> > pool instead of allocating memory) are generally useful beyond the >>> scope of >>> > this KIP for both performance issues (allocating lots of short-lived >>> large >>> > buffers is a performance bottleneck) and other areas where memory limits >>> > are a problem (KIP-81) >>> > >>> > Thank you, >>> > >>> > Radai. >>> >>> >>> >>> -- >>> Gwen Shapira >>> Product Manager | Confluent >>> 650.450.2760 | @gwenshap >>> Follow us: Twitter | blog >>> >> >>
-- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
