I've updated the KIP page to specify the new config would co-exist with "queued.max.request" to minimize the impact on compatibility.
On Tue, Nov 8, 2016 at 7:02 AM, radai <radai.rosenbl...@gmail.com> wrote: > My personal opinion on this is that control of memory was always the > intent behind queued.max.requests and so this KIP could completely obsolete > it. > For now its probably safest to leave it as-is (making memory-bound > "opt-in") and revisit this at a later date > > On Mon, Nov 7, 2016 at 2:32 PM, Gwen Shapira <g...@confluent.io> wrote: > >> Hey Radai, >> >> Looking at the proposal, it looks like a major question is still >> unresolved? >> "This configuration parameter can either replace queued.max.requests >> completely, or co-exist with it (by way of either-or or respecting >> both bounds and not picking up new requests when either is hit)." >> >> On Mon, Nov 7, 2016 at 1:08 PM, radai <radai.rosenbl...@gmail.com> wrote: >> > Hi, >> > >> > I would like to initiate a vote on KIP-72: >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-72%3A+ >> Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests >> > >> > The kip allows specifying a limit on the amount of memory allocated for >> > reading incoming requests into. This is useful for "sizing" a broker and >> > avoiding OOMEs under heavy load (as actually happens occasionally at >> > linkedin). >> > >> > I believe I've addressed most (all?) concerns brought up during the >> > discussion. >> > >> > To the best of my understanding this vote is about the goal and >> > public-facing changes related to the new proposed behavior, but as for >> > implementation, i have the code up here: >> > >> > https://github.com/radai-rosenblatt/kafka/tree/broker-memory >> -pool-with-muting >> > >> > and I've stress-tested it to work properly (meaning it chugs along and >> > throttles under loads that would DOS 10.0.1.0 code). >> > >> > I also believe that the primitives and "pattern"s introduced in this KIP >> > (namely the notion of a buffer pool and retrieving from / releasing to >> said >> > pool instead of allocating memory) are generally useful beyond the >> scope of >> > this KIP for both performance issues (allocating lots of short-lived >> large >> > buffers is a performance bottleneck) and other areas where memory limits >> > are a problem (KIP-81) >> > >> > Thank you, >> > >> > Radai. >> >> >> >> -- >> Gwen Shapira >> Product Manager | Confluent >> 650.450.2760 | @gwenshap >> Follow us: Twitter | blog >> > >
