Hey Radai, Looking at the proposal, it looks like a major question is still unresolved? "This configuration parameter can either replace queued.max.requests completely, or co-exist with it (by way of either-or or respecting both bounds and not picking up new requests when either is hit)."
On Mon, Nov 7, 2016 at 1:08 PM, radai <radai.rosenbl...@gmail.com> wrote: > Hi, > > I would like to initiate a vote on KIP-72: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-72%3A+Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests > > The kip allows specifying a limit on the amount of memory allocated for > reading incoming requests into. This is useful for "sizing" a broker and > avoiding OOMEs under heavy load (as actually happens occasionally at > linkedin). > > I believe I've addressed most (all?) concerns brought up during the > discussion. > > To the best of my understanding this vote is about the goal and > public-facing changes related to the new proposed behavior, but as for > implementation, i have the code up here: > > https://github.com/radai-rosenblatt/kafka/tree/broker-memory-pool-with-muting > > and I've stress-tested it to work properly (meaning it chugs along and > throttles under loads that would DOS 10.0.1.0 code). > > I also believe that the primitives and "pattern"s introduced in this KIP > (namely the notion of a buffer pool and retrieving from / releasing to said > pool instead of allocating memory) are generally useful beyond the scope of > this KIP for both performance issues (allocating lots of short-lived large > buffers is a performance bottleneck) and other areas where memory limits > are a problem (KIP-81) > > Thank you, > > Radai. -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
