Thanks for the proposal, Radai. +1 On Fri, Aug 5, 2016 at 8:38 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> Hi Radai, > > Thanks for the proposal. I think it makes sense to be able to limit the > request queue by bytes. I haven't made up my mind on whether having both > limits is better than having a single one yet, but we probably need to make > a call on that before we can start a vote. > > Just a quick point with regards to ByteBoundedBlockingQueue. It's not > currently used in the code and the existing unit tests are very basic. > Also, looking at the implementation, it has two obvious bugs (it fails if > one passes None for `sizeFunction` and `currentByteSize` is not final and > it should be to comply with the JMM). So, we probably need to extend the > unit tests for that class if we want to use it (ideally there would be > multi-threaded tests too). It also is a bit inefficient as its adds > additional locks on top of the LinkedBlockingQueue ones, but the benchmarks > should hopefully show if that is something that needs to be addressed or > not. > > Ismael > > On Thu, Aug 4, 2016 at 7:28 PM, radai <radai.rosenbl...@gmail.com> wrote: > > > Hello, > > > > I'd like to initiate a discussion about > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 72%3A+Allow+Sizing+Incoming+Request+Queue+in+Bytes > > > > The goal of the KIP is to allow configuring a bound on the capacity (as > in > > bytes of memory used) of the incoming request queue, in addition to the > > current bound on the number of messages. > > > > This comes after several incidents at Linkedin where a sudden "spike" of > > large message batches caused an out of memory exception. > > > > Thank you, > > > > Radai > > >
