Hi Radai, Thanks for the proposal. I think it makes sense to be able to limit the request queue by bytes. I haven't made up my mind on whether having both limits is better than having a single one yet, but we probably need to make a call on that before we can start a vote.
Just a quick point with regards to ByteBoundedBlockingQueue. It's not currently used in the code and the existing unit tests are very basic. Also, looking at the implementation, it has two obvious bugs (it fails if one passes None for `sizeFunction` and `currentByteSize` is not final and it should be to comply with the JMM). So, we probably need to extend the unit tests for that class if we want to use it (ideally there would be multi-threaded tests too). It also is a bit inefficient as its adds additional locks on top of the LinkedBlockingQueue ones, but the benchmarks should hopefully show if that is something that needs to be addressed or not. Ismael On Thu, Aug 4, 2016 at 7:28 PM, radai <radai.rosenbl...@gmail.com> wrote: > Hello, > > I'd like to initiate a discussion about > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-72%3A+Allow+Sizing+Incoming+Request+Queue+in+Bytes > > The goal of the KIP is to allow configuring a bound on the capacity (as in > bytes of memory used) of the incoming request queue, in addition to the > current bound on the number of messages. > > This comes after several incidents at Linkedin where a sudden "spike" of > large message batches caused an out of memory exception. > > Thank you, > > Radai >
