Re: ElasticByteBufferPool is ever growing thus can cause memory leak.

2022-02-04 Thread Mukund Madhav Thakur
Hi Everyone, I have committed the initial work of vectored IO in the feature branch. https://github.com/apache/hadoop/commits/feature-vectored-io Created an epic to track the future work https://issues.apache.org/jira/browse/HADOOP-18103. Please try this out and review and let me know what you thi

Re: ElasticByteBufferPool is ever growing thus can cause memory leak.

2022-01-12 Thread Steve Loughran
DirectBufferPool does the weak refs effectively, but it only recycles buffers of the same size; the elastic one will return the first largest buffer. I think we may want to have a variant of the ElasticBufferPool which uses weak refs..we can adopt that and leave the (old) one alone. It is marked a

Re: ElasticByteBufferPool is ever growing thus can cause memory leak.

2022-01-12 Thread Chris Nauroth
Thanks for discussing this, Mukund. Another difference between the 2 classes is that ElasticByteBufferPool supports a caller's preference for either on-heap or off-heap buffers and internally calls either allocate or allocateDirect. Do you intend to preserve that functionality too in a single merg