Bryan Beaudreault created HADOOP-18593:
------------------------------------------
Summary: Reduce byte array allocation in DecompressorStreams
Key: HADOOP-18593
URL: https://issues.apache.org/jira/browse/HADOOP-18593
Project: Hadoop Common
Issue Type: Improvement
Reporter: Bryan Beaudreault
Compression codecs all implement CompressionCodec interface, which has a
createInputStream method which takes an InputStream. These end up creating a
class which extends DecompressorStream, which allocates 2 byte arrays of
configurable size.
Currently its impossible to re-use these DecompressorStreams. Heavy users of
decompression, can end up seeing large amounts of allocations coming from the
creation of DecompressorStreams. Realtime systems like HBase have put a lot of
work into reducing allocations, either through pooling and/or off-heaping.
It would be very useful to be able to either re-use DecompressorStreams or add
constructors which accept a ByteArrayManager or other source of byte array
allocator. Either way could reduce allocations considerably in this case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
