GitHub user xvrl opened a pull request:
https://github.com/apache/kafka/pull/2967
KAFKA-5150 reduce lz4 decompression overhead
- reuse decompression buffers, keeping one per thread
- switch lz4 input stream to operate directly on ByteBuffers
- more tests with both compressible / incompressible data, multiple
blocks, and various other combinations to increase code coverage
- fixes bug that would cause EOFException instead of invalid block size
for invalid incompressible blocks
Overall this improves LZ4 decompression performance by up to 23x for small
batches.
Most improvements are seen for batches of size 1 with messages on the order
of ~100B.
At least 10x improvements for for batch sizes of < 10 messages, with
messages of < 10kB
See benchmark code and results here
https://gist.github.com/xvrl/05132e0643513df4adf842288be86efd
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xvrl/kafka kafka-5150
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2967.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2967
----
commit 0efc6e7f15b6994a6665da5975e69c77426cf904
Author: Xavier Léauté <[email protected]>
Date: 2017-05-03T20:40:45Z
KAFKA-5150 reduce lz4 decompression overhead
- reuse decompression buffers, keeping one per thread
- switch lz4 input stream to operate directly on ByteBuffers
- more tests with both compressible / incompressible data, multiple
blocks, and various other combinations to increase code coverage
- fixes bug that would cause EOFException instead of invalid block size
for invalid incompressible blocks
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
