Hello Dongjin, I'm wondering if you have looked into the different implementor's buffer usage? So far as I read from the code:
1. LZ4 used a shared 64KB for decompression, and when reading it used ByteBuffer copy from the decompression buffer. 2. Snappy used shared uncompressed buffer, but when reading it uses SnappyNative.arrayCopy JNI which could be slow. 3. GZIP used shared 8KB (inflator), and another shared 16KB for reading it out with System.arraycopy. 4. ZSTD: dynamically allocate 128KB (i.e. not shared), and also use read internal for skip, so skip does not benefit that much. It seems to me that for different types the buffer used quite differently. Also, aside from ZSTD, are there any other types that have levels? Guozhang On Mon, Jun 24, 2019 at 4:30 PM Dongjin Lee <dong...@apache.org> wrote: > Hello. Here is the new discussion thread for KIP-390: Allow fine-grained > configuration for compression. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Allow+fine-grained+configuration+for+compression > > Here is some history: Initially, the draft implementation was done with > traditional configuration scheme (Type A). Then, Becket Qin ( > becket....@gmail.com) and Mickael Maison (mickael.mai...@gmail.com) > proposed that the map style configuration > like listener.security.protocol.map > or max.connections.per.ip.overrides (Type B) would be better. From then on, > the discussion got struck. > > So last weekend, I re-implemented the feature against the latest trunk, for > all public interface alternatives (i.e., Type A & B.), and updated the KIP > document. You can find the details in this PR: > https://github.com/apache/kafka/pull/5927 > > Please have a look when you are free. All kinds of feedbacks are welcomed! > > Regards, > Dongjin > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>* > -- -- Guozhang
