Hi Ilya, Thank you for sharing this here. I believe this contribution will be accepted by the Community. Moreover, it shows so remarkable performance boost.
I'm pretty sure this patch will be reviewed by Ignite Native Persistence experts soon. What do you mean by can't survive PDS node restart? Sincerely, Dmitriy Pavlov чт, 23 авг. 2018 г. в 13:09, Ilya Kasnacheev <ilya.kasnach...@gmail.com>: > Hello! > > I want to share with the developer community my compression prototype. > > Long story short, it compresses BinaryObject's byte[] as they are written > to Durable Memory page, operating on a pre-built dictionary. Typical > compression ratio is 0.4 (meaning 2.5x compression) using custom > LZW+Huffman. Metadata, indexes and primitive values are unaffected > entirely. > > This is akin to DB2's table-level compression[1] but independently > invented. > > On Yardstick tests performance hit is -6% with PDS and up to -25% (in > throughput) with In-Memory loads. It also means you can fit ~twice as much > data into the same IM cluster, or have higher ram/disk ratio with PDS > cluster, saving on hardware or decreasing latency. > > The code is available as PR 4295[2] (set IGNITE_ENABLE_COMPRESSION=true to > activate). Note that it will not presently survive a PDS node restart. > The impact is very small, the patch should be applicable to most 2.x > releases. > > Sure there's a long way before this prototype can have hope of being > included, but first I would like to hear input from fellow igniters. > > See also IEP-20[3]. > > 1. > > https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10.5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052331.html > 2. https://github.com/apache/ignite/pull/4295 > 3. > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite > > Regards, > > -- > Ilya Kasnacheev >
