Ok, thanks. IMO we need to store the dictionary in Durable memory before merging into master.
чт, 23 авг. 2018 г. в 15:12, Ilya Kasnacheev <ilya.kasnach...@gmail.com>: > Hello! > > Currently, the dictionary for decompression is only stored on heap. After > restart there's compressed data in the PDS, but there's no dictionary :) > > Regards, > > -- > Ilya Kasnacheev > > 2018-08-23 14:58 GMT+03:00 Dmitriy Pavlov <dpavlov....@gmail.com>: > > > Hi Ilya, > > > > Thank you for sharing this here. I believe this contribution will be > > accepted by the Community. Moreover, it shows so remarkable performance > > boost. > > > > I'm pretty sure this patch will be reviewed by Ignite Native Persistence > > experts soon. > > > > What do you mean by can't survive PDS node restart? > > > > Sincerely, > > Dmitriy Pavlov > > > > чт, 23 авг. 2018 г. в 13:09, Ilya Kasnacheev <ilya.kasnach...@gmail.com > >: > > > > > Hello! > > > > > > I want to share with the developer community my compression prototype. > > > > > > Long story short, it compresses BinaryObject's byte[] as they are > written > > > to Durable Memory page, operating on a pre-built dictionary. Typical > > > compression ratio is 0.4 (meaning 2.5x compression) using custom > > > LZW+Huffman. Metadata, indexes and primitive values are unaffected > > > entirely. > > > > > > This is akin to DB2's table-level compression[1] but independently > > > invented. > > > > > > On Yardstick tests performance hit is -6% with PDS and up to -25% (in > > > throughput) with In-Memory loads. It also means you can fit ~twice as > > much > > > data into the same IM cluster, or have higher ram/disk ratio with PDS > > > cluster, saving on hardware or decreasing latency. > > > > > > The code is available as PR 4295[2] (set IGNITE_ENABLE_COMPRESSION=true > > to > > > activate). Note that it will not presently survive a PDS node restart. > > > The impact is very small, the patch should be applicable to most 2.x > > > releases. > > > > > > Sure there's a long way before this prototype can have hope of being > > > included, but first I would like to hear input from fellow igniters. > > > > > > See also IEP-20[3]. > > > > > > 1. > > > > > > https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10. > > 5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052331.html > > > 2. https://github.com/apache/ignite/pull/4295 > > > 3. > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > 20%3A+Data+Compression+in+Ignite > > > > > > Regards, > > > > > > -- > > > Ilya Kasnacheev > > > > > >
