SAP Hana does the compression by 1) compressing SQL parameters before execution, and 2) storing only compressed data in memory. This way all SQL queries work as normal with zero modifications or performance overhead. Only results of the query can be (optionally) decompressed back before returning to the user.
-- Nikita Ivanov On Mon, Jul 25, 2016 at 1:40 PM, Sergey Kozlov <skoz...@gridgain.com> wrote: > Hi > > For approach 1: Put a large object into a partiton cache will force to > update the dictionary placed on replication cache. It seeis it may be > time-expense operation. > Appoach 2-3 are make sense for rare cases as Sergi commented. > Aslo I see a danger of OOM if we've got high compression level and try to > restore orginal value in memory. > > On Mon, Jul 25, 2016 at 10:39 AM, Alexey Kuznetsov < > akuznet...@gridgain.com> > wrote: > > > Sergi, > > > > Of course it will introduce some slowdown, but with compression more data > > could be stored in memory > > and not will be evicted to disk. In case of compress by dictionary > > substitution it will be only one more lookup > > and should be fast. > > > > In general we could provide only API for compression out of the box, and > > users that really need some sort of compression > > will implement it by them self. This will not require much effort I > think. > > > > > > > > On Mon, Jul 25, 2016 at 2:18 PM, Sergi Vladykin < > sergi.vlady...@gmail.com> > > wrote: > > > > > This will make sense only for rare cases when you have very large > objects > > > stored, which can be effectively compressed. And even then it will > > > introduce slowdown on all the operations, which often will not be > > > acceptable. I guess only few users will find this feature useful, thus > I > > > think it does not worth the effort. > > > > > > Sergi > > > > > > > > > > > > 2016-07-25 9:28 GMT+03:00 Alexey Kuznetsov <akuznet...@gridgain.com>: > > > > > > > Hi, All! > > > > > > > > I would like to propose one more feature for Ignite 2.0. > > > > > > > > Data compression for data in binary format. > > > > > > > > Binary format is stored as field name + field data. > > > > So we have a description. > > > > How about to add one more byte to binary data descriptor: > > > > > > > > *Compressed*: > > > > 0 - Data stored as is (no compression). > > > > 1 - Data compressed by dictionary (something like DB2 row > compression > > > [1], > > > > but for all binary types). We could have system or user defined > > > replicated > > > > cache for such dictionary and *cache.compact()* method that will scan > > > > cache, build dictionary and compact data. > > > > 2 - Data compressed by Java built in ZIP. > > > > 3 - Data compressed by some user custom algorithm. > > > > > > > > Of course it is possible to compress data in current Ignite 1.x but > in > > > this > > > > case compressed data cannot be accessed from SQL engine, if we > > implement > > > > support for compression on Ignite core level SQL engine will be able > to > > > > detect that data is compressed and properly handle such data. > > > > > > > > What do you think? > > > > If community consider this feature useful I will create issue in > JIRA. > > > > > > > > [1] > > > > > > > > > > > > > > http://www.ibm.com/developerworks/data/library/techarticle/dm-1205db210compression/ > > > > > > > > -- > > > > Alexey Kuznetsov > > > > > > > > > > > > > > > -- > > Alexey Kuznetsov > > > > > > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com >
