Hi Vladislav Pyatkov, > What means the thesis? Your dataset is fit for columnar compression, ea. > repeating values and/or a timeseries-like Columnar compression relies on repeating values. If every row has a different value for a particular column, columnar compression does not give an advantage for that column.
> As I undersent, you are overwrite file io factory (TBLigniteFileIoFactory). > But in this lavel available only bytes, not data entries. We deserialize the Ignite pages ourselves to get the data entries. > What caused the performance drop to 31% in your test? That testcase has a 4 cpu node with 4 clients continuously running queries. If a node is cpu starved, compression can not give a performance advantage and it is better to read directly from disk instead. Pascal -----Original Message----- From: Vladislav Pyatkov <vldpyat...@gmail.com> Sent: Monday, 1 April 2019 10:49 To: dev@ignite.apache.org Subject: Re: in-memory compression Hi, I looked you report in site and have some questions. What means the thesis? *Your dataset is fit for columnar compression, ea. repeating values and/or a timeseries-like* *dataset.* As I undersent, you are overwrite file io factory (TBLigniteFileIoFactory). But in this lavel available only bytes, not data entries. What caused the performance drop to 31% in your test? On Mon, Apr 1, 2019 at 8:54 AM <999.comput...@gmail.com> wrote: > Hi developers, > > We have released TBLignite compression, an Ignite plugin that provides > in-memory compression: http://tblcore.com/download/. > Compression rates are similar to those of SQL Server 2016 columnar > compression (10-20x) and our testing show significant performance > improvements for datasets that are larger than the available amount of > memory. > > Currently we are at version 0.1, but we are working on a thoroughly > tested > 1.0 release so everyone can scratch their compression itch. A couple > of questions regarding this: > 1) Are there any Ignite regression tests that you can recommend for > 3rd party software? Basically we are looking for a way to test all the > possible page formats that we need to support. > 2) Once we release version 1.0, we would like TBLignite to be added to > the Ignite "3rd party binary" page. Who decides what gets on this page? > 3) And are there any acceptance criteria or tests that need to be passed? > > We would like to hear from you. > Pascal Schuchhard > TBLcore > > -- Vladislav Pyatkov
