Hi Shylaja, If you'd like we can continue this on the ticket you opened. Here are my concerns -
1. QPL Java Library[1] (JNI bindings to Intel's QPL) does not have any license information on the repo. This needs to be corrected. Please see the types of licenses we can use[2] for further information. 2. Can you describe how the compressor will behave when the cluster is made up of heterogeneous hardware? For example, let's say we have a mix of machines where some support Intel's IAA and some don't? 3. Does QPL have checksumming built in? thanks, Dinesh [1] https://github.com/intel/qpl-java [2] https://www.apache.org/legal/resolved.html#category-a On Mon, Jan 22, 2024 at 6:37 PM Kokoori, Shylaja <shylaja.koko...@intel.com> wrote: > Dinesh & Abe, > > Thank you very much for your feedback. > > > > The algorithm used by this HW compressor is compatible with Deflate but > there is a constraint of 4K window size. Therefore the concern is that > existing data may not decompress correctly as is. That is why we chose the > path of adding a new compressor. > > Another reason is that, there are some additional features available in > the hardware which are not compatible with zlib. With this approach we > could enable those features as well. > > > > We are also planning to accelerate existing compressors, if that is the > preferred approach we will try to come up with a solution to work around > the 4k window limitation. > > > > Thank you, > > Shylaja > > > > *From:* Dinesh Joshi <djo...@apache.org> > *Sent:* Monday, January 22, 2024 11:18 AM > *To:* dev@cassandra.apache.org > *Subject:* Re: CASSANDRA-19268: Improve Cassandra compression performance > using hardware accelerators > > > > Shylaja, > > > > Cassandra uses ZStd, LZ4 and other compression libraries via JNI to > compress data. The intel hardware accelerator support is integrated into > those libraries and we can benefit from it. If there are special parameters > that need to be passed in to these libraries we can make those changes on > the database but as such Cassandra does not directly implement the > compression algorithms itself. > > > > Dinesh >
