> > Indeed, I don't think it was discussed publicly. The LZ4 frame format > has several things going for it: > - it allows streaming compression and decompression (meaning you can > avoid loading a huge compressed buffer at once)
Is this something we make use of or intend to make use of? > - it embeds the decompressed size, allowing exact allocation of the > decompressed buffer IIUC, We already do this in the IPC specification (the first 8 bytes of the compressed buffer are used for this). - it has an optional checksum This seems like a good thing, so probably worth keeping (although it would be the only place where we do checksums today). Is https://github.com/lz4/lz4-java the fast Java lz4 library in > question? Yes it is. On Thu, Mar 11, 2021 at 10:12 AM Joris Peeters <joris.mg.peet...@gmail.com> wrote: > "Is https://github.com/lz4/lz4-java the fast Java lz4 library in > question? The incompleteness of this implementation is a known problem > for other user communities, not only Arrow. It would be a great public > service to improve it so that it fully implements the lz4 frame > specification." > > Very much +1. > > On Thu, Mar 11, 2021 at 6:08 PM Steve Kim <chairm...@gmail.com> wrote: > > > I prefer the lz4 frame format for the reasons that Antoine stated. > > > > To be friendly to users, the Arrow IPC documentation could mention > > that lz4 compression may break Java interoperability. If block > > dependency is the only obstacle to Java interoperability, the Arrow > > IPC implementation could disable this flag as an _implementation > > detail_. But I think that the _specification_ of lz4 compression in > > the Arrow IPC format should not carve out exceptions to accommodate > > current (temporary, I hope) implementation shortfalls in lz4 > > libraries. > > > > Is https://github.com/lz4/lz4-java the fast Java lz4 library in > > question? The incompleteness of this implementation is a known problem > > for other user communities, not only Arrow. It would be a great public > > service to improve it so that it fully implements the lz4 frame > > specification. > > > > Steve > > >
