If you look at
https://github.com/lz4/lz4-java/graphs/contributors?from=2019-12-28&to=2021-03-17&type=c,
lz4-java seems to be receiving very little maintenance. So I think
there are two possible avenues:
1) contribute the missing support ourselves
2) use another LZ4 library for Java
Solution #2 seems more reasonable to me.
Regards
Antoine.
Le 11/03/2021 à 21:05, Micah Kornfield a écrit :
FYI, I opened up https://github.com/lz4/lz4-java/issues/176 to discuss
support for dependent frames.
On Thu, Mar 11, 2021 at 11:59 AM David Li <lidav...@apache.org> wrote:
At least for Flight, I don't think we'd use that. Right now the way
compression is supported is the same way as with Feather, i.e. the body
buffers in each individual record batch sent on the wire are compressed,
but not the stream as a whole. (And so far we haven't found a compelling
benefit for compression in Flight in general.)
Best,
David
On Thu, Mar 11, 2021, at 14:34, Antoine Pitrou wrote:
Le 11/03/2021 à 19:54, Micah Kornfield a écrit :
Indeed, I don't think it was discussed publicly. The LZ4 frame format
has several things going for it:
- it allows streaming compression and decompression (meaning you can
avoid loading a huge compressed buffer at once)
Is this something we make use of or intend to make use of?
Good question. Currently we don't. Perhaps David Li wants to answer
this, since he's been working a lot on Flight.
- it embeds the decompressed size, allowing exact allocation of the
decompressed buffer
IIUC, We already do this in the IPC specification (the first 8 bytes
of the
compressed buffer are used for this).
Ah, you're right. It doesn't matter then.
- it has an optional checksum
This seems like a good thing, so probably worth keeping (although it
would
be the only place where we do checksums today).
(or of course we could add an optional higher-level checksum in the IPC
format)
Regards
Antoine.
