Indeed, I also want to add that I tried to checkout 1.10.0 submodule of lz4 in lz4-java git repository and I built it and all tests of lz4-java just passed fine. That is nice to know but that does not change the fact that there is nobody behind it to drive the releases, do the bug fixes and similar. My visibility into what it is doing is zero and that project misses a person to take care of it in a broader way.
On Fri, Aug 2, 2024 at 1:15 PM Brandon Williams <dri...@gmail.com> wrote: > I just want to note that lz4 is used for network compression, either > between nodes or more importantly for clients, so interoperability is > key and we need to be careful about changing things here. > > Kind Regards, > Brandon > > On Fri, Aug 2, 2024 at 3:05 AM Štefan Miklošovič <smikloso...@apache.org> > wrote: > > > > I just want to raise awareness about lz4-java library we use for LZ4 > compressor. We are using the version 1.8.0, there is already version 1.10.0 > of the underlying lz4 project which lz4-java integrates. > > > > We can see from NEWS (1) that after 1.8.0 there are a lot of performance > improvements and in 1.10.0 they implemented multithreading compression (2) > which provides great compression times. This seems to be supported for > "cli" only, I am not completely sure yet if multithreaded compression would > be something which we might use for our case too though. > > > > Anyway, the thing is lz4-java seems to be a dead project. There is no > official release from 1.8.0, the submodule of lz4 was updated to 1.9.3 but > it was never released and the main contributor behind lz4-java is > unresponsive. (I even wrote him an email with no response yet). There are > numerous requests from users to release a new version, questions about the > state of the project but the response is none. It truly seems like the > project was abandoned. > > > > That is quite unfortunate and I am not sure what we should do about > that. I think that people will eventually fork so the project might > continue in some fashion. Nevertheless, I think the state of that library > is in a quite bad position. > > > > We might look into alternatives but I do not think that switching it to > anything else will be easy because these custom libraries are often not > compatible between themselves as the way they implement the specification > might slightly vary. > > > > On lz4.org (1), there is Apache Commons listed as a lib which > implements block and frame specification and it is interoperable so maybe > we might take a look into how we would replace it if lz4-java is indeed > abandoned? > > > > There is a note about lz4-java (or any lib in that category) like this: > > > > They use the block compression format, but add their own frame / header > logic (or none at all) Consequently, they are not interoperable with LZ4 > command line utility, nor (generally) between themselves. > > > > Do we even want to make any progress in this area or are we happy to > have it on 1.8.0 forever? > > > > (1) https://github.com/lz4/lz4/blob/dev/NEWS > > (2) https://github.com/lz4/lz4/blob/dev/NEWS#L2 > > (3) https://lz4.org/ >
