I just want to note that lz4 is used for network compression, either between nodes or more importantly for clients, so interoperability is key and we need to be careful about changing things here.
Kind Regards, Brandon On Fri, Aug 2, 2024 at 3:05 AM Štefan Miklošovič <smikloso...@apache.org> wrote: > > I just want to raise awareness about lz4-java library we use for LZ4 > compressor. We are using the version 1.8.0, there is already version 1.10.0 > of the underlying lz4 project which lz4-java integrates. > > We can see from NEWS (1) that after 1.8.0 there are a lot of performance > improvements and in 1.10.0 they implemented multithreading compression (2) > which provides great compression times. This seems to be supported for "cli" > only, I am not completely sure yet if multithreaded compression would be > something which we might use for our case too though. > > Anyway, the thing is lz4-java seems to be a dead project. There is no > official release from 1.8.0, the submodule of lz4 was updated to 1.9.3 but it > was never released and the main contributor behind lz4-java is unresponsive. > (I even wrote him an email with no response yet). There are numerous requests > from users to release a new version, questions about the state of the project > but the response is none. It truly seems like the project was abandoned. > > That is quite unfortunate and I am not sure what we should do about that. I > think that people will eventually fork so the project might continue in some > fashion. Nevertheless, I think the state of that library is in a quite bad > position. > > We might look into alternatives but I do not think that switching it to > anything else will be easy because these custom libraries are often not > compatible between themselves as the way they implement the specification > might slightly vary. > > On lz4.org (1), there is Apache Commons listed as a lib which implements > block and frame specification and it is interoperable so maybe we might take > a look into how we would replace it if lz4-java is indeed abandoned? > > There is a note about lz4-java (or any lib in that category) like this: > > They use the block compression format, but add their own frame / header logic > (or none at all) Consequently, they are not interoperable with LZ4 command > line utility, nor (generally) between themselves. > > Do we even want to make any progress in this area or are we happy to have it > on 1.8.0 forever? > > (1) https://github.com/lz4/lz4/blob/dev/NEWS > (2) https://github.com/lz4/lz4/blob/dev/NEWS#L2 > (3) https://lz4.org/
