Oof this is a tough one. I agree with both of you on the concerns. I don’t have good answers but I do see a few options:
- upgrade to 1.10.0 (optionally and w rigorous testing ofc) and then start to think of a deprecation plan for LZ4 in project. That would be unfortunate but we could probably do some testing around Zstd config to get comparable-ish performance. I rely heavily on LZ4 currently so this is less ideal but maybe necessary? - since the project is Apache licensed we could fork / adopt for our needs? I’m not familiar enough with the hurdles we’d have to surmount around the licensing but at least it’s a compatible license. Maybe Mick knows more about what would be required for this route? - see if someone from the community wants to pick up ownership of lz4-Java. This would be the simplest if it works. The project would have a new owner / steward. - write our own compatible JNI wrapper around the lz4 project instead of using LZ4 Java. This would be the most work but give us ownership of the frameing and compatibility issues going forward. Jordan On Fri, Aug 2, 2024 at 04:48 Štefan Miklošovič <smikloso...@apache.org> wrote: > Indeed, I also want to add that I tried to checkout 1.10.0 submodule of > lz4 in lz4-java git repository and I built it and all tests of lz4-java > just passed fine. That is nice to know but that does not change the fact > that there is nobody behind it to drive the releases, do the bug fixes and > similar. My visibility into what it is doing is zero and that project > misses a person to take care of it in a broader way. > > On Fri, Aug 2, 2024 at 1:15 PM Brandon Williams <dri...@gmail.com> wrote: > >> I just want to note that lz4 is used for network compression, either >> between nodes or more importantly for clients, so interoperability is >> key and we need to be careful about changing things here. >> >> Kind Regards, >> Brandon >> >> On Fri, Aug 2, 2024 at 3:05 AM Štefan Miklošovič <smikloso...@apache.org> >> wrote: >> > >> > I just want to raise awareness about lz4-java library we use for LZ4 >> compressor. We are using the version 1.8.0, there is already version 1.10.0 >> of the underlying lz4 project which lz4-java integrates. >> > >> > We can see from NEWS (1) that after 1.8.0 there are a lot of >> performance improvements and in 1.10.0 they implemented multithreading >> compression (2) which provides great compression times. This seems to be >> supported for "cli" only, I am not completely sure yet if multithreaded >> compression would be something which we might use for our case too though. >> > >> > Anyway, the thing is lz4-java seems to be a dead project. There is no >> official release from 1.8.0, the submodule of lz4 was updated to 1.9.3 but >> it was never released and the main contributor behind lz4-java is >> unresponsive. (I even wrote him an email with no response yet). There are >> numerous requests from users to release a new version, questions about the >> state of the project but the response is none. It truly seems like the >> project was abandoned. >> > >> > That is quite unfortunate and I am not sure what we should do about >> that. I think that people will eventually fork so the project might >> continue in some fashion. Nevertheless, I think the state of that library >> is in a quite bad position. >> > >> > We might look into alternatives but I do not think that switching it to >> anything else will be easy because these custom libraries are often not >> compatible between themselves as the way they implement the specification >> might slightly vary. >> > >> > On lz4.org (1), there is Apache Commons listed as a lib which >> implements block and frame specification and it is interoperable so maybe >> we might take a look into how we would replace it if lz4-java is indeed >> abandoned? >> > >> > There is a note about lz4-java (or any lib in that category) like this: >> > >> > They use the block compression format, but add their own frame / header >> logic (or none at all) Consequently, they are not interoperable with LZ4 >> command line utility, nor (generally) between themselves. >> > >> > Do we even want to make any progress in this area or are we happy to >> have it on 1.8.0 forever? >> > >> > (1) https://github.com/lz4/lz4/blob/dev/NEWS >> > (2) https://github.com/lz4/lz4/blob/dev/NEWS#L2 >> > (3) https://lz4.org/ >> >
