Rei used to be semi-active on the Cassandra lists. He took over lz4-java a few years ago. Wonder if he’d be willing to pass the torch if he no longer has time? CC’ing him in case he’s following email.
> On Aug 6, 2024, at 8:47 AM, Jordan West <jorda...@gmail.com> wrote: > > Oof this is a tough one. I agree with both of you on the concerns. I don’t > have good answers but I do see a few options: > > - upgrade to 1.10.0 (optionally and w rigorous testing ofc) and then start to > think of a deprecation plan for LZ4 in project. That would be unfortunate but > we could probably do some testing around Zstd config to get comparable-ish > performance. I rely heavily on LZ4 currently so this is less ideal but maybe > necessary? > > - since the project is Apache licensed we could fork / adopt for our needs? > I’m not familiar enough with the hurdles we’d have to surmount around the > licensing but at least it’s a compatible license. Maybe Mick knows more about > what would be required for this route? > > - see if someone from the community wants to pick up ownership of lz4-Java. > This would be the simplest if it works. The project would have a new owner / > steward. > > - write our own compatible JNI wrapper around the lz4 project instead of > using LZ4 Java. This would be the most work but give us ownership of the > frameing and compatibility issues going forward. > > Jordan > > On Fri, Aug 2, 2024 at 04:48 Štefan Miklošovič <smikloso...@apache.org > <mailto:smikloso...@apache.org>> wrote: >> Indeed, I also want to add that I tried to checkout 1.10.0 submodule of lz4 >> in lz4-java git repository and I built it and all tests of lz4-java just >> passed fine. That is nice to know but that does not change the fact that >> there is nobody behind it to drive the releases, do the bug fixes and >> similar. My visibility into what it is doing is zero and that project misses >> a person to take care of it in a broader way. >> >> On Fri, Aug 2, 2024 at 1:15 PM Brandon Williams <dri...@gmail.com >> <mailto:dri...@gmail.com>> wrote: >>> I just want to note that lz4 is used for network compression, either >>> between nodes or more importantly for clients, so interoperability is >>> key and we need to be careful about changing things here. >>> >>> Kind Regards, >>> Brandon >>> >>> On Fri, Aug 2, 2024 at 3:05 AM Štefan Miklošovič <smikloso...@apache.org >>> <mailto:smikloso...@apache.org>> wrote: >>> > >>> > I just want to raise awareness about lz4-java library we use for LZ4 >>> > compressor. We are using the version 1.8.0, there is already version >>> > 1.10.0 of the underlying lz4 project which lz4-java integrates. >>> > >>> > We can see from NEWS (1) that after 1.8.0 there are a lot of performance >>> > improvements and in 1.10.0 they implemented multithreading compression >>> > (2) which provides great compression times. This seems to be supported >>> > for "cli" only, I am not completely sure yet if multithreaded compression >>> > would be something which we might use for our case too though. >>> > >>> > Anyway, the thing is lz4-java seems to be a dead project. There is no >>> > official release from 1.8.0, the submodule of lz4 was updated to 1.9.3 >>> > but it was never released and the main contributor behind lz4-java is >>> > unresponsive. (I even wrote him an email with no response yet). There are >>> > numerous requests from users to release a new version, questions about >>> > the state of the project but the response is none. It truly seems like >>> > the project was abandoned. >>> > >>> > That is quite unfortunate and I am not sure what we should do about that. >>> > I think that people will eventually fork so the project might continue in >>> > some fashion. Nevertheless, I think the state of that library is in a >>> > quite bad position. >>> > >>> > We might look into alternatives but I do not think that switching it to >>> > anything else will be easy because these custom libraries are often not >>> > compatible between themselves as the way they implement the specification >>> > might slightly vary. >>> > >>> > On lz4.org <http://lz4.org/> (1), there is Apache Commons listed as a lib >>> > which implements block and frame specification and it is interoperable so >>> > maybe we might take a look into how we would replace it if lz4-java is >>> > indeed abandoned? >>> > >>> > There is a note about lz4-java (or any lib in that category) like this: >>> > >>> > They use the block compression format, but add their own frame / header >>> > logic (or none at all) Consequently, they are not interoperable with LZ4 >>> > command line utility, nor (generally) between themselves. >>> > >>> > Do we even want to make any progress in this area or are we happy to have >>> > it on 1.8.0 forever? >>> > >>> > (1) https://github.com/lz4/lz4/blob/dev/NEWS >>> > (2) https://github.com/lz4/lz4/blob/dev/NEWS#L2 >>> > (3) https://lz4.org/
