Ah, I believe Klaus Post wrote S2 as a faster version of Snappy. I believe it is his own algorithm. He aimed for very fast execution on non-compressible data, and for snappy compatibility in one direction.
https://pkg.go.dev/github.com/klauspost/compress/s2 "S2 is designed to have high throughput on content that cannot be compressed. This is important, so you don't have to worry about spending CPU cycles on already compressed data." -- Klaus Post His compress package _also_ provides zstandard, if that is what you want. My benching showed that S2 has an edge; from https://github.com/glycerine/rpc25519?tab=readme-ov-file#some-quick-compression-benchmarks-2025-january-11 These benchmarks include network overhead time, but that should have been very similar between each run. *compressor bandwidth total elapsed time for one-way transfer---------- ------------ --------------------------------------- s2 163.412400 MB/sec; total time for upload: 9.2 seconds lz4 157.343690 MB/sec; total time for upload: 9.6 secondszstd:01 142.088387 MB/sec; total time for upload: 10.6 secondszstd:03 130.871089 MB/sec; total time for upload: 11.6 secondszstd:07 121.209097 MB/sec; total time for upload: 12.5 secondszstd:11 27.766271 MB/sec; total time for upload: 54.4 seconds* On Monday, June 2, 2025 at 11:37:05 PM UTC+1 Robert Engels wrote: > Ah thanks, but then in the S2 docs it states to use zstandard for better > compression https://github.com/klauspost/compress/tree/master/zstd#zstd > > On Jun 2, 2025, at 5:30 PM, Robert Snedegar <vik...@gmail.com> wrote: > > > > > > On Mon, Jun 2, 2025 at 1:52 PM Robert Engels <ren...@ix.netcom.com> wrote: > >> Have you looked at the S2 Geometry library? >> http://s2geometry.io/about/overview.html >> >> > That's a different S2 than Kanak is talking about. I think it is the S2 > implementation in https://github.com/klauspost/compress. > > > On Jun 2, 2025, at 3:33 PM, Kanak Bhatia <bhatia...@gmail.com> wrote: >> >> Hi Everyone, >> >> I’m currently working with MinIO and diving into its use of s2 >> compression for optimizing object storage performance. One area I’m curious >> about is the S2 compression index and whether there are feasible techniques >> to compact or optimize it further. >> >> I’m trying to determine if compacting the index is even possible—and if >> so, what approaches could be taken. Some potential directions I’m exploring >> or considering include: >> >> Reducing or collapsing redundant index entries >> >> Adjusting block boundaries or batch sizes to impact index structure >> >> >> If anyone has experience with this or has experimented with compression >> index optimizations in S2 (inside or outside of MinIO), I’d really >> appreciate your input—especially around what’s practical, what’s not, and >> any relevant trade-offs. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts...@googlegroups.com. >> To view this discussion visit >> https://groups.google.com/d/msgid/golang-nuts/c7ef55d8-79ca-426c-835a-5c6b8ec40bf9n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/golang-nuts/c7ef55d8-79ca-426c-835a-5c6b8ec40bf9n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts...@googlegroups.com. >> To view this discussion visit >> https://groups.google.com/d/msgid/golang-nuts/DCFA8695-E5EC-4DAD-BB81-52E8DBA628BE%40ix.netcom.com >> >> <https://groups.google.com/d/msgid/golang-nuts/DCFA8695-E5EC-4DAD-BB81-52E8DBA628BE%40ix.netcom.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/5e6b03ad-d95a-4d85-9c38-e7ab6af539ban%40googlegroups.com.
