Kanak -- notice you can trade space for time using the various settings of zstandard. Compression is data dependent, but if you can afford the time, zstandard at 11 will do its hardest to make the compressed data as small as possible. It will take alot of time, but depending on your application, it may be worth it.
On Tuesday, June 3, 2025 at 12:12:30 AM UTC+1 Jason E. Aten wrote: > Ah, I believe Klaus Post wrote S2 as a faster version of Snappy. I believe > it is his own algorithm. He > aimed for very fast execution on non-compressible data, and for snappy > compatibility in one direction. > > https://pkg.go.dev/github.com/klauspost/compress/s2 > > "S2 is designed to have high throughput on content that cannot be > compressed. This is important, so you don't have to worry about spending > CPU cycles on already compressed data." -- Klaus Post > > His compress package _also_ provides zstandard, if that is what you want. > My benching > showed that S2 has an edge; from > https://github.com/glycerine/rpc25519?tab=readme-ov-file#some-quick-compression-benchmarks-2025-january-11 > > These benchmarks include network overhead time, but that should have been > very similar between each run. > > > > > > > > > *compressor bandwidth total elapsed time for one-way > transfer---------- ------------ > --------------------------------------- s2 163.412400 MB/sec; total > time for upload: 9.2 seconds lz4 157.343690 MB/sec; total time for > upload: 9.6 secondszstd:01 142.088387 MB/sec; total time for upload: > 10.6 secondszstd:03 130.871089 MB/sec; total time for upload: 11.6 > secondszstd:07 121.209097 MB/sec; total time for upload: 12.5 > secondszstd:11 27.766271 MB/sec; total time for upload: 54.4 seconds* > > > On Monday, June 2, 2025 at 11:37:05 PM UTC+1 Robert Engels wrote: > >> Ah thanks, but then in the S2 docs it states to use zstandard for better >> compression https://github.com/klauspost/compress/tree/master/zstd#zstd >> >> On Jun 2, 2025, at 5:30 PM, Robert Snedegar <vik...@gmail.com> wrote: >> >> >> >> >> >> On Mon, Jun 2, 2025 at 1:52 PM Robert Engels <ren...@ix.netcom.com> >> wrote: >> >>> Have you looked at the S2 Geometry library? >>> http://s2geometry.io/about/overview.html >>> >>> >> That's a different S2 than Kanak is talking about. I think it is the S2 >> implementation in https://github.com/klauspost/compress. >> >> >> On Jun 2, 2025, at 3:33 PM, Kanak Bhatia <bhatia...@gmail.com> wrote: >>> >>> Hi Everyone, >>> >>> I’m currently working with MinIO and diving into its use of s2 >>> compression for optimizing object storage performance. One area I’m curious >>> about is the S2 compression index and whether there are feasible techniques >>> to compact or optimize it further. >>> >>> I’m trying to determine if compacting the index is even possible—and if >>> so, what approaches could be taken. Some potential directions I’m exploring >>> or considering include: >>> >>> Reducing or collapsing redundant index entries >>> >>> Adjusting block boundaries or batch sizes to impact index structure >>> >>> >>> If anyone has experience with this or has experimented with compression >>> index optimizations in S2 (inside or outside of MinIO), I’d really >>> appreciate your input—especially around what’s practical, what’s not, and >>> any relevant trade-offs. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/golang-nuts/c7ef55d8-79ca-426c-835a-5c6b8ec40bf9n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/c7ef55d8-79ca-426c-835a-5c6b8ec40bf9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/golang-nuts/DCFA8695-E5EC-4DAD-BB81-52E8DBA628BE%40ix.netcom.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/DCFA8695-E5EC-4DAD-BB81-52E8DBA628BE%40ix.netcom.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/8d633e8f-e425-40f1-9028-479c2e71ad04n%40googlegroups.com.
