On Mon, Mar 25, 2024 at 04:50:28PM -0400, Neal Gompa wrote: > On Mon, Mar 25, 2024 at 4:40 PM Zbigniew Jędrzejewski-Szmek > <zbys...@in.waw.pl> wrote: > > > > On Mon, Mar 25, 2024 at 07:29:09PM +0100, Kevin Kofler via devel wrote: > > > Daniel Alley wrote: > > > > One more point: createrepo_c uses zstd compression level 10, but the > > > > range > > > > goes all the way up to level 22. I would oppose making the default much > > > > computationally heavier than it is currently, but if spending 20x longer > > > > to compress the repo 10% more is desirable to the fedora project, then > > > > createrepo_c could perhaps add a the ability to select a compression > > > > level. > > > > > > > > zstd at high compression levels is very nearly as good at compressing as > > > > xz and sometimes better, while remaining much faster to decompress. -- > > > > > > Considering that compression happens once on the server and downloading > > > and > > > decompression happens many times on many computers, I think we should use > > > the highest possible compression level. > > > > +1 > > > > Keep in mind we also want to make the compose process faster too, I > don't know if it's worth it to spend 20x more time compressing > repodata when we keep trying to get back hours and minutes in the > compose time.
I wanted to write that the compression times are small enough for this not not matter, but indeed, at the very highest levels, they do become noticable. $ mv 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml filelists.xml $ time zstd -k -9 filelists.xml filelists.xml : 5.38% ( 863 MiB => 46.4 MiB, filelists.xml.zst) zstd -k -9 4.96s user 0.18s system 103% cpu 4.971 total $ time zstd -k -21 filelists.xml Warning : compression level higher than max, reduced to 19 filelists.xml : 4.74% ( 863 MiB => 40.9 MiB, filelists.xml.zst) zstd -k -21 321.49s user 0.31s system 99% cpu 5:22.20 total $ time zstd -k -21 -T8 filelists.xml Warning : compression level higher than max, reduced to 19 filelists.xml : 4.74% ( 863 MiB => 40.9 MiB, filelists.xml.zst) zstd -k -21 -T8 874.57s user 0.70s system 732% cpu 1:59.51 total $ time xz -k -v 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml (1/1) 100 % 44.3 MiB / 862.9 MiB = 0.051 33 MiB/s 0:26 xz -k -v 196.88s user 0.63s system 749% cpu 26.337 total (This is multithreaded, and gives a compression ratio of 5.14%.) Dunno, I think anything below a minute should be OK… Zbyszek -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
