Also note that zstd's default compression level is 3 per default (from a 1 to 19 scale). I did not test other compression levels.
On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier < ecordonn...@snapchat.com> wrote: > I ran a build of core-image-full-cmdline using xz and zstd, using > pre-populated downloads and sstate-cache directories but with empty tmp > directory. Here are the numbers: > zstd: > bitbake core-image-full-cmdline took 2m52.768s (real), the resulting > directory tmp/deploy/ipk is 1.6GB big. > xz: > bitbake core-image-full-cmdline took 4m14.214s (real), the resulting > directory tmp/deploy/ipk/ is 996M big > > So the build with xz is 47% slower (254/172) than with zstd for this > "incremental build" use-case. > > See the 5 biggest packages, the difference in compression-ratio increases > with big files: > build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5 > total 1.6G > -rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39 > gcc-dbg_12.2.0-r0_core2-64.ipk > -rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39 > openssl-dbg_3.0.5-r0_core2-64.ipk > -rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39 > openssl-ptest_3.0.5-r0_core2-64.ipk > -rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39 > binutils-dbg_2.39-r0_core2-64.ipk > build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5 > total 963M > -rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44 > gcc-dbg_12.2.0-r0_core2-64.ipk > -rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44 > openssl-dbg_3.0.5-r0_core2-64.ipk > -rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44 > openssl-ptest_3.0.5-r0_core2-64.ipk > -rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44 > binutils-dbg_2.39-r0_core2-64.ipk > ... > > I think for use-cases where the size of the ipk packages matters, xz is > better. For use-cases where it does not matter (ipk packages not deployed), > build-time matters more than compression-ratio and zstd is better. > > Regarding the enablement of zstd in opkg per default, I'll send a new > version of the patch without this line. > My thinking for enabling zstd per default in opkg was that zstd is already > enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in > opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses > libarchive for doing the actual compression/decompression). > > On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart <alex.stew...@ni.com> wrote: > >> >> >> On 9/13/22 15:20, Alex Feinman wrote: >> > I do have some numbers. When I was selling this change internally, I >> > did a comparison on our internal build. >> > Combined write IPK times (Σ t do_package_write_ipk) >> > xz 162m 35s >> > gz 52m 13s >> > zstd 33m 49s >> > Compression rate for zstd was closer to xz than to gz but not as good >> > as xz. For systems that have to cache packages on the device with >> > limited storage xz might be a better option, but for the bulk of >> > projects zstd is the best choice >> > Additionally, zstd offers much faster decompression than xz so the >> > rootfs build step that includes unpacking all of the ipks, takes 3m >> > 58s with xz and 2m 44s with zstd. >> > One other thing of note - if your build includes debug packages, some >> > may be quite large. E.g. one of our components produces a 2.2 GB debug >> > package (uncompressed). On large files xz requires a disproportionally >> > large amount of time resulting in 15 minutes needed to simply write >> > ipk for the abovementioned packages, whereas zstd took about 45 sec. >> > For frequent tasks like bitbaking a single package this translates in >> > a lot of saved time. >> >> Those are certainly compelling performance improvements. Assuming that >> the final data-segment size is within 5%-ish of xz, then I would agree >> with the rest of the thread that it should probably be the contemporary >> default. >> >> And if we make it the default compressor for OE IPKs, then obviously my >> criticism in the original PR is satisfied. >> >> > Bottom line - I think making xz a default package compressor was not >> > entirely thought through. gzip or zstd is what the default should be. >> >> ZStandard support was only added to opkg last September [1]. Before >> that, xz was the new hotness that replaced gzip. :) >> >> [1] >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e= >> >> >> > One final note: I could not find a reasonable explanation for why >> > opkg-tools require code changes to support a different compressor. BSD >> > tar and GNU tar both can easily accept compressors that they have no >> > idea about (via -I option) because all of them provide a unified >> > command line interface for use in pipes. If this were done similar to >> > tar, we could have used any compressor we wanted, including the >> > multithreaded versions (zstdmt) >> >> Well, presumably IPK creation tools can only support the matrix of >> compression algorithms which your opkg binary can decompress. I suppose >> someone could try to implement a plugable compression module system for >> opkg. But given that nearly everyone uses opkg in an embedded context, >> I'm not sure it would get much use. >> >> > >> > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <raj.k...@gmail.com> wrote: >> > >> > On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart >> > <alex.stew...@ni.com> wrote: >> > > >> > > ACK from me - apart from enabling zstd by default. >> > > >> > > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org >> > < >> https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$ >> > >> > wrote: >> > > > This allows the use of zstd for opkg packages by using >> > OPKGBUILDCMD: >> > > > OPKGBUILDCMD = "opkg-build -Z zstd" >> > > > >> > > > Signed-off-by: Alex Feinman <afein...@snap.com> >> > > > Signed-off-by: Etienne Cordonnier <ecordonn...@snap.com> >> > > > --- >> > > > meta/recipes-devtools/opkg/opkg_0.6.0.bb >> > < >> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$ >> > >> > | 3 ++- >> > > > 1 file changed, 2 insertions(+), 1 deletion(-) >> > > > >> > > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb >> > < >> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$ >> > >> > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb >> > < >> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$ >> > >> > > > index 7b351e8123..e38d9d6f3f 100644 >> > > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb >> > < >> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$ >> > >> > > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb >> > < >> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$ >> > >> > > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest >> > > > target_localstatedir := "${localstatedir}" >> > > > OPKGLIBDIR ??= "${target_localstatedir}/lib" >> > > > >> > > > -PACKAGECONFIG ??= "libsolv" >> > > > +PACKAGECONFIG ??= "libsolv zstd" >> > > >> > > Building in zstd support by default is a little suspect to me. >> > > >> > > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by >> > > default. So zstd support would be unnecessary for a distro >> > integrator >> > > who just uses upstream OE-core. >> > > >> > > For distros which use zstd compression in their packages, I think >> it >> > > would be more appropriate to overwrite the opkg PACKAGECONFIG in a >> > > .bbappend. >> > > >> > >> > This is perhaps fine. I do wonder if there is some performance >> > comparison data between xz and zstd compressed ipks >> > with opkg, it might help users on making this choice and also if we >> > should consider using >> > zstd by default at some point or not. >> > >> > > Is there something I'm not considering here? >> > > >> > > > >> > > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\ >> > > > gnupg gpgme libgpg-error,\ >> > > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = >> > "--enable-gpg,--disable-gpg,\ >> > > > PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl" >> > > > PACKAGECONFIG[ssl-curl] = >> > "--enable-ssl-curl,--disable-ssl-curl,curl openssl" >> > > > PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256" >> > > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd" >> > > > PACKAGECONFIG[libsolv] = >> > "--with-libsolv,--without-libsolv,libsolv" >> > > > >> > > > EXTRA_OECONF:class-native = >> > "--localstatedir=/${@os.path.relpath('${localstatedir}', >> > '${STAGING_DIR_NATIVE}')} >> > --sysconfdir=/${@os.path.relpath('${sysconfdir}', >> > '${STAGING_DIR_NATIVE}')}" >> > > > >> > > > >> > > > >> > > >> > > -- >> > > Alex Stewart >> > > Software Engineer - NI Real-Time OS >> > > NI (National Instruments) >> > > >> > > alex.stew...@ni.com >> > > >> > > >> > > >> > > >> > >> >> -- >> Alex Stewart >> Software Engineer - NI Real-Time OS >> NI (National Instruments) >> >> alex.stew...@ni.com >> >>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#170653): https://lists.openembedded.org/g/openembedded-core/message/170653 Mute This Topic: https://lists.openembedded.org/mt/93654146/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
