On Wed, Sep 14, 2022 at 8:37 AM Alex Stewart <alex.stew...@ni.com> wrote: > > Thanks for checking. > > I'd be interested to know if setting a higher compression level for zstd > can get us to a similar compression ratio to xz. If so, then I think it > could be some real value to distro maintainers to be able to *tune* > their compression.
yeah it will be interesting to say try something like level 9 but I think times might regress with that but it might be good to know the balance and perhaps suggest size mode and performance mode of zstd instead of xz > > That's not blocking for your new PR though. > > > On 9/14/22 05:08, Etienne Cordonnier wrote: > > Also note that zstd's default compression level is 3 per default (from > > a 1 to 19 scale). I did not test other compression levels. > > > > On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier > > <ecordonn...@snapchat.com> wrote: > > > > I ran a build of core-image-full-cmdline using xz and zstd, using > > pre-populated downloads and sstate-cache directories but with > > empty tmp directory. Here are the numbers: > > zstd: > > bitbake core-image-full-cmdline took 2m52.768s (real), the > > resulting directory tmp/deploy/ipk is 1.6GB big. > > xz: > > bitbake core-image-full-cmdline took 4m14.214s (real), the > > resulting directory tmp/deploy/ipk/ is 996M big > > > > So the build with xz is 47% slower (254/172) than with zstd for > > this "incremental build" use-case. > > > > See the 5 biggest packages, the difference in compression-ratio > > increases with big files: > > build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5 > > total 1.6G > > -rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39 > > gcc-dbg_12.2.0-r0_core2-64.ipk > > -rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39 > > openssl-dbg_3.0.5-r0_core2-64.ipk > > -rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39 > > openssl-ptest_3.0.5-r0_core2-64.ipk > > -rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39 > > binutils-dbg_2.39-r0_core2-64.ipk > > build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5 > > total 963M > > -rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44 > > gcc-dbg_12.2.0-r0_core2-64.ipk > > -rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44 > > openssl-dbg_3.0.5-r0_core2-64.ipk > > -rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44 > > openssl-ptest_3.0.5-r0_core2-64.ipk > > -rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44 > > binutils-dbg_2.39-r0_core2-64.ipk > > ... > > > > I think for use-cases where the size of the ipk packages matters, > > xz is better. For use-cases where it does not matter (ipk packages > > not deployed), build-time matters more than compression-ratio and > > zstd is better. > > > > Regarding the enablement of zstd in opkg per default, I'll send a > > new version of the patch without this line. > > My thinking for enabling zstd per default in opkg was that zstd is > > already enabled per default in libarchive's PACKAGECONFIG, and > > disabling zstd in opkg's PACKAGECONFIG removes only a few lines of > > code from opkg (opkg uses libarchive for doing the actual > > compression/decompression). > > > > On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart > > <alex.stew...@ni.com> wrote: > > > > > > > > On 9/13/22 15:20, Alex Feinman wrote: > > > I do have some numbers. When I was selling this change > > internally, I > > > did a comparison on our internal build. > > > Combined write IPK times (Σ t do_package_write_ipk) > > > xz 162m 35s > > > gz 52m 13s > > > zstd 33m 49s > > > Compression rate for zstd was closer to xz than to gz but > > not as good > > > as xz. For systems that have to cache packages on the device > > with > > > limited storage xz might be a better option, but for the > > bulk of > > > projects zstd is the best choice > > > Additionally, zstd offers much faster decompression than xz > > so the > > > rootfs build step that includes unpacking all of the ipks, > > takes 3m > > > 58s with xz and 2m 44s with zstd. > > > One other thing of note - if your build includes debug > > packages, some > > > may be quite large. E.g. one of our components produces a > > 2.2 GB debug > > > package (uncompressed). On large files xz requires a > > disproportionally > > > large amount of time resulting in 15 minutes needed to > > simply write > > > ipk for the abovementioned packages, whereas zstd took about > > 45 sec. > > > For frequent tasks like bitbaking a single package this > > translates in > > > a lot of saved time. > > > > Those are certainly compelling performance improvements. > > Assuming that > > the final data-segment size is within 5%-ish of xz, then I > > would agree > > with the rest of the thread that it should probably be the > > contemporary > > default. > > > > And if we make it the default compressor for OE IPKs, then > > obviously my > > criticism in the original PR is satisfied. > > > > > Bottom line - I think making xz a default package compressor > > was not > > > entirely thought through. gzip or zstd is what the default > > should be. > > > > ZStandard support was only added to opkg last September [1]. > > Before > > that, xz was the new hotness that replaced gzip. :) > > > > [1] > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e= > > > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=> > > > > > > > One final note: I could not find a reasonable explanation > > for why > > > opkg-tools require code changes to support a different > > compressor. BSD > > > tar and GNU tar both can easily accept compressors that they > > have no > > > idea about (via -I option) because all of them provide a > > unified > > > command line interface for use in pipes. If this were done > > similar to > > > tar, we could have used any compressor we wanted, including the > > > multithreaded versions (zstdmt) > > > > Well, presumably IPK creation tools can only support the > > matrix of > > compression algorithms which your opkg binary can decompress. > > I suppose > > someone could try to implement a plugable compression module > > system for > > opkg. But given that nearly everyone uses opkg in an embedded > > context, > > I'm not sure it would get much use. > > > > > > > > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj > > <raj.k...@gmail.com> wrote: > > > > > > On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart > > > <alex.stew...@ni.com> wrote: > > > > > > > > ACK from me - apart from enabling zstd by default. > > > > > > > > On 9/13/22 07:37, Etienne Cordonnier via > > lists.openembedded.org > > > > <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$> > > > > > > > <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$> > > > wrote: > > > > > This allows the use of zstd for opkg packages by using > > > OPKGBUILDCMD: > > > > > OPKGBUILDCMD = "opkg-build -Z zstd" > > > > > > > > > > Signed-off-by: Alex Feinman <afein...@snap.com> > > > > > Signed-off-by: Etienne Cordonnier <ecordonn...@snap.com> > > > > > --- > > > > > meta/recipes-devtools/opkg/opkg_0.6.0.bb > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$> > > > > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$> > > > | 3 ++- > > > > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > > > > > > > diff --git > > a/meta/recipes-devtools/opkg/opkg_0.6.0.bb > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$> > > > > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$> > > > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$> > > > > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$> > > > > > index 7b351e8123..e38d9d6f3f 100644 > > > > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$> > > > > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$> > > > > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$> > > > > > > > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$> > > > > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest > > > > > target_localstatedir := "${localstatedir}" > > > > > OPKGLIBDIR ??= "${target_localstatedir}/lib" > > > > > > > > > > -PACKAGECONFIG ??= "libsolv" > > > > > +PACKAGECONFIG ??= "libsolv zstd" > > > > > > > > Building in zstd support by default is a little > > suspect to me. > > > > > > > > Unless I'm mistaken, OE-core will only build > > xz-compressed IPKs by > > > > default. So zstd support would be unnecessary for a distro > > > integrator > > > > who just uses upstream OE-core. > > > > > > > > For distros which use zstd compression in their > > packages, I think it > > > > would be more appropriate to overwrite the opkg > > PACKAGECONFIG in a > > > > .bbappend. > > > > > > > > > > This is perhaps fine. I do wonder if there is some > > performance > > > comparison data between xz and zstd compressed ipks > > > with opkg, it might help users on making this choice and > > also if we > > > should consider using > > > zstd by default at some point or not. > > > > > > > Is there something I'm not considering here? > > > > > > > > > > > > > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\ > > > > > gnupg gpgme libgpg-error,\ > > > > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = > > > "--enable-gpg,--disable-gpg,\ > > > > > PACKAGECONFIG[curl] = > > "--enable-curl,--disable-curl,curl" > > > > > PACKAGECONFIG[ssl-curl] = > > > "--enable-ssl-curl,--disable-ssl-curl,curl openssl" > > > > > PACKAGECONFIG[sha256] = > > "--enable-sha256,--disable-sha256" > > > > > +PACKAGECONFIG[zstd] = > > "--enable-zstd,--disable-zstd,zstd" > > > > > PACKAGECONFIG[libsolv] = > > > "--with-libsolv,--without-libsolv,libsolv" > > > > > > > > > > EXTRA_OECONF:class-native = > > > "--localstatedir=/${@os.path.relpath('${localstatedir}', > > > '${STAGING_DIR_NATIVE}')} > > > --sysconfdir=/${@os.path.relpath('${sysconfdir}', > > > '${STAGING_DIR_NATIVE}')}" > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Alex Stewart > > > > Software Engineer - NI Real-Time OS > > > > NI (National Instruments) > > > > > > > > alex.stew...@ni.com > > > > > > > > > > > > > > > > > > > > > > > -- > > Alex Stewart > > Software Engineer - NI Real-Time OS > > NI (National Instruments) > > > > alex.stew...@ni.com > > > > -- > Alex Stewart > Software Engineer - NI Real-Time OS > NI (National Instruments) > > alex.stew...@ni.com >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#170665): https://lists.openembedded.org/g/openembedded-core/message/170665 Mute This Topic: https://lists.openembedded.org/mt/93654146/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
