Thanks for checking.
I'd be interested to know if setting a higher compression level for zstd
can get us to a similar compression ratio to xz. If so, then I think it
could be some real value to distro maintainers to be able to *tune*
their compression.
That's not blocking for your new PR though.
On 9/14/22 05:08, Etienne Cordonnier wrote:
Also note that zstd's default compression level is 3 per default (from
a 1 to 19 scale). I did not test other compression levels.
On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier
<ecordonn...@snapchat.com> wrote:
I ran a build of core-image-full-cmdline using xz and zstd, using
pre-populated downloads and sstate-cache directories but with
empty tmp directory. Here are the numbers:
zstd:
bitbake core-image-full-cmdline took 2m52.768s (real), the
resulting directory tmp/deploy/ipk is 1.6GB big.
xz:
bitbake core-image-full-cmdline took 4m14.214s (real), the
resulting directory tmp/deploy/ipk/ is 996M big
So the build with xz is 47% slower (254/172) than with zstd for
this "incremental build" use-case.
See the 5 biggest packages, the difference in compression-ratio
increases with big files:
build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
total 1.6G
-rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
binutils-dbg_2.39-r0_core2-64.ipk
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
total 963M
-rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
...
I think for use-cases where the size of the ipk packages matters,
xz is better. For use-cases where it does not matter (ipk packages
not deployed), build-time matters more than compression-ratio and
zstd is better.
Regarding the enablement of zstd in opkg per default, I'll send a
new version of the patch without this line.
My thinking for enabling zstd per default in opkg was that zstd is
already enabled per default in libarchive's PACKAGECONFIG, and
disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
code from opkg (opkg uses libarchive for doing the actual
compression/decompression).
On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
<alex.stew...@ni.com> wrote:
On 9/13/22 15:20, Alex Feinman wrote:
> I do have some numbers. When I was selling this change
internally, I
> did a comparison on our internal build.
> Combined write IPK times (Σ t do_package_write_ipk)
> xz 162m 35s
> gz 52m 13s
> zstd 33m 49s
> Compression rate for zstd was closer to xz than to gz but
not as good
> as xz. For systems that have to cache packages on the device
with
> limited storage xz might be a better option, but for the
bulk of
> projects zstd is the best choice
> Additionally, zstd offers much faster decompression than xz
so the
> rootfs build step that includes unpacking all of the ipks,
takes 3m
> 58s with xz and 2m 44s with zstd.
> One other thing of note - if your build includes debug
packages, some
> may be quite large. E.g. one of our components produces a
2.2 GB debug
> package (uncompressed). On large files xz requires a
disproportionally
> large amount of time resulting in 15 minutes needed to
simply write
> ipk for the abovementioned packages, whereas zstd took about
45 sec.
> For frequent tasks like bitbaking a single package this
translates in
> a lot of saved time.
Those are certainly compelling performance improvements.
Assuming that
the final data-segment size is within 5%-ish of xz, then I
would agree
with the rest of the thread that it should probably be the
contemporary
default.
And if we make it the default compressor for OE IPKs, then
obviously my
criticism in the original PR is satisfied.
> Bottom line - I think making xz a default package compressor
was not
> entirely thought through. gzip or zstd is what the default
should be.
ZStandard support was only added to opkg last September [1].
Before
that, xz was the new hotness that replaced gzip. :)
[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=>
> One final note: I could not find a reasonable explanation
for why
> opkg-tools require code changes to support a different
compressor. BSD
> tar and GNU tar both can easily accept compressors that they
have no
> idea about (via -I option) because all of them provide a
unified
> command line interface for use in pipes. If this were done
similar to
> tar, we could have used any compressor we wanted, including the
> multithreaded versions (zstdmt)
Well, presumably IPK creation tools can only support the
matrix of
compression algorithms which your opkg binary can decompress.
I suppose
someone could try to implement a plugable compression module
system for
opkg. But given that nearly everyone uses opkg in an embedded
context,
I'm not sure it would get much use.
>
> On Tue, Sep 13, 2022 at 12:43 PM Khem Raj
<raj.k...@gmail.com> wrote:
>
> On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
> <alex.stew...@ni.com> wrote:
> >
> > ACK from me - apart from enabling zstd by default.
> >
> > On 9/13/22 07:37, Etienne Cordonnier via
lists.openembedded.org
<https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$>
>
<https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
> wrote:
> > > This allows the use of zstd for opkg packages by using
> OPKGBUILDCMD:
> > > OPKGBUILDCMD = "opkg-build -Z zstd"
> > >
> > > Signed-off-by: Alex Feinman <afein...@snap.com>
> > > Signed-off-by: Etienne Cordonnier <ecordonn...@snap.com>
> > > ---
> > > meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> | 3 ++-
> > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git
a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > index 7b351e8123..e38d9d6f3f 100644
> > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
> > > target_localstatedir := "${localstatedir}"
> > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
> > >
> > > -PACKAGECONFIG ??= "libsolv"
> > > +PACKAGECONFIG ??= "libsolv zstd"
> >
> > Building in zstd support by default is a little
suspect to me.
> >
> > Unless I'm mistaken, OE-core will only build
xz-compressed IPKs by
> > default. So zstd support would be unnecessary for a distro
> integrator
> > who just uses upstream OE-core.
> >
> > For distros which use zstd compression in their
packages, I think it
> > would be more appropriate to overwrite the opkg
PACKAGECONFIG in a
> > .bbappend.
> >
>
> This is perhaps fine. I do wonder if there is some
performance
> comparison data between xz and zstd compressed ipks
> with opkg, it might help users on making this choice and
also if we
> should consider using
> zstd by default at some point or not.
>
> > Is there something I'm not considering here?
> >
> > >
> > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
> > > gnupg gpgme libgpg-error,\
> > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
> "--enable-gpg,--disable-gpg,\
> > > PACKAGECONFIG[curl] =
"--enable-curl,--disable-curl,curl"
> > > PACKAGECONFIG[ssl-curl] =
> "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
> > > PACKAGECONFIG[sha256] =
"--enable-sha256,--disable-sha256"
> > > +PACKAGECONFIG[zstd] =
"--enable-zstd,--disable-zstd,zstd"
> > > PACKAGECONFIG[libsolv] =
> "--with-libsolv,--without-libsolv,libsolv"
> > >
> > > EXTRA_OECONF:class-native =
> "--localstatedir=/${@os.path.relpath('${localstatedir}',
> '${STAGING_DIR_NATIVE}')}
> --sysconfdir=/${@os.path.relpath('${sysconfdir}',
> '${STAGING_DIR_NATIVE}')}"
> > >
> > >
> > >
> >
> > --
> > Alex Stewart
> > Software Engineer - NI Real-Time OS
> > NI (National Instruments)
> >
> > alex.stew...@ni.com
> >
> >
> >
> >
>
--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)
alex.stew...@ni.com
--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)
alex.stew...@ni.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170664):
https://lists.openembedded.org/g/openembedded-core/message/170664
Mute This Topic: https://lists.openembedded.org/mt/93654146/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
