Thanks for sharing, really appreciate it. How close were compression
rates in numbers ? I think some users might be sensitive to the
archive sizes, if its something
with in acceptable range, I do agree, zstd would be a better default these days.
On Tue, Sep 13, 2022 at 1:20 PM Alex Feinman <afein...@snap.com> wrote:
>
> I do have some numbers. When I was selling this change internally, I did a
> comparison on our internal build.
>
> Combined write IPK times (Σ t do_package_write_ipk)
>
> xz 162m 35s
>
> gz 52m 13s
>
> zstd 33m 49s
>
> Compression rate for zstd was closer to xz than to gz but not as good as xz.
> For systems that have to cache packages on the device with limited storage xz
> might be a better option, but for the bulk of projects zstd is the best choice
>
> Additionally, zstd offers much faster decompression than xz so the rootfs
> build step that includes unpacking all of the ipks, takes 3m 58s with xz and
> 2m 44s with zstd.
>
> One other thing of note - if your build includes debug packages, some may be
> quite large. E.g. one of our components produces a 2.2 GB debug package
> (uncompressed). On large files xz requires a disproportionally large amount
> of time resulting in 15 minutes needed to simply write ipk for the
> abovementioned packages, whereas zstd took about 45 sec. For frequent tasks
> like bitbaking a single package this translates in a lot of saved time.
>
> Bottom line - I think making xz a default package compressor was not entirely
> thought through. gzip or zstd is what the default should be.
>
> One final note: I could not find a reasonable explanation for why opkg-tools
> require code changes to support a different compressor. BSD tar and GNU tar
> both can easily accept compressors that they have no idea about (via -I
> option) because all of them provide a unified command line interface for use
> in pipes. If this were done similar to tar, we could have used any compressor
> we wanted, including the multithreaded versions (zstdmt)
>
>
> On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <raj.k...@gmail.com> wrote:
>>
>> On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart <alex.stew...@ni.com> wrote:
>> >
>> > ACK from me - apart from enabling zstd by default.
>> >
>> > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org wrote:
>> > > This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
>> > > OPKGBUILDCMD = "opkg-build -Z zstd"
>> > >
>> > > Signed-off-by: Alex Feinman <afein...@snap.com>
>> > > Signed-off-by: Etienne Cordonnier <ecordonn...@snap.com>
>> > > ---
>> > > meta/recipes-devtools/opkg/opkg_0.6.0.bb | 3 ++-
>> > > 1 file changed, 2 insertions(+), 1 deletion(-)
>> > >
>> > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > index 7b351e8123..e38d9d6f3f 100644
>> > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
>> > > target_localstatedir := "${localstatedir}"
>> > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
>> > >
>> > > -PACKAGECONFIG ??= "libsolv"
>> > > +PACKAGECONFIG ??= "libsolv zstd"
>> >
>> > Building in zstd support by default is a little suspect to me.
>> >
>> > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
>> > default. So zstd support would be unnecessary for a distro integrator
>> > who just uses upstream OE-core.
>> >
>> > For distros which use zstd compression in their packages, I think it
>> > would be more appropriate to overwrite the opkg PACKAGECONFIG in a
>> > .bbappend.
>> >
>>
>> This is perhaps fine. I do wonder if there is some performance
>> comparison data between xz and zstd compressed ipks
>> with opkg, it might help users on making this choice and also if we
>> should consider using
>> zstd by default at some point or not.
>>
>> > Is there something I'm not considering here?
>> >
>> > >
>> > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>> > > gnupg gpgme libgpg-error,\
>> > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>> > > PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
>> > > PACKAGECONFIG[ssl-curl] = "--enable-ssl-curl,--disable-ssl-curl,curl
>> > > openssl"
>> > > PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
>> > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
>> > > PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
>> > >
>> > > EXTRA_OECONF:class-native =
>> > > "--localstatedir=/${@os.path.relpath('${localstatedir}',
>> > > '${STAGING_DIR_NATIVE}')}
>> > > --sysconfdir=/${@os.path.relpath('${sysconfdir}',
>> > > '${STAGING_DIR_NATIVE}')}"
>> > >
>> > >
>> > >
>> >
>> > --
>> > Alex Stewart
>> > Software Engineer - NI Real-Time OS
>> > NI (National Instruments)
>> >
>> > alex.stew...@ni.com
>> >
>> >
>> >
>> >
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170611):
https://lists.openembedded.org/g/openembedded-core/message/170611
Mute This Topic: https://lists.openembedded.org/mt/93654146/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
