Hello,
I have implemented the necessary changes in Pungi, the software that
creates Fedora compose. So this change has a potential of moving forward.
I have created 2 pull-requests for release engineering for this change:
https://pagure.io/pungi-fedora/pull-request/871
https://pagure.io/pungi-fedora/pull-request/872
Fedora release engineering would have the ability to test this change
and provide fresh test results after Pungi 4.2.4 is released.
I hope this change could land in Fedora 33.
As a reminder, here is the change proposal:
https://fedoraproject.org/wiki/Changes/OptimizeSquashFS
Regards,
On 13/05/2020 13:32, Bohdan Khomutskyi wrote:
Hello,
It was a long time since the last message in this change proposal.
Recently I was working to reduce the impact of the increased
compression ratio on the installation image size for Fedora. I have
achieved outstanding results -- working proof of concept. With the
following change: https://github.com/rhinstaller/anaconda/pull/2292 ,
not only the higher compression does not impact the installation time.
In certain cases, the installation time is even reduced. This is
because of the fact the filesystem internal structure aware process is
used to install the system from the SquashFS. The new process also
allows for taking advantage of the multi-core architecture of the
system during installation -- does the decompression on multiple
processors in parallel.
The combination of https://github.com/rhinstaller/anaconda/pull/2292
and https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
should reduce _both_ the image size and the installation time. The
installation time will be reduced in case the system is installed from
the SquashFS. This is the case in Fedora Workstation.
For optimization of the SquashFS, I will work on requesting the
support of the required functionality in the Pungi compose build software.
On 25/01/2020 16:36, Bohdan Khomutskyi wrote:
Hello,
I opened a request to anaconda team with a draft patch to use
unsquashfs instead of rsync:
https://github.com/rhinstaller/anaconda/pull/2292.
That should lower the installation time from Live media.
Adjustments should be made to make this patch work, I was not able to
install the image using this patch. Anaconda installer crashes.
Chris,
> Could this be read amplification?
> That's probably mitigated with unsquashfs.
That could be read amplification, it will be mitigated with unsquashfs.
The memory usage should not be a problem: xz (1) manual page, states
that 2 MiB of memory required to decompress an archive with 1MiB
block size. My previous observations found that the squashfs is
currently decompressed in single thread. I welcome additional
testing, but I think the memory limit will not be a problem.
> If image size is a significant consideration, then evaluation of erofs
> seems indicated. It promises both significant compression and CPU
> performance. The intended use case is for Android device read-only
> partitions with both limited storage and CPU/power capacity.
I briefly reviewed the document and found that LZ4 is the only
supported algorithm in EROFS, even if EROFS has perfect layout of the
filesystem, it's to good to be true it can outperform the XZ in
compression ratio tests. The difference for SquashFS LZ4hc 1M vs XZ
1M is >40%.
On Fri, Jan 24, 2020 at 2:17 AM Chris Murphy <li...@colorremedies.com
<mailto:li...@colorremedies.com>> wrote:
On Mon, Jan 20, 2020 at 1:38 AM Bohdan Khomutskyi
<bkhom...@redhat.com <mailto:bkhom...@redhat.com>> wrote:
>
> In my previous message, I mentioned that CPU is underutilized
during installation. I haven't investigated further why, but I
suspect it's due to the inefficiency caused by the usage of the
loop device and/or inefficiency in the rsync itself.
Could this be read amplification?
This paper on erofs suggests read amplification can be a significant
side effect with squashfs. It could be exacerbated with random reads,
and I expect it gets worse with larger block size. That's probably
mitigated with unsquashfs.
Specifically page 4, 2nd paragraph.
https://www.usenix.org/system/files/atc19-gao.pdf
This also makes me wonder about the memory consumption effect of a 1M
block size, especially for Fedora ARM where it looks like
Raspberry Pi
2B
Most of the ARM images are raw.xz but some are bootable ISOs, dvd and
netinstall. And those contain a squashfs sysroot. Even if there's no
out of memory problem, it could result in paging. All ISOs setup
swap-on-ZRAM these days, lives, DVD, and netinstall. I think the ARM
case needs testing before committing to 1M block size across all
ISOs,
or implementing changes in Fedora release engineering.
--
Chris Murphy
--
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts
--
Bohdan Khomutskyi
Red Hat
--
Bohdan Khomutskyi
Red Hat
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
