On 2025/1/17 09:14, Neal Gompa wrote:
On Thu, Jan 16, 2025 at 8:00 PM Gao Xiang <hsiang...@linux.alibaba.com> wrote:
(resend this due to lack of subscription. and I just quote some reply to
express my basic ideas on this stuff)
There's one thing the EROFS developers never talk about, and that's metadata
compression. EROFS doesn't do it, but Squashfs does, and it has done since
2002.
Did you notice that https://erofs.docs.kernel.org/en/latest/faq.html ?
The tests are all done in way to flater EROFS and put Squashfs in the worst
light. For example see
https://erofs.docs.kernel.org/en/latest/comparsion/dedupe.html. They
deliberately disable Squashfs metadata compression because EROFS doesn't
support it, and use 128K blocks despite Squashfs supporting 1Mbyte blocks (look
for the line [1] SquashFS uses –b 131072 by default, -noI will disable its
metadata compression.).
It just shows that SquashFS doesn't support extent-based deduplication (because
SquashFS on-disk indexes just record compressed sizes and cannot be randomly
indexed), since EROFS doesn't support metadata compression so I benchmarked
against SquashFS and explicitly point out the metadata compression is off on
the SquashFS side to show the benefits of deduplication. Why it's called
unbiased?
Don't make me wrong, if -b 1048576 (1m chunks) is used, then the chunk
deduplication possibility is low.
But not anyone cares image size (especially for smartphone) just by using large
extent sizes (like 1MiB) and real-time performance is important for them,
decompress 1MiB to get 4KiB random data is unacceptable for them.
By doing this they deliberately reduce the compression and speed of Squashfs in
their tests. This IMHO is biased and unethical. But that is how it is.
Also if I were a random filesystem author, I will never argue just due to lack
of development resource.
Whether EROFS will use for Fedora LiveOS finally, honestly I don't care
(although I wish it could be used more widely), I will focus on new features
that end users really care and it's really an input. I only care EROFS can be
totally usable on RHEL/CentOS since we have customers to use these OSes and I
will serve for their detailed use cases.
Please note, EROFS was once proposed to resolve SquashFS unacceptable random
runtime performance issues on the high-performance Android scenerios, and
almost all smartphone vendors on the market use EROFS now. As for other
scenarios, I would just call it as a plus.
If users think metadata compression is useful to minimize their images, and I
will consider to add this in my spare time (Because it's never useful for my
paid job too, they even don't care about multithreaded compression). Also I
did not ask random vendors to contribute EROFS, just because Bytedance or other
vendors need it and they develop new features for their use cases.
I certainly would appreciate features like metadata compression, BCJ
filters, and anything to improve image creation time when -Ededupe is
used. I also wanted to use zstd compression instead of lzma (since it
decompresses way faster), but the compression performance isn't quite
there yet.
Since it's not my paid job, I will try my best to work them out in
my spare time. But really, for our own use cases we don't care much
about them, it's also called "a labor of love".
Runtime speed is really valuable for us because typical media that
Linux live environments are running from are slower than even most
mobile device flash storage modules. At worst, we're talking about
actually booting from a DVD (which is still a thing people do in some
places), but we also need maximal space savings because of download
bandwidth and fitting on small media.
Anyway, EROFS was once specifically designed for Android devices
initially, I will try to make it work better for other useful cases,
questions, issues, and features are much helpful for me to improve
the filesystem itself.
Thanks,
Gao Xiang
--
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
