Switching XZ for ZSTD?
Hi, I just installed Fedora on 2 of my PCs a couple of weeks ago. One version of Fedora 39 release and one of Fedora 40 to see where things are going. I learned about this XZ-hack from Ars Technica & The Economist. I got to the Fedora Magazine article and wasn't really clear on that. So I followed the discussion to this thread in this Development mailing list. I read a lot of it but _still_ can't 100% figure out what the final solution is going to be. I have a question about that. I'm for sure OK that a responsibly developed FOSS project can contribute value and should be welcomed. ISTM that if a package is used on critical-path or security-path by default in a Distro it needs a higher bar. IIUC from this thread and online discussions about XZ & alternatives that 1] Lack of committer 'Real' identity confidence and verification is a problem. 2] Undetected differences source + packaging in repo vs tarballs are unchecked. 3] Under-resourced development creates risk; 'Many eyes' bench depth in development is needed. 4] XZ has a single, unsupported committer. 5] ZSTD is developed & used at Facebook. 6] ZSTD matches or outperforms XZ and most other compression in most metrics. 7] ZSTD is already used for default compression by Distros. I get that there's never going to be 100% perfect solution. But wouldnt' switching Fedora from using XZ to ZSTD by default fix a lot of the uncertainty around at least this current issue? Is that being considered in Fedora? Or is the focus trying to fix XZ to continue to use it? Thanks for any help to understand all this :-) Cheers! Arnie -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hi Steve, >> Who's to say that one doesn't have the same basic issue? Same with any other >> project in FOSS for that matter. That's the idea I was trying to make. There are no guarantees are there? But you can minimize the social problems. The 'basic issue' I see is the "one or two" developers, some that nobody knows in person, vis-à-vis "many" developers on a big project. For me it's most important when the project is on a Distro critical- or security-path. Cheers! Arnie On Thursday, April 4th, 2024 at 9:41 AM, Steve Cossette wrote: > I have definitely not read 75% of the comments and articles about the xz > issues but I understand the general reason why this happened. > > Issue here is, let's say we do switch to an alternative, whatever it is. > Who's to say that one doesn't have the same basic issue? Same with any other > project in FOSS for that matter. > > I'd say keep using XZ if the maintainers are quick to fix issues and quick to > respond to the community's issues, this one for example. Everyone does > mistakes. It's fine as long as we learn from them. > > On Thu, Apr 4, 2024 at 9:26 AM Arnie T via devel > wrote: > >> Hi, >> >> I just installed Fedora on 2 of my PCs a couple of weeks ago. One version of >> Fedora 39 release and one of Fedora 40 to see where things are going. >> >> I learned about this XZ-hack from Ars Technica & The Economist. >> >> I got to the Fedora Magazine article and wasn't really clear on that. >> >> So I followed the discussion to this thread in this Development mailing list. >> >> I read a lot of it but _still_ can't 100% figure out what the final solution >> is going to be. >> >> I have a question about that. >> >> I'm for sure OK that a responsibly developed FOSS project can contribute >> value and should be welcomed. >> >> ISTM that if a package is used on critical-path or security-path by default >> in a Distro it needs a higher bar. >> >> IIUC from this thread and online discussions about XZ & alternatives that >> >> 1] Lack of committer 'Real' identity confidence and verification is a >> problem. >> 2] Undetected differences source + packaging in repo vs tarballs are >> unchecked. >> 3] Under-resourced development creates risk; 'Many eyes' bench depth in >> development is needed. >> 4] XZ has a single, unsupported committer. >> 5] ZSTD is developed & used at Facebook. >> 6] ZSTD matches or outperforms XZ and most other compression in most metrics. >> 7] ZSTD is already used for default compression by Distros. >> >> I get that there's never going to be 100% perfect solution. >> >> But wouldnt' switching Fedora from using XZ to ZSTD by default fix a lot of >> the uncertainty around at least this current issue? >> >> Is that being considered in Fedora? >> Or is the focus trying to fix XZ to continue to use it? >> >> Thanks for any help to understand all this :-) >> >> Cheers! >> >> Arnie >> -- >> ___ >> devel mailing list -- devel@lists.fedoraproject.org >> To unsubscribe send an email to devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org >> Do not reply to spam, report it: >> https://pagure.io/fedora-infrastructure/new_issue-- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hi Daniel, >> All that being said, there are plenty of bits of software that could start >> using zstd by default and it would probably make sense to do so. I know this isn't the best test but just looking at locate xz | grep xz$ | grep kernel.*xz$ | wc -l 13206 ISTM there's a log of .xz compressed packages just related to the kernel. And I would guess that to use them at runtime would need using XZ. I think for example Arch uses ZSTD for this already? Cheers! Arnie -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hi, > See also an upstream GNU discussion on whether more GNU packages > should start providing zstd, or even lzip, tarballs in addition to xz: > https://lists.gnu.org/archive/html/bug-standards/2024-04/msg00032.html I'm sure not going to tell any developers here something they don't know! But for anybody that's just starting to look at this I thought these were really helpful. https://manishrjain.com/compression-algo-moving-data https://linuxreviews.org/Comparison_of_Compression_Algorithms From both of those I get that ZSTD is a pretty good option to consider. Cheers! Arnie -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hello Rich, > There's also the issue that liblzma is widely used and offers specific > features which zstd does not[1]. > > [1] https://github.com/facebook/zstd/issues/395#issuecomment-535875379 Is that about this? https://github.com/facebook/zstd/tree/dev/contrib/seekable_format From a Distro decision viewpoint does an alternative like ZSTD have to solve all the problems XZ does to be considered? Even if it solves a bunch of other problems? Like the 'many eyes' one? My old manager was always quoting about "Analysis Paralysis" and "Don't let the perfect be the enemy of the good". I'm no expert on this for sure but it seems that changing what CAN be changed has some value. And dealing with the rest when you can. So I'm just curious what "Good enough" looks like to act on something for Fedora ? Cheers! Arnie -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hi, > There's no such thing as a "distro decision" on this one, as was > explained in the thread already. I'm sure the 'explanation' is all clear to you and the other Developers. I'm also sure that it's not all that clear to non-Developers. If the explanation was clear and obvious to me, here or anywhere ales, I wouldn't be asking the question. So, sorry, I guess. I guess I don't understand how a Distro decision is different from a Distro IN-decision. For example from what I can read you were in contact with this Jia Tan 'person' during this story. I hope that a Distro decision would support whatever it takes to give you the tools, time and support to make sure that this sort of thing doesn't sneak past you or anyone. If there's no way to make that kind of decision then it seems to me Developers could use better support. This all seems like a very big deal. Which is why I guess I am reading about it on The Economist. And why I'm hoping that the Distro has some options to take some actions. Cheers! Arnie -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hello Stephen, > How a decision to drop xz for some other compression library for software > would be a fairly slow process. First a person who is willing to do the work > would come up with a proposal on why it should be done and how it could be > done. They would be expected to also test to see how much trouble this would > be (aka find all the packages which use xz and could be changed to another > library, which ones couldn't and what the effects would be.) Once that is > done, they would make a general proposal to be reviewed by whatever technical > committee a distribution has (Fedora has one whose acronym is FESCO, Debian > has another or multiple others, etc). This would be reviewed and if accepted > it would go as a future release work with a staged plan where some packages > are moved in X release, some in X+1, and some final plan for X+2 (or backed > out completely for some reason before then). There would be some amount of > software which would rely on xz no matter what because either the upstream > has no interest in changing or it is meant to use xz period. > ... > Currently most groups are between 0 and 1. There are a lot of things which > need to be looked at before moving off can be looked at as a goal to make > sure we aren't making things worse. > > I hope the above helps Thanks, I understand more of your explanation of how it's done. I don't know how much time was needed to decide for example an Arch Distro change "Now using Zstandard instead of xz for package compression" https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ OK, that's my mistake. I thought that moving to open source Linux OS Distro like Redhat-related Fedora would result big or important issues can be fixed more efficiently than at Microsoft. I guess I'm learning that even important or wise choices (not saying _this_ is) can't be done with taking a long time. Even if they are security related issues. Thanks one more time for the nice explanation! Cheers! Arnie-- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hi Stephen, Thanks for the explanation. I just caught up with the article at the New York Times, Did One Guy Just Stop a Huge Cyberattack? https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html And the comic that looks like it fits the problem I'm most noticing here! https://xkcd.com/2347/ I have to admit that I still don't know what the best or most official "At least do this" instruction page is for a Fedora user. I don't see anything at the main https://fedoraproject.org/ website or its "News & Announcements" page. In this thread its becoming about the details of the process. But not yet about a solution. All of which I get. And in private emails people are insisting on sending to me about how I'm unreasonable for asking the questions, and "should have" understood this or that. So, with your discussion the best guess I can some up with is to make sure XZ is downgraded and just hope that one of this Jia Tan's 6000+ commits are still hidden in some other project with not enough eyes. Or that the XKCD coming true doesn't happen again. Cheers! Arnie-- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hi Guinevere, > TL;DR: as with most security issues, end users should update their systems. > > I think you may be caught in some news exaggeration. Don't get me wrong, this > hack was a huge thing, but it was discovered early enough that most (i'd > guess almost all) fedora users wont' have to do anything. > > For Fedora, the problem package was only in Fedora 40 Beta and Fedora > Rawhide. If you are not running these packages, this isn't more than a "wow, > that was a near miss" for the end user. If you are running either version, > the xz maintainer has already rolled back the problem update, so if you use > "dnf update" you are safe. > > Because of a stroke of luck (finding this as early as we did) its as simple > as that, we have an assumed good version that users can 'update' to, and > beyond that, us developers need to verify that the assumed good version is > actually good, and if it isn't, issue new updates. That was simply explained without burying it. Thanks. Someone again in private complained at me for "I should have read" the Fedora Magazine. Somehow I am supposed to know that Fedora *Magazine* is the official info source for FedoraProject, not the front page or even "News & Announcements". I guess I do now. Now read what is written at https://fedoramagazine.org/cve-2024-3094-security-alert-f40-rawhide/. Let me say I wish I had found your comment written in your way sooner! Even when you suspect it may be the case it's harder to evade any news exaggeration when it's not clear where to look or the places you do look are written in ways you can't clearly understand. So one more time, Thanks. Cheers! Arnie-- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
Hello Kevin, > I'm hopeful some things will come out of this as it's a chance for us to > look at our processes and improve them. I'm glad that's happening. It seems to me that improving those processes would be Distro decisions. Which I keep understanding don't really exist. At least not quickly. So I'm confused still. But glad. > > 1] Lack of committer 'Real' identity confidence and verification is a > > problem. > IMHO this isn't a problem. We don't have a right to demand anything from > open source projects. We can ask, we can urge, we can contribute and > change things, we can choose to not use something, or fork something. I really don't suggest 'demanding' anything. I do think it's wise to make choices to avoid it. Like just my example of a critical-path XZ with one developer vs a critical-path ZSTD built & maintained in a Facebook FOSS project. I know from just a business view I would never enter into a 'critical' contract without "knowing" the principal persons. Of course you must know what you need "knowing" to be. > > 2] Undetected differences source + packaging in repo vs tarballs are > > unchecked. > > > Yeah, a lot of the discussion has been in this area. > > I'm wondering if perhaps we shouldn't revisit source-git, or at least > a variant of it where we keep the upstream sources in a branch always > and apply packaging on top of that and build from there. I'm not sure what the packaging tools and rules are here. It seems to me that repo source with an attested commit (signature? published hash?) can serve as the one source of truth. Then users can pull the commit or the on-demand API-generated tarball. I guess that could be subject to for example Github's or Gitlab's API tarball generators being hacked. But that seems less probable of a concern. > > 3] Under-resourced development creates risk; 'Many eyes' bench depth in > > development is needed. > > Yep. I think also visibility of changes can be improved. > So, maintainers know more about whats in a new version and how it works. You can implement tools to increase the visibility for sure. And procedures. Also just the "given enough eyeballs, all bugs are shallow" that comes with using a larger project helps. Thanks for the discussion. Cheers! Arnie -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Uptodate builds for F40 FedoraMediaWriter and KDE LiveISOs ?
Hello, I'm setting up some F40 boxes for a client. Their Dev-box requirement is to install & use *current* stable release versions of F40 & KDE. For running OS and any tools etc. I installed Fedora Media Writer from distro repos. When you exec FMW to write to a usb key the built-in downloadable iso options only include up to F40 *BETA*. Is there a setting or a newer version that pulls current F40 *RELEASE*? The alternative is to DL source isos from the site and manually install. The current available LiveISO for F40 KDE Spin doesn't include KDE Plasma 6.1. Just 6.0.x still. The Rawhide iso spin looks like it does. But I don't want to point them at Rawhide. I want to stick with current release. Where can you get current F40 + KDE Spin isos? Arn -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Uptodate builds for F40 FedoraMediaWriter and KDE LiveISOs ?
> I think, you should download from > > https://fedoraproject.org/spins/kde/ > > and use FMW with the downloaded image or use balenaEtcher to write to the USB > stick > > > > > -- > Peter Boy > https://fedoraproject.org/wiki/User:Pboy > p...@fedoraproject.org Right. From that page I downloaded this LiveISO ls -al Fedora-KDE-Live-x86_64-40-1.14.iso -rw-r--r-- 1 arn1 arn1 2.5G Jun 23 16:57 Fedora-KDE-Live-x86_64-40-1.14.iso and launched from it. The plasma version the downloaded image contains is plasma-workspace-wayland-6.0.3-2.fc40.x86_64 According to https://src.fedoraproject.org/rpms/plasma-workspace, the current release is plasma-workspace-*-6.1.0-3.fc40 I'm looking for an uptodate LiveISO that packages current releases. And for a Fedora Media Writer that is F40 *release* aware. Are either of those available? -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Moving away from the term "karma" in Bodhi
Rama-Kandra: That is our karma. Neo: You believe in karma? Rama-Kandra: Karma's a word. Like "love". A way of saying 'what I am here to do.' I do not resent my karma - I'm grateful for it. Grateful for my wonderful wife, for my beautiful daughter. They are gifts. And so I do what I must do to honour them. This pearl-clutching is an embarrassment. Word-policing is offensive. Much more than the use of a word widely used in non-religious contexts. Do you really think that people are using 'karma' here in its religious context? With the intention of disrespecting anybody? Do you think that forcing your ideas of word usage is going to change the beliefs of anyone? Other than to convince people that the word police are at it again? Oh, wow. 'Belief' has religious context too. My bad. Where's my hand-sanitizer? Devs complain constantly they don't have time to fix bug X that's really been disenfranchising people for years on end, but will spend hours & days beating this to death. Hilarious, ironic point somebody made about "Bodhi". Since group-think & social-media are the new normal what does the AI think? "Bodhi" is a term derived from the Sanskrit and Pali languages, meaning "awakening" or "enlightenment." It is most commonly associated with Buddhism and refers to the understanding or realization of the true nature of reality. Here are some key points about Bodhi: Bodhi Tree: The Bodhi tree (Ficus religiosa) is significant in Buddhism as it is the tree under which Siddhartha Gautama, who became the Buddha, attained enlightenment. This event is a pivotal moment in Buddhist tradition. Bodhi as Enlightenment: In the context of Buddhism, achieving Bodhi means gaining insight into the nature of suffering, the impermanence of life, and the interconnectedness of all beings. It is the realization of Nirvana, the ultimate goal of Buddhist practice. Bodhisattva: In Mahayana Buddhism, a Bodhisattva is an enlightened being who seeks to help others achieve enlightenment rather than entering Nirvana themselves. This concept emphasizes compassion and altruism. Bodhi in Meditation: The pursuit of Bodhi often involves meditation, ethical conduct, and the development of wisdom. Practitioners engage in various forms of meditation to cultivate mindfulness and insight. Cultural Significance: Bodhi has also been embraced in various cultural contexts, symbolizing spiritual awakening and personal growth beyond Buddhism. Overall, Bodhi represents a profound spiritual goal in Buddhism, emphasizing the importance of awakening to the truth of existence and the compassionate pursuit of helping others along the path to enlightenment. Yeah. That's gotta go too. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
