Re: Should we stop distributing source tarballs?
Am 05.04.24 um 06:25 schrieb Juraj Oravec: On streda 3. apríla 2024 18:34:04 CEST Albert Vaca Cintora wrote: Hi KDE folks, The recent xz backdoor scandal made me realize how bad and obsolete distributing tarballs is. The source of truth for our code are the repositories, and releases can simply be tags on those repos. As a big free software community, I think we should lead by example and get rid of tarballs altogether (as I hope to see in other projects as well) after the recent events. Packagers can git pull. If we ever replace git with something else, that something else will have tags as well. What's the advantage of providing tarballs? Albert Hello Albert, The release tarballs can be signed with GPG (or is it PGP?) which provide another layer of protection to make sure the release is authenthic. If KDE wants to lead by example and use only git tags for releases, at least the tags should be signed with GPG for verification. It would be best to have all commits in the repository signed (in Gitlab "Verified"). While we are unable to make sure that the historical commits are also signed, since most of them are not, at least new commits and tags should be signed. Maybe the commits can be signed retrospectively (while breaking the repository history), but this is probablôy just my dream. If all commits in the xz repo would have been signed, the backdoor would have been sneaked in as well -- only that the commit would have been signed. Also if the tags would have been signed, the releases with the backdoor would have been published exactly as is -- only difference: The respective tags would have been signed. Just sayin ... With modern approach for "reproducible" builds in the Linux distributions, it is required to provide a way to make sure that the release is authentic, the tarballs allows that, but with current use of git tags we do not even provide a way to make sure the tag was made by trusted developer or a release team, iinstead the tag could be faked by anyone providing another way of entry. Have a nice day. Juraj
Re: Should we stop distributing source tarballs?
On piatok 5. apríla 2024 9:04:14 CEST Tobias Leupold wrote: > Am 05.04.24 um 06:25 schrieb Juraj Oravec: > > Hello Albert, > > > > The release tarballs can be signed with GPG (or is it PGP?) which > > provide another layer of protection to make sure the release is > > authenthic. > > > > If KDE wants to lead by example and use only git tags for releases, > > at least the tags should be signed with GPG for verification. > > > > It would be best to have all commits in the repository signed (in > > Gitlab "Verified"). While we are unable to make sure that the > > historical commits are also signed, since most of them are not, at > > least new commits and tags should be signed. Maybe the commits can > > be signed retrospectively (while breaking the repository history), > > but this is probablôy just my dream. > > If all commits in the xz repo would have been signed, the backdoor > would have been sneaked in as well -- only that the commit would have > been signed. Also if the tags would have been signed, the releases > with the backdoor would have been published exactly as is -- only > difference: The respective tags would have been signed. > > Just sayin ... You are correct, it would not solve a problem of corrupted tarballs. I am saying this for the "git tag" approach proposed in the first mail. How do we ensure that the repository was not tempered with by third party along the way by lets say governments or network companies? The governments wants (and in some states they already do) install a root certificates into your machines so that they can interfere in the encrypted https traffic. If the commits or at least tags are not signed, it makes it easy for them (in the name of safety) to redirect the packager or user to different server with tempered repository. Noone will suspect anything since there is no mechanism to make sure it is authentic. Other than hard working honest developers, nothing can protect us from the xz type of attack. Juraj signature.asc Description: This is a digitally signed message part.
Re: Should we stop distributing source tarballs?
It seems a lot of people feel conservative in favor of tarballs, so maybe I aimed too far. At least I think the discussion brought some interesting points that we can explore further. Some I identified: - The tarballs should contain no changes with respect to git, or minimal changes obviously justifiable in a diff. - Tarballs should only be generated in a reproducible manner using scripts. Ideally by the CI only. - We should start to sign tarballs in the CI. - We should start to sign commits and tags. Git recently made this super easy by allowing signing with the ssh keys that we all are already using to push things, so no excuses for not enabling this. Sample config below: [user] signingkey = [commit] gpgsign = true [gpg] format = ssh [tag] forceSignAnnotated = true
Re: Should we stop distributing source tarballs?
On Freitag, 5. April 2024 12:04:28 CEST Albert Vaca Cintora wrote: > It seems a lot of people feel conservative in favor of tarballs, so > maybe I aimed too far. At least I think the discussion brought some > interesting points that we can explore further. Some I identified: > > - The tarballs should contain no changes with respect to git, or > minimal changes obviously justifiable in a diff. > - Tarballs should only be generated in a reproducible manner using > scripts. Ideally by the CI only. > - We should start to sign tarballs in the CI. We could easily add a new service for signing and publishing the tarballs to our CI/CD system. The necessary basic infrastructure has been added in the last few months as part of our migration from Binary Factory to GitLab. Regards, Ingo signature.asc Description: This is a digitally signed message part.
Re: Should we stop distributing source tarballs?
On Friday, April 5, 2024 12:04:28 PM CEST Albert Vaca Cintora wrote: > It seems a lot of people feel conservative in favor of tarballs, so > maybe I aimed too far. At least I think the discussion brought some > interesting points that we can explore further. Some I identified: > > - The tarballs should contain no changes with respect to git, or > minimal changes obviously justifiable in a diff. I would argue that there should be no change at all. Adapting the versions and adding the version to the AppStream file should be done in a git commit and not done in the tarball. This is already done by everyone using releaseme and following the steps from https://community.kde.org/ReleasingSoftware > - Tarballs should only be generated in a reproducible manner using > scripts. Ideally by the CI only. > - We should start to sign tarballs in the CI. I disagree. I want my tarball to be signed with my GPG key stored in my Yubiky and not by a generic KDE key. It should be a proof that I as a maintainer of a project did the release and not someone else. Same with the upload to download.kde.org, while this adds some overhead in the process, I think it is important that KDE Sysadmins are the one who move the tarball to their final location and do some minimal check (checksum match, it's not a random person doing the release, ...). > - We should start to sign commits and tags. Git recently made this > super easy by allowing signing with the ssh keys that we all are > already using to push things, so no excuses for not enabling this. > Sample config below: > > [user] > signingkey = > [commit] > gpgsign = true > [gpg] > format = ssh > [tag] > forceSignAnnotated = true +1 git tags are already signed for people following the releaseme workflow. Signing commits is also good and I encourage everyone to do it but I wouldn't make it a requirement as it increases the barrier to contribution for new contributors. signature.asc Description: This is a digitally signed message part.
Re: Should we stop distributing source tarballs?
On Friday, 5 April 2024 12:04:28 CEST, Albert Vaca Cintora wrote: It seems a lot of people feel conservative in favor of tarballs, so maybe I aimed too far. At least I think the discussion brought some interesting points that we can explore further. Some I identified: - The tarballs should contain no changes with respect to git, or minimal changes obviously justifiable in a diff. Like I wrote earlier in this thread, this should hold true already since the translations are synced to git. - Tarballs should only be generated in a reproducible manner using scripts. Ideally by the CI only. - We should start to sign tarballs in the CI. The scripts already exists for the bundles and users of releasme. Letting the CI create tarballs seems indeed like a good way to reduce possibilities of human tampering. With regard to signing: At first I thought that it means something that the person responsible for the release is also signing the tarballs. But maybe there could even be two signatures in the future, one by CI and one by the release person (although that would probably mean a few challenges for the tooling). - We should start to sign commits and tags. Git recently made this super easy by allowing signing with the ssh keys that we all are already using to push things, so no excuses for not enabling this. We already sign commits for the 3 release bundles and users of relaseme. But I'm surprised about the total absence of social aspects of the xz incidence during this discussion. Admittedly we fall back to community maintenance if a maintainer becomes unavailable, so the exact xz scenario doesn't seem likely. But there are probably a few projects on the fringes. Do we have safeguards in place if a nefarious person tries to steps up as maintainer? Can we do something to help projects, which are struggling? Regards, Heiko
Re: Should we stop distributing source tarballs?
Am Freitag, 5. April 2024, 13:45:35 CEST schrieb Carl Schwan: > On Friday, April 5, 2024 12:04:28 PM CEST Albert Vaca Cintora wrote: > > - Tarballs should only be generated in a reproducible manner using > > scripts. Ideally by the CI only. > > - We should start to sign tarballs in the CI. > > I disagree. I want my tarball to be signed with my GPG key stored in my > Yubiky and not by a generic KDE key. It should be a proof that I as a > maintainer of a project did the release and not someone else. Same with the > upload to download.kde.org, while this adds some overhead in the process, I > think it is important that KDE Sysadmins are the one who move the tarball > to their final location and do some minimal check (checksum match, it's not > a random person doing the release, ...). Signing with a KDE key could have some benefits, though. It's far easier for distros (or users) to check KDE software against a single, well known key. On could mitigate the downside that you mentioned by having the script check the tag signature against a keyring of trusted keys. Cheers, Johannes signature.asc Description: This is a digitally signed message part.
Re: Should we stop distributing source tarballs?
On Sat, Apr 6, 2024 at 1:43 AM Heiko Becker wrote: > On Friday, 5 April 2024 12:04:28 CEST, Albert Vaca Cintora wrote: > > It seems a lot of people feel conservative in favor of tarballs, so > > maybe I aimed too far. At least I think the discussion brought some > > interesting points that we can explore further. Some I identified: > > > > - The tarballs should contain no changes with respect to git, or > > minimal changes obviously justifiable in a diff. > > Like I wrote earlier in this thread, this should hold true already since > the translations are synced to git. > I would consider any tarball that contains modified files (ie. is not a one to one representation of the git tag it represents) to be invalid for the purposes of KDE project releases. Now that translations are synced into our Git repositories there is no reason why we should have differences. > > > - Tarballs should only be generated in a reproducible manner using > > scripts. Ideally by the CI only. > > - We should start to sign tarballs in the CI. > > The scripts already exists for the bundles and users of releasme. Letting > the CI create tarballs seems indeed like a good way to reduce > possibilities > of human tampering. > With regard to signing: At first I thought that it means something that > the > person responsible for the release is also signing the tarballs. But maybe > there could even be two signatures in the future, one by CI and one by the > release person (although that would probably mean a few challenges for the > tooling). > > > - We should start to sign commits and tags. Git recently made this > > super easy by allowing signing with the ssh keys that we all are > > already using to push things, so no excuses for not enabling this. > > We already sign commits for the 3 release bundles and users of relaseme. > > But I'm surprised about the total absence of social aspects of the xz > incidence during this discussion. > Admittedly we fall back to community maintenance if a maintainer becomes > unavailable, so the exact xz scenario doesn't seem likely. But there are > probably a few projects on the fringes. Do we have safeguards in place if > a > nefarious person tries to steps up as maintainer? Can we do something to > help projects, which are struggling? > This is really the key issue at hand. If you look at the timeline in question, the malicious actor(s) in the XZ incident moved slowly, starting as a general contributor, accruing trust and eventually made their way up to getting maintainer access and the ability to make releases. Even with the various checks and balances we have in place around granting developer accounts, who gets permissions to update our websites, requiring people to go through tickets to publish releases on download.kde.org, etc. a similar kind of attack played out over a long enough period of time is still one that none of our technological protections would be able to stop - as people who have been long term contributors and have worked their way up to being a maintainer will tend to obtain the necessary permission or trust of those with permissions. Such attacks however will always fail when confronted with enough sunlight however, something that Linus' law of "with enough eyes all bugs are shallow" covers fairly well. To adequately protect ourselves we need to ensure that projects that only have 1 or 2 maintainers, as well as those that are "community maintained" are monitored (historically a function that kde-commits@ has filled). There is also something to be said that for larger projects such as ours (ie. the ones who are targets) that we should also be keeping an eye on the smaller libraries that we depend on - as while we may be too hard to attack it is much easier to attack those we depend on who don't have the same resources we do (which is basically what happened with XZ/OpenSSH). These would be projects with just one or two maintainers with fairly infrequent activity, and that are often located in a maintainers personal namespace on GitHub or Gitlab.com, or on personal hosting of their own. Examples of projects in this sort of space would be the likes of QtKeychain or kColorPicker (just picking two that immediately spring to mind), but there is no reason that shouldn't also cover non-Qt libraries. There are probably a couple of different options we could explore in this space as to how to best accomplish this. We should also consider whether it is acceptable for us to depend on any project that uses a build system that is either bespoke or not modern (ie. autotools, self-written bash scripts, etc) - as that was a significant contributor to how the XZ attack was able to be perpetrated. My vote would be that it is not acceptable and we should be strongly pushing any project that does not use a modern build system to migrate to one that is. > > Regards, > Heiko > Cheers, Ben
Re: Should we stop distributing source tarballs?
On Sat, Apr 6, 2024 at 4:23 AM Johannes Zarl-Zierl wrote: > Am Freitag, 5. April 2024, 13:45:35 CEST schrieb Carl Schwan: > > On Friday, April 5, 2024 12:04:28 PM CEST Albert Vaca Cintora wrote: > > > - Tarballs should only be generated in a reproducible manner using > > > scripts. Ideally by the CI only. > > > - We should start to sign tarballs in the CI. > > > > I disagree. I want my tarball to be signed with my GPG key stored in my > > Yubiky and not by a generic KDE key. It should be a proof that I as a > > maintainer of a project did the release and not someone else. Same with > the > > upload to download.kde.org, while this adds some overhead in the > process, I > > think it is important that KDE Sysadmins are the one who move the tarball > > to their final location and do some minimal check (checksum match, it's > not > > a random person doing the release, ...). > > Signing with a KDE key could have some benefits, though. It's far easier > for > distros (or users) to check KDE software against a single, well known key. > > On could mitigate the downside that you mentioned by having the script > check > the tag signature against a keyring of trusted keys. > Please see https://invent.kde.org/sysadmin/release-keyring/ - our process for validating tarballs for release already includes ensuring the GPG signatures provided are included in that keyring. All modern releases of KDE software that come with a GPG signature whose key is not in that keyring should be rejected. Developers should also consider adding their keys to Gitlab at https://invent.kde.org/-/profile/gpg_keys Following this, your GPG key will be published at https://invent.kde.org/$username.gpg > > Cheers, > Johannes Cheers, Ben
