Re: [gentoo-user] Re: Re[4]: Re: Portage, git and shallow cloning
On Sat, Jul 7, 2018 at 1:51 AM Martin Vaeth wrote: > > Davyd McColl wrote: > > > I ask because prior to the GitHub incident, I didn't have signature > > verification enabled > > Currently, it is not practical to change this, see my other posting. > You clearly don't understand what it actually checks. It is completely practical to enable this today (though not as secure as it could be). I'll elaborate in a reply to the other email. -- Rich
Re: [gentoo-user] Re: Re[4]: Re: Portage, git and shallow cloning
On Sat, Jul 7, 2018 at 1:34 AM Martin Vaeth wrote: > > Rich Freeman wrote: > > > > Biggest issue with git signature verification is that right now it > > will still do a full pull/checkout before verifying > > Biggest issue is that git signature happens by the developer who > last commited which means that in practice you need dozens/hundreds > of keys. This is untrue. The last git signature is made by infra or the CI-bot, and this is the signature that portage checks. Portage will NOT accept a developer key, or any other key in your keychain, as being valid. It will, of course, not work on the regular git repo used for committing for this reason. You need to use a repo that is signed by infra (which typically includes metadata/etc as well). I'll trim most of the rest of your email and only reply to significant bits, because you seem to not understand the point above which invalidates almost everything you wrote. The concerns you raise would be an issue if you were checking individual developer keys. > > So currently, it is impossible to do *any* automatic tree verification, > unless you manually fetch/update all of the developer keys. > As noted, you don't need to fetch any developer keys, and if you do fetch them, portage will ignore them. > > > unless you stick --force in your pull > > Unfortunately, it is not that simple: git pull --force only works if > the checked out tree is old enough (in which case git pull without --force > would have worked also, BTW). You completely trimmed the context around my quote. I was talking about the malicious commits in the recent attack. They were force-pushed, so it doesn't matter how complete your repository is - they simply would not be pulled without --force. You seem to be providing advice for how to do a pull with a shallow repository, which I'm not talking about. > > Honestly, I think git is a good fit for a lot of Gentoo users. > > At least since the ChangeLogs have been removed. > IMHO it was the wrong decision to not keep them in the rsync tree > (The tool to regenerate them from git was/is available). Changelogs are redundant with git, and they take a ton of space (which of late everybody seems to be super-concerned about). I don't get that on one hand people get twitchy about /usr/portage taking more than 1GB, and on the other hand they want a bazillion text files dumped all over the place, and as a bonus they want them prepended to instead of appended so that rsync resends the whole thing instead of just the tail... But, this was endlessly debated before the decision was made. Trust me, I read every post before voting to have them removed. > > > it is different, but all the history/etc is the sort of thing I think > > would appeal to many here. > > Having the ChangeLogs would certainly be sufficient for the majority > of users. It is very rare that a user really needs to access the > older version of the file, and in that case it is simple enough > to fetch it manually from e.g. github. It is very rare that somebody would want to use Gentoo at all. My point is that the sorts of people who like Gentoo would probably tend to like git. But, to each their own... > > > Security is obviously getting a renewed focus across the board > > Unfortunately, due to the mentioned keys problem, git is > currently the *unsafest* method for syncing. The "keys problem" has nothing to do with the security of git verification, because those keys are not used by git verification on the end-user side. An infra-controlled key is used for verification whether you sync with git or rsync. Either way you're relying on infra checking the developer keys at time of commit. Now, as I already mentioned git syncing is currently less safe due to it doing the checkout before the verification, and they are in the process of fixing this. > (BTW, due to the number of committers the portage tree has a quite > strict policy w.r.t. forced pushes. Overlays, especially of single > users, might have different policies and thus can fail quite often > due to the "git pull" bug.) It probably should be a configurable option in repos.conf, but honestly, forced pushes are not something that should be considered a good practice. There are times that it is the best option, but those are rare, IMO. -- Rich
[gentoo-user] PYTHON_SINGLE_TARGET inconsistent?
Hello,
I have a question considering PYTHON_SINGLE_TARGET because it seems to behave
inconsistently on my system and I cannot find any documentation about it, that
would
guide me in the right direction how to fix it.
Running emerge --info I see the following (I'm on ~amd64):
... PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7 python3_5" ...
This output is the default for ~amd64, neither PYTHON_SINGLE_TARGET nor
PYTHON_TARGETS
has been set explicitly.
If I do an emerge -av asciidoc, I see the following output:
ebuild R] app-text/asciidoc-8.6.10::gentoo USE="-examples -graphviz
-highlight {-test}" PYTHON_SINGLE_TARGET="python2_7 -pypy"
PYTHON_TARGETS="python2_7 -pypy"
Setting PYTHON_SINGLE_TARGET and PYTHON_TARGETS in /etc/portage/make.conf
explicitly to:
PYTHON_SINGLE_TARGET="python3_5"
PYTHON_TARGETS="python2_7 python3_5"
Now emerge --info doesn't show any difference. It's still the same output.
Doing an emerge -av asciidoc I get now this output:
!!! The ebuild selected to satisfy "asciidoc" has unmet requirements.
- app-text/asciidoc-8.6.10::gentoo USE="-examples -graphviz -highlight -test"
ABI_X86="(64)" PYTHON_SINGLE_TARGET="-pypy -python2_7"
PYTHON_TARGETS="python2_7 -pypy"
The following REQUIRED_USE flag constraints are unsatisfied:
exactly-one-of ( python_single_target_pypy python_single_target_python2_7 )
This seems for me to be really inconsistent, considering the fact that we
usually request users to provide an emerge --info,
which wouldn't show any difference.
Why am I able to merge in the first case the package, but in the second case I
get an
error? I really cannot understand where the python_single_target_python2_7 is
set in the
first case.
Anyone who has more insight into that case?
Cheers
Andreas
Re: [gentoo-user] PYTHON_SINGLE_TARGET inconsistent?
The package you're referring to have only support of python2 interpreters
(python2_7 (CPython) and pypy (not the pypy3)).
It seems, by default (if neither of PYTHON_SINGLE_TARGET is set), portage
tries to do the "magic" (well, in my opinion, it opposes to the Gentoo
Philosophy, and it should throw the same error even with "defaults") and
automatically chooses the 2_7 single target for it.
But when you explicitly specify the targets in make.conf, it obeys and don't
do that magic anymore, so condition of "exactly one of supported
single_targets (pypy, pyhton2.7)" is not meeting anymore".
That's why you getting the error.
В письме от суббота, 7 июля 2018 г. 21:31:48 MSK пользователь Andreas Fink
написал:
> Hello,
> I have a question considering PYTHON_SINGLE_TARGET because it seems to
> behave inconsistently on my system and I cannot find any documentation
> about it, that would guide me in the right direction how to fix it.
> Running emerge --info I see the following (I'm on ~amd64):
> ... PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7 python3_5"
> ...
>
> This output is the default for ~amd64, neither PYTHON_SINGLE_TARGET nor
> PYTHON_TARGETS has been set explicitly.
> If I do an emerge -av asciidoc, I see the following output:
> ebuild R] app-text/asciidoc-8.6.10::gentoo USE="-examples -graphviz
> -highlight {-test}" PYTHON_SINGLE_TARGET="python2_7 -pypy"
> PYTHON_TARGETS="python2_7 -pypy"
>
> Setting PYTHON_SINGLE_TARGET and PYTHON_TARGETS in /etc/portage/make.conf
> explicitly to: PYTHON_SINGLE_TARGET="python3_5"
> PYTHON_TARGETS="python2_7 python3_5"
>
> Now emerge --info doesn't show any difference. It's still the same output.
> Doing an emerge -av asciidoc I get now this output:
> !!! The ebuild selected to satisfy "asciidoc" has unmet requirements.
> - app-text/asciidoc-8.6.10::gentoo USE="-examples -graphviz -highlight
> -test" ABI_X86="(64)" PYTHON_SINGLE_TARGET="-pypy -python2_7"
> PYTHON_TARGETS="python2_7 -pypy"
>
> The following REQUIRED_USE flag constraints are unsatisfied:
> exactly-one-of ( python_single_target_pypy
> python_single_target_python2_7 )
>
>
> This seems for me to be really inconsistent, considering the fact that we
> usually request users to provide an emerge --info, which wouldn't show any
> difference.
> Why am I able to merge in the first case the package, but in the second case
> I get an error? I really cannot understand where the
> python_single_target_python2_7 is set in the first case.
> Anyone who has more insight into that case?
>
> Cheers
> Andreas
Re: [gentoo-user] PYTHON_SINGLE_TARGET inconsistent?
It's not automatically doing magic but using things specified in the
profile. In this case look at ${PORTDIR}/profile/base/package.use
Setting PYTHON_SINGLE_TARGET (which is expanded to those USEFlags) in your
make.conf will shadow those from the profile and spit out an error.
2018-07-07 21:45 GMT+03:00 Vadim A. Misbakh-Soloviov :
> The package you're referring to have only support of python2 interpreters
> (python2_7 (CPython) and pypy (not the pypy3)).
>
> It seems, by default (if neither of PYTHON_SINGLE_TARGET is set), portage
> tries to do the "magic" (well, in my opinion, it opposes to the Gentoo
> Philosophy, and it should throw the same error even with "defaults") and
> automatically chooses the 2_7 single target for it.
>
> But when you explicitly specify the targets in make.conf, it obeys and
> don't
> do that magic anymore, so condition of "exactly one of supported
> single_targets (pypy, pyhton2.7)" is not meeting anymore".
>
> That's why you getting the error.
>
> В письме от суббота, 7 июля 2018 г. 21:31:48 MSK пользователь Andreas Fink
> написал:
> > Hello,
> > I have a question considering PYTHON_SINGLE_TARGET because it seems to
> > behave inconsistently on my system and I cannot find any documentation
> > about it, that would guide me in the right direction how to fix it.
> > Running emerge --info I see the following (I'm on ~amd64):
> > ... PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7
> python3_5"
> > ...
> >
> > This output is the default for ~amd64, neither PYTHON_SINGLE_TARGET nor
> > PYTHON_TARGETS has been set explicitly.
> > If I do an emerge -av asciidoc, I see the following output:
> > ebuild R] app-text/asciidoc-8.6.10::gentoo USE="-examples
> -graphviz
> > -highlight {-test}" PYTHON_SINGLE_TARGET="python2_7 -pypy"
> > PYTHON_TARGETS="python2_7 -pypy"
> >
> > Setting PYTHON_SINGLE_TARGET and PYTHON_TARGETS in /etc/portage/make.conf
> > explicitly to: PYTHON_SINGLE_TARGET="python3_5"
> > PYTHON_TARGETS="python2_7 python3_5"
> >
> > Now emerge --info doesn't show any difference. It's still the same
> output.
> > Doing an emerge -av asciidoc I get now this output:
> > !!! The ebuild selected to satisfy "asciidoc" has unmet requirements.
> > - app-text/asciidoc-8.6.10::gentoo USE="-examples -graphviz -highlight
> > -test" ABI_X86="(64)" PYTHON_SINGLE_TARGET="-pypy -python2_7"
> > PYTHON_TARGETS="python2_7 -pypy"
> >
> > The following REQUIRED_USE flag constraints are unsatisfied:
> > exactly-one-of ( python_single_target_pypy
> > python_single_target_python2_7 )
> >
> >
> > This seems for me to be really inconsistent, considering the fact that we
> > usually request users to provide an emerge --info, which wouldn't show
> any
> > difference.
> > Why am I able to merge in the first case the package, but in the second
> case
> > I get an error? I really cannot understand where the
> > python_single_target_python2_7 is set in the first case.
> > Anyone who has more insight into that case?
> >
> > Cheers
> > Andreas
>
>
>
>
>
>
Re: [gentoo-user] PYTHON_SINGLE_TARGET inconsistent?
On Sat, 7 Jul 2018 21:51:31 +0300
Franz Fellner wrote:
> It's not automatically doing magic but using things specified in the
> profile. In this case look at ${PORTDIR}/profile/base/package.use
> Setting PYTHON_SINGLE_TARGET (which is expanded to those USEFlags) in your
> make.conf will shadow those from the profile and spit out an error.
>
> 2018-07-07 21:45 GMT+03:00 Vadim A. Misbakh-Soloviov :
>
> > The package you're referring to have only support of python2 interpreters
> > (python2_7 (CPython) and pypy (not the pypy3)).
> >
> > It seems, by default (if neither of PYTHON_SINGLE_TARGET is set), portage
> > tries to do the "magic" (well, in my opinion, it opposes to the Gentoo
> > Philosophy, and it should throw the same error even with "defaults") and
> > automatically chooses the 2_7 single target for it.
> >
> > But when you explicitly specify the targets in make.conf, it obeys and
> > don't
> > do that magic anymore, so condition of "exactly one of supported
> > single_targets (pypy, pyhton2.7)" is not meeting anymore".
> >
> > That's why you getting the error.
> >
> > В письме от суббота, 7 июля 2018 г. 21:31:48 MSK пользователь Andreas Fink
> > написал:
> > > Hello,
> > > I have a question considering PYTHON_SINGLE_TARGET because it seems to
> > > behave inconsistently on my system and I cannot find any documentation
> > > about it, that would guide me in the right direction how to fix it.
> > > Running emerge --info I see the following (I'm on ~amd64):
> > > ... PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7
> > python3_5"
> > > ...
> > >
> > > This output is the default for ~amd64, neither PYTHON_SINGLE_TARGET nor
> > > PYTHON_TARGETS has been set explicitly.
> > > If I do an emerge -av asciidoc, I see the following output:
> > > ebuild R] app-text/asciidoc-8.6.10::gentoo USE="-examples
> > -graphviz
> > > -highlight {-test}" PYTHON_SINGLE_TARGET="python2_7 -pypy"
> > > PYTHON_TARGETS="python2_7 -pypy"
> > >
> > > Setting PYTHON_SINGLE_TARGET and PYTHON_TARGETS in /etc/portage/make.conf
> > > explicitly to: PYTHON_SINGLE_TARGET="python3_5"
> > > PYTHON_TARGETS="python2_7 python3_5"
> > >
> > > Now emerge --info doesn't show any difference. It's still the same
> > output.
> > > Doing an emerge -av asciidoc I get now this output:
> > > !!! The ebuild selected to satisfy "asciidoc" has unmet requirements.
> > > - app-text/asciidoc-8.6.10::gentoo USE="-examples -graphviz -highlight
> > > -test" ABI_X86="(64)" PYTHON_SINGLE_TARGET="-pypy -python2_7"
> > > PYTHON_TARGETS="python2_7 -pypy"
> > >
> > > The following REQUIRED_USE flag constraints are unsatisfied:
> > > exactly-one-of ( python_single_target_pypy
> > > python_single_target_python2_7 )
> > >
> > >
> > > This seems for me to be really inconsistent, considering the fact that we
> > > usually request users to provide an emerge --info, which wouldn't show
> > any
> > > difference.
> > > Why am I able to merge in the first case the package, but in the second
> > case
> > > I get an error? I really cannot understand where the
> > > python_single_target_python2_7 is set in the first case.
> > > Anyone who has more insight into that case?
> > >
> > > Cheers
> > > Andreas
> >
> >
> >
> >
> >
> >
That was the file I was looking for. Thank you :)
[gentoo-user] Re: Re[4]: Re: Portage, git and shallow cloning
Rich Freeman wrote: > On Sat, Jul 7, 2018 at 1:34 AM Martin Vaeth wrote: >> >> Biggest issue is that git signature happens by the developer who >> last commited which means that in practice you need dozens/hundreds >> of keys. > > This is untrue. [...] > It will, of course, not work on the regular git repo [...] > You need to use a repo that is signed by infra > (which typically includes metadata/etc as well). I was speaking about gentoo's git repository, of course (the one which was attacked on github), not about a Frankensteined one with metadata history filling megabytes of disk space unnecessarily. Who has that much disk space to waste? For the official git repository your assertions are simply false, as you apprently admit: It is currently not possible to use the official git repo (or the github clone of it which was attacked) in a secure manner. >> > unless you stick --force in your pull >> >> Unfortunately, it is not that simple: git pull --force only works if > [...] > You completely trimmed the context around my quote. [...] > they simply would not be pulled without --force. I was saying that they would not be pulled *with* --force either, because pull --force is not as strong as you think it is (it would have shown you conflicts to resolve manually). You would have to use the commands that I have posted. > You seem to be providing advice for how to do a pull with a shallow > repository No, what I said is not related to a shallow repository. It has to do with pulling a forced push, in general. >> At least since the ChangeLogs have been removed. >> IMHO it was the wrong decision to not keep them in the rsync tree >> (The tool to regenerate them from git was/is available). > > Changelogs are redundant with git, and they take a ton of space (which > of late everybody seems to be super-concerned about) Compared to the git history, they take very little space. If you squash the portage tree, it is hardly measurable. And with the ChangeLogs, rsync would still be a sane option for most users. Without ChangeLogs many users are unnecessarily forced to change and to sacrifice the space for git history. > and as a bonus they want them prepended to > instead of appended so that rsync resends the whole thing instead of > just the tail... Your implicit claim is untrue. rsync - as used by portage - always transfers whole files, only. > But, this was endlessly debated before the decision was made. The decision was about removing the ChangeLogs from the git repository. This was certainly the correct decision, because - as you said - the ChangeLogs *can* be regenerated from the git history and thus it makes no sense to modify/store them redundantly. But I was speaking about the distribution of ChangeLogs in rsync: Whenever the infrastructure uses egencache to generate the metadata, it could simply pass --update-changelogs so that rsync users still would have ChangeLogs: They cannot get them from git history. > My > point is that the sorts of people who like Gentoo would probably tend > to like git. "Liking" git does not mean that one has to use it also for things for which it brings nothing. And for most users none of its features is useful for the portage tree. With one exception: ChangeLogs. That's why I am adverising to bring them back to the rsync tree. > The "keys problem" has nothing to do with the security of git > verification, because those keys are not used by git verification on > the end-user side. Whoever is that git/developer affine that he prefers git despite it costs more disk space will certainly want to use the actual source repository and not a worse rsync-clone repository. > It probably should be a configurable option in repos.conf, but > honestly, forced pushes are not something that should be considered a > good practice. 1. portage shouldn't decide about practices of overlays. 2. also in the official gentoo repository force pushes happen occassionally. Last occurrence was e.g. when undoing the malevolent forced push ;) 3. git pull fails not only for forced pushes but also in several other occassions; for instance, if your filesystem changed inodes numbers (e.g. squash + overlayfs after a resquash+remount). 4. Even if the user made the mistake to edit a file, portage should not just die on syncing.
[gentoo-user] Re: Re[4]: Re: Portage, git and shallow cloning
Rich Freeman wrote: > On Sat, Jul 7, 2018 at 1:51 AM Martin Vaeth wrote: >> Davyd McColl wrote: >> >> > I ask because prior to the GitHub incident, I didn't have signature >> > verification enabled >> >> Currently, it is not practical to change this, see my other posting. > > You clearly don't understand what it actually checks. Davyd and I were obviously speaking about the gentoo repository (the official one and the one on github which got hacked). For these repositories verification is practically not possible. (That there are also *other* repositories - with huge metadata history - which might be easier to verify is a different story). Perversely, the official comments after the hack had suggested that you should have enabled signature verification for the hacked repository which was simply practically not possible.
Re: [gentoo-user] Re: Re[4]: Re: Portage, git and shallow cloning
On Sat, Jul 7, 2018 at 5:29 PM Martin Vaeth wrote: > > Rich Freeman wrote: > > On Sat, Jul 7, 2018 at 1:34 AM Martin Vaeth wrote: > >> > >> Biggest issue is that git signature happens by the developer who > >> last commited which means that in practice you need dozens/hundreds > >> of keys. > > > > This is untrue. [...] > > It will, of course, not work on the regular git repo [...] > > You need to use a repo that is signed by infra > > (which typically includes metadata/etc as well). > > I was speaking about gentoo's git repository, of course > (the one which was attacked on github), not about a Frankensteined one > with metadata history filling megabytes of disk space unnecessarily. > Who has that much disk space to waste? Doesn't portage create that metadata anyway when you run it, negating any space savings at the cost of CPU to regenerate the cache? > > For the official git repository your assertions are simply false, > as you apprently admit: It is currently not possible to use the > official git repo (or the github clone of it which was attacked) > in a secure manner. > Sure, but this also doesn't support signature verification at all (at least not by portage - git can of course manually verify any commit), so your points still don't apply. > > and as a bonus they want them prepended to > > instead of appended so that rsync resends the whole thing instead of > > just the tail... > > Your implicit claim is untrue. rsync - as used by portage - always > transfers whole files, only. rsync is capable of transferring partial files. I can't vouch for how portage is using it, but both the rsync command line program and librsync can do partial file transfers. However, this is based on offsets from the start of the file, so appending to a file will result in the first part of the file being identical, but prepending will break rsync's algorithm. > > > But, this was endlessly debated before the decision was made. > > The decision was about removing the ChangeLogs from the git > repository. This was certainly the correct decision, because - > as you said - the ChangeLogs *can* be regenerated from the > git history and thus it makes no sense to modify/store them > redundantly. There were two decisions: https://projects.gentoo.org/council/meeting-logs/20141014-summary.txt "do we need to continue to create new ChangeLog entries once we're operating in git?" No. https://projects.gentoo.org/council/meeting-logs/20160410-summary.txt "The council does not require that ChangeLogs be generated or distributed through the rsync system. It is at the discretion of our infrastructure team whether or not this service continues." Accepted (4 yes, 1 no, 2 abstention) > > It probably should be a configurable option in repos.conf, but > > honestly, forced pushes are not something that should be considered a > > good practice. > > 1. portage shouldn't decide about practices of overlays. Hence the reason I suggested it should be a repos.conf option. > 2. also in the official gentoo repository force pushes happen >occassionally. Last occurrence was e.g. when undoing the >malevolent forced push ;) Sure, but that was a fast-forward from the last good commit, so it wouldn't require a force pull unless a user had done a force pull on the bad repo. > 3. git pull fails not only for forced pushes but also in several >other occassions; for instance, if your filesystem changed inodes >numbers (e.g. squash + overlayfs after a resquash+remount). If you're using squashfs git pull probably isn't the right solution for you. > 4. Even if the user made the mistake to edit a file, portage should >not just die on syncing. emerge --sync won't die in a situation like in general. Maybe it will if there is a merge conflict, but I don't think the correct default in this case should be to just wipe out the user's changes. I'm all for making that an option, however. -- Rich
