Re: [gentoo-dev] [RFC] GLEP 74: Full-tree verification using Manifest files

[Robin H. Johnson]

On Thu, Oct 26, 2017 at 10:12:25PM +0200, Michał Górny wrote:
> Hi, everyone.
> 
> After a week of hard work, I'd like to request your comments
> on the draft of GLEP 74. This GLEP aims to replace the old tree-signing
> GLEPs 58 and 60 with a superior implementation and more complete
> specification.
Edits inline, with trimming content.

Very strong proposal, I approve of this replacing my earlier work, as it learns
from the prototypes, failed implementation, and the intervening years of Gentoo
experience.

> 2. Alike the original Manifest2, the files should be split into two
>groups — files whose authenticity is critical, and those whose
>mismatch may be accepted in non-strict mode. The same classification
>should apply both to files listed in Manifests, and to stray files
>present only in the repository.
nit: s/Alike/Like/, or rewrite the sentence.

> Manifest file locations and nesting
> ---
> The ``Manifest`` file located in the root directory of the repository
> is called top-level Manifest, and it is used to perform the full-tree
> verification. In order to verify the authenticity, it must be signed
> using OpenPGP, using the armored cleartext format.
Are detached signatures also permitted (for all levels of Manifest)?
> 
> The Manifest files can also specify ``IGNORE`` entries to skip Manifest
> verification of subdirectories and/or files. Files and directories
> starting with a dot are always implicitly ignored. All files that
> are not ignored must be covered by at least one of the Manifests.
Do we need to keep that implicit ignore rule? Rather, convert it to being
always explicit.

There is only one such file in the rsync checkout presently:
metadata/.checksum-test-marker (see bug #572168, it is used to detect
mis-configured mirrors).

A SVN or Git repo might also have dot-named directories.

> All the files covered by a Manifest tree must reside on the same
> filesystem. It is an error to specify entries applying to files
> on another filesystem. If subdirectories of the Manifest tree reside
> on a different filesystem, they must be explicitly excluded
> via ``IGNORE``.
Distfiles aren't required to be in the same filesystem.

> New Manifest tags
> -
...
> ``IGNORE ``
>   Ignores a subdirectory or file from Manifest checks. If the specified
>   path is present, it and its contents are omitted from the Manifest
>   verification (always pass).
Should this be accepted even by strict-mode? Alternatively, should strict mode
require that other content is kept outside of the tree?

> Algorithm for full-tree verification
> 
...
> 2. Start at the top-level Manifest file. Verify its OpenPGP signature.
>Optionally verify the ``TIMESTAMP`` entry if present. Remove
>the top-level Manifest from the *present* set.
This spec does not state how the timestamp should be verified. 
Borrow from the original GLEP?

> 4. Process all ``IGNORE`` entries. Remove any paths matching them
>from the *present* set.
>
> 5. Collect all files covered by ``DATA``, ``MISC``, ``OPTIONAL``,
>``EBUILD`` and ``AUX`` entries into the *covered* set.
Clarification request: point out again in this section, that IGNORE entries are
prohibited from also matching any other entry. It is mentioned further up, but
a reminder is good.

> Checksum algorithms
> ---
> This section is informational only. Specifying the exact set
> of supported algorithms is outside the scope of this specification.
...
> The method of introducing new hashes is defined by GLEP 59 [#GLEP59]_.
> It is recommended that any new hashes are named after the Python
> ``hashlib`` module algorithm names, transformed into uppercase.
Would we ever consider algorithm parameters? Yes, outside of this spec, but 
checking anyway.

> Manifest compression
> 
...
> The specification permits uncompressed Manifests to exist alongside
> their compressed counterparts, and multiple compressed formats
> to coexist. If that is the case, the files must have the same
> uncompressed content and the specification is free to choose either
> of the files using the same base name.
GLEP61, for the transition period, required compressed & uncompressed Manifests
in the same directory to have identical content. Include mention of that here.

Saying that either can be used is a potential issue.

> Tree design
> ---
...

Add a minor header here, to say this is the end of the 'Tree design' section?
> In the independent model, each sub-Manifest file is independent
> of the parent Manifests. As a result, each of them needs to be signed
> and verified independently. However, the parent Manifests still need
> to list sub-Manifests (albeit without verification data) in order
> to detect removal or replacement of subdirectories. This has
> the following implications:
...

> File verification model
> ---
> 
> The verification model aims to

Re: [gentoo-dev] [RFC] GLEP 74: Full-tree verification using Manifest files

[Hanno Böck]

Hi,

On Thu, 26 Oct 2017 22:12:25 +0200
Michał Górny  wrote:

> After a week of hard work, I'd like to request your comments
> on the draft of GLEP 74. This GLEP aims to replace the old
> tree-signing GLEPs 58 and 60 with a superior implementation and more
> complete specification.

Thanks for working on this, it's really one of the biggest security
issues Gentoo has these days that need to be fixed.

I hope I'll find time to read it in detail, but by skimming through it
I noted that the downgrade attack prevention is kinda not very clear.
It says in the timestamp section "The package manager can use it  to
detect an outdated repository checkout." But it doesn't say how exactly.

Should a package manager reject a sync if it is too old? or not install
packages if a sync hasn't happened for some time? What is considered
"outdated"? I think that should be clarified how exactly it's supposed
to work.

-- 
Hanno Böck
https://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42

Re: [gentoo-dev] [RFC] GLEP 74: Full-tree verification using Manifest files

[Dean Stephens]

On 10/27/17 17:48, Hanno Böck wrote:
> Should a package manager reject a sync if it is too old? or not install
> packages if a sync hasn't happened for some time? What is considered
> "outdated"? I think that should be clarified how exactly it's supposed
> to work.
> 
If such a rejection is to be the default, an override option should be
required as part of the spec. There are use cases where using an "old"
repository would be necessary, even if only temporarily.

Re: [gentoo-dev] [RFC] GLEP 74: Full-tree verification using Manifest files

[Dean Stephens]

On 10/27/17 02:22, Michał Górny wrote:
> Yes. We can't technically distinguish intentional package removal by user 
> from malicious third party stripping them. This is something that a package 
> manager extension might handle but it doesn't belong in the spec.
> 
"Implementations may provide mechanisms for verifying partial
repositories or accepting repositories which could not be fully
verified, such mechanisms are outside the scope of this document."

Especially given: "The package manager may reject any package or even
the whole repository if it may refer to files for which the verification
failed."

Re: [gentoo-dev] [RFC] GLEP 74: Full-tree verification using Manifest files

[M. J. Everitt]

On 28/10/17 03:41, Dean Stephens wrote:
> On 10/27/17 17:48, Hanno Böck wrote:
>> Should a package manager reject a sync if it is too old? or not install
>> packages if a sync hasn't happened for some time? What is considered
>> "outdated"? I think that should be clarified how exactly it's supposed
>> to work.
>>
> If such a rejection is to be the default, an override option should be
> required as part of the spec. There are use cases where using an "old"
> repository would be necessary, even if only temporarily.
>
I_KNOW_WHAT_I_AM_DOING=1

:]




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [RFC] GLEP 74: Full-tree verification using Manifest files

[Allan Wegan]

On 28.10.2017 05:27, M. J. Everitt wrote:
> On 28/10/17 03:41, Dean Stephens wrote:
>> On 10/27/17 17:48, Hanno Böck wrote:
>>> Should a package manager reject a sync if it is too old? or not install
>>> packages if a sync hasn't happened for some time? What is considered
>>> "outdated"? I think that should be clarified how exactly it's supposed
>>> to work.
>>>
>> If such a rejection is to be the default, an override option should be
>> required as part of the spec. There are use cases where using an "old"
>> repository would be necessary, even if only temporarily.
>>
> I_KNOW_WHAT_I_AM_DOING=1
>
> :]

That is already reserved for disabling the signature checks :P

I would suggest --max-repository-age-days= with 
defaulting to as much days as the maximum update intervall of the
repository + 1.

But then the repository actually has to be newly signed at least once
each  days to prevent users from getting false positive replay
attack detection errors breaking their update process...



-- 
Allan Wegan

Jabber: allanwe...@ffnord.net
 OTR-Fingerprint: E4DCAA40 4859428E B3912896 F2498604 8CAA126F
Jabber: allanwe...@jabber.ccc.de
 OTR-Fingerprint: A1AAA1B9 C067F988 4A424D33 98343469 29164587
ICQ: 209459114
 OTR-Fingerprint: 71DE5B5E 67D6D758 A93BF1CE 7DA06625 205AC6EC



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: one hash to decide them all

[R0b0t1]

On Tue, Oct 24, 2017 at 9:40 PM, Robin H. Johnson  wrote:
> On Tue, Oct 24, 2017 at 11:33:39PM +0200, Allan Wegan wrote:
>> >> That is currently the case with portage, but not an inevitable
>> >> consequence of having 3 hash functions in the Manifest. Portage could
>> >> be made to check only one or two of them (even by default), giving
>> >> the tie-breaking ability to those who need it, and speeding up things
>> >> for those who don't.
>> > No, it can't. The specification (GLEP 59) requires it to check all
>> > hashes.
>>
>> Of course it can: The code of the specification just has to be changed
>> before changing the code of portage. The question is not whether it is
>> possible to make portage skip hash verification - but whether it is a
>> good idea to make it do that...
>>
>> I would not mind as long as the default is to always check all the
>> hashes and the option to disable it is properly named (like
>> "--disable-hash-verification" or something similar) and documented.
> At that point, and this is a serious proposal:
> The package manager shall decide which hashes to check, but is required
> to check at least one hash. The choice may be 'fastest', 'most secure',
> or any local factor.
>
> For local values of 'most secure' or 'fastest'.
>
> I wrote GLEP59, and specified at the time that all hashes should be
> checked, based on prior experience with hash implementation problems.
>
> Skipping them entirely is a bad idea, but only checking one of them is a
> reasonable suggestion.
>

The talk on gentoo-dev related to the security of hash functions still
has me confused as to why anyone would want to remove the
alternatives. Arguably this gives the setup most of its security.

I acknowledge Mr. Böck's observation that most programs only use one
hash. When I first encountered portage, I thought the use of multiple
hashes was a very novel and security conscious design choice. The cost
is negligible compared to the difficulty it adds to generating a
collision.

Of course, the difficulty of generating a collision that results in
usable code is still very high, but there was an interesting paper
that described how it could be easier than most people think. I am not
sure how to find it again. It was based on the principle that the
solution space for collisions that were valid was actually far denser
than the solution space of all collisions. One of the results of this
is that finding usefully malicious collisions is probably easier than
finding a collision in general, although there may be more non-useful
collisions overall.

> I retract my prior suggestion that there should be 3 hashes, as I
> realize a key statements in GLEP59 that were NEVER implemented:
>>> - Removal of depreciated checksums shall happen after no less than 18
>>>   months or one major Portage version cycle, whichever is greater.
>>> ...
>>> After the majority of Portage installations include SHA512 support:
>>> - Remove SHA256.
>
> This GLEP to update the GLEP59 specification should state the following:
> - The package manager or verification tool is required to verify at
>   least one hash, preferably the strongest supported hash.
> - Multiple hashes may be present for transitional periods.
> - SHA3 or BLAKE2B shall be added to the official Manifest2 hashes.
>


I'm still kind of confused as to why these changes are being made. Can
they be justified?

> For implementation:
> - Generation of WHIRLPOOL and SHA256 shall be disabled in the next
>   Portage minor release (as soon as possible)
> - Generation of the next choice of hash, SHA3 or BLAKE2B shall be
>   enabled in an upcoming minor Portage release (soon)
> - 18 months after the next GLEP is approved, SHA512 shall be dropped
>   (put the date into the Portage code so it happens automatically this
>   time, unlike SHA256 that should have been removed in 2010!).
>

This makes sense, but I would hope deprecation can be justified in a useful way.

Cheers,
 R0b0t1