[gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
Hi, everyone. I think the affairs have settled enough and I've finished filling in the pre-GLEP for REQUIRED_USE auto-enforcing. It's got all the algorithms, rationale and separated reference implementation. If there are no major concerns raised, I will soon start working on writing an optimized implementation for pkgcore/pkgcheck and integrating the verification algos with the CI. The pre-GLEP for review is here: https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse TIA. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
> On Sat, 08 Jul 2017, Michał Górny wrote: > The pre-GLEP for review is here: > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse On first glance: Section "Processing algorithm": | 2. Check whether the REQUIRED_USE constraint matches restrictions | set in #Restrictions on REQUIRED_USE format. If it does not, report | a REQUIRED_USE mismatch and abort. Why is this needed? This case should never occur if the REQUIRED_USE syntax doesn't allow it. Section "REQUIRED_USE verification algorithm": | * An any-of group (||) evaluates to true if at least one of the | items in it evaluates to true. | * An exactly-one-of group (^^) evaluates to true if exactly one of | the items in it evaluates to true, and all the remaining items | evaluate to false. | * An at-most-one-of group (??) evaluates to true if at most one of | the items in it evaluates to true. It should be added that any empty group (|| or ^^ or ??) evalutates to true, because that's what PMS specifies: https://projects.gentoo.org/pms/6/pms.html#x1-780008.2 Ulrich pgpuHyyNDQclx.pgp Description: PGP signature
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 8 Jul 2017 12:26:59 +0200 Ulrich Mueller wrote: > | * An any-of group (||) evaluates to true if at least one of the > | items in it evaluates to true. > | * An exactly-one-of group (^^) evaluates to true if exactly one of > | the items in it evaluates to true, and all the remaining items > | evaluate to false. > | * An at-most-one-of group (??) evaluates to true if at most one of > | the items in it evaluates to true. > > It should be added that any empty group (|| or ^^ or ??) evalutates to > true, because that's what PMS specifies: > https://projects.gentoo.org/pms/6/pms.html#x1-780008.2 A bit OT, but that is *definitely* counter intuitive. What's the rationale and usecase behind this ?
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 8 Jul 2017 13:49:56 +0200 Alexis Ballier wrote: > On Sat, 8 Jul 2017 12:26:59 +0200 > Ulrich Mueller wrote: > > | * An any-of group (||) evaluates to true if at least one of the > > | items in it evaluates to true. > > | * An exactly-one-of group (^^) evaluates to true if exactly one of > > | the items in it evaluates to true, and all the remaining items > > | evaluate to false. > > | * An at-most-one-of group (??) evaluates to true if at most one of > > | the items in it evaluates to true. > > > > It should be added that any empty group (|| or ^^ or ??) evalutates > > to true, because that's what PMS specifies: > > https://projects.gentoo.org/pms/6/pms.html#x1-780008.2 > > A bit OT, but that is *definitely* counter intuitive. What's the > rationale and usecase behind this ? Annoying special cases like || ( foo? ( ... ) bar? ( ... ) ) . The original reason was that old versions of Portage would simply remove unmet "flag? ( )" blocks internally. It was kept in EAPI 0 because stuff in the tree used it back then. -- Ciaran McCreesh
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 08 Jul 2017 11:43:39 +0200 Michał Górny wrote: > Hi, everyone. > > I think the affairs have settled enough and I've finished filling > in the pre-GLEP for REQUIRED_USE auto-enforcing. It's got all > the algorithms, rationale and separated reference implementation. > > If there are no major concerns raised, I will soon start working > on writing an optimized implementation for pkgcore/pkgcheck > and integrating the verification algos with the CI. > > The pre-GLEP for review is here: > > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse Constraint group reordering algorithm I really think we should only consider REQUIRED_USE with forced/masked useflags instantiated there. And ban (in repoman) REQUIRED_USE that contain some "False": "a? ( b )" with 'a' free and 'b' masked is perfectly ok now but it hides a serious problem in the package/profile. Instantiating this would give: "a? ( False )" and be an error just like we have depend.bad & co. This is independent of auto solving or not, it's already wrong. Reordering is a dangerous path as we've already seen since it can create unexpected loops for the solver. Working on instantiated REQUIRED_USE constraints would probably simplify quite a bit your GLEP too: you already have the "Verifying that the constraints do not alter immutable flags" part that roughly does the same thing as instantiating, except if you assume it's already true you can skip the reordering. Concept for transforming REQUIRED_USE into implications Ok, now I probably understand better the concept of common prefix. I'm definitely biased here, but I would feel better with a more recursive presentation of it. Assume we have 'validate(list of clauses)'; basically, the common prefix idea is that for an implication 'foo? ( consequences = list of clauses )' you first validate the consequences as if it were a REQUIRED_USE (so that the subtree rooted at foo is not self-conflicting) and then consider the whole thing as a clause. The idea would then be to have similar checks as to what you've written but working on trees (ASTs) instead of flattened clauses. This would avoid having to deal with unique identities (these would come for free) and IMHO would be easier to understand. I'm not sure how to do this though, I'll ping you when I have some idea. One big advantage of working on ASTs is that it becomes trivial to suggest a proper reordering. --- Restrictions on REQUIRED_USE format I still fail to see the point here. One can simply apply the rewriting you suggest below and be done with it. Rationale is not very convincing to me: - avoiding unpredictable results of automatic flag adjustments: A deterministic algorithm is, by definition, predictable. - improving readability of REQUIRED_USE constraints: No need for a restriction for that. If people want to shoot themselves in the foot, it is not a PMS problem. I see that like proposing death penalty for those who commit suicide :) - keeping the specification and implementation relatively simple: You already define everything for working without restriction. Plus, unlimited implication nesting has the same complexity. --- Do you have numbers on the checker run on all inputs from gentoo-x86 ? Since we're dealing with heuristics those are particularly important to validate we're not rejecting too many constructs. Alexis.
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 8 Jul 2017 13:01:39 +0100 Ciaran McCreesh wrote: > On Sat, 8 Jul 2017 13:49:56 +0200 > Alexis Ballier wrote: > > On Sat, 8 Jul 2017 12:26:59 +0200 > > Ulrich Mueller wrote: > > > | * An any-of group (||) evaluates to true if at least one of the > > > | items in it evaluates to true. > > > | * An exactly-one-of group (^^) evaluates to true if exactly one > > > of | the items in it evaluates to true, and all the remaining > > > items | evaluate to false. > > > | * An at-most-one-of group (??) evaluates to true if at most one > > > of | the items in it evaluates to true. > > > > > > It should be added that any empty group (|| or ^^ or ??) > > > evalutates to true, because that's what PMS specifies: > > > https://projects.gentoo.org/pms/6/pms.html#x1-780008.2 > > > > A bit OT, but that is *definitely* counter intuitive. What's the > > rationale and usecase behind this ? > > Annoying special cases like || ( foo? ( ... ) bar? ( ... ) ) . The > original reason was that old versions of Portage would simply remove > unmet "flag? ( )" blocks internally. It was kept in EAPI 0 because > stuff in the tree used it back then. > Wasn't REQUIRED_USE something completely new with no prior usage in EAPI 3 ?
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 8 Jul 2017 16:14:09 +0200 Alexis Ballier wrote: > On Sat, 8 Jul 2017 13:01:39 +0100 > Ciaran McCreesh wrote: > > On Sat, 8 Jul 2017 13:49:56 +0200 > > Alexis Ballier wrote: > > > On Sat, 8 Jul 2017 12:26:59 +0200 > > > Ulrich Mueller wrote: > > > > | * An any-of group (||) evaluates to true if at least one of > > > > the | items in it evaluates to true. > > > > | * An exactly-one-of group (^^) evaluates to true if exactly > > > > one of | the items in it evaluates to true, and all the > > > > remaining items | evaluate to false. > > > > | * An at-most-one-of group (??) evaluates to true if at most > > > > one of | the items in it evaluates to true. > > > > > > > > It should be added that any empty group (|| or ^^ or ??) > > > > evalutates to true, because that's what PMS specifies: > > > > https://projects.gentoo.org/pms/6/pms.html#x1-780008.2 > > > > > > A bit OT, but that is *definitely* counter intuitive. What's the > > > rationale and usecase behind this ? > > > > Annoying special cases like || ( foo? ( ... ) bar? ( ... ) ) . The > > original reason was that old versions of Portage would simply remove > > unmet "flag? ( )" blocks internally. It was kept in EAPI 0 because > > stuff in the tree used it back then. > > Wasn't REQUIRED_USE something completely new with no prior usage in > EAPI 3 ? Yes, but the spec defines dependency-like structures and their meanings once and consistently, rather than all over the place and inconsistently. As much as I hate the weird || ( use ? ( ) ) and empty block rules, it would be worse to have them apply in some situations but not others. -- Ciaran McCreesh
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 8 Jul 2017 15:23:39 +0100 Ciaran McCreesh wrote: > On Sat, 8 Jul 2017 16:14:09 +0200 > Alexis Ballier wrote: > > On Sat, 8 Jul 2017 13:01:39 +0100 > > Ciaran McCreesh wrote: > > > On Sat, 8 Jul 2017 13:49:56 +0200 > > > Alexis Ballier wrote: > > > > On Sat, 8 Jul 2017 12:26:59 +0200 > > > > Ulrich Mueller wrote: > > > > > | * An any-of group (||) evaluates to true if at least one of > > > > > the | items in it evaluates to true. > > > > > | * An exactly-one-of group (^^) evaluates to true if exactly > > > > > one of | the items in it evaluates to true, and all the > > > > > remaining items | evaluate to false. > > > > > | * An at-most-one-of group (??) evaluates to true if at most > > > > > one of | the items in it evaluates to true. > > > > > > > > > > It should be added that any empty group (|| or ^^ or ??) > > > > > evalutates to true, because that's what PMS specifies: > > > > > https://projects.gentoo.org/pms/6/pms.html#x1-780008.2 > > > > > > > > A bit OT, but that is *definitely* counter intuitive. What's the > > > > rationale and usecase behind this ? > > > > > > Annoying special cases like || ( foo? ( ... ) bar? ( ... ) ) . The > > > original reason was that old versions of Portage would simply > > > remove unmet "flag? ( )" blocks internally. It was kept in EAPI 0 > > > because stuff in the tree used it back then. > > > > Wasn't REQUIRED_USE something completely new with no prior usage in > > EAPI 3 ? > > Yes, but the spec defines dependency-like structures and their > meanings once and consistently, rather than all over the place and > inconsistently. > > As much as I hate the weird || ( use ? ( ) ) and empty block rules, it > would be worse to have them apply in some situations but not others. Indeed, makes sense. Would it also make sense to have some more logical meaning in a future EAPI ? I mean, in every context I've ever seen, applying a rule to the empty set is the neutral of that rule, so that it preserves associativity. That'd mean: || ( ) is false, && ( ) is true, ^^ ( ) is false, ?? ( ) is false. Alexis.
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 8 Jul 2017 16:39:29 +0200 Alexis Ballier wrote: > > As much as I hate the weird || ( use ? ( ) ) and empty block rules, > > it would be worse to have them apply in some situations but not > > others. > > Indeed, makes sense. Would it also make sense to have some more > logical meaning in a future EAPI ? I mean, in every context I've ever > seen, applying a rule to the empty set is the neutral of that rule, > so that it preserves associativity. > That'd mean: || ( ) is false, && ( ) is true, ^^ ( ) is false, ?? ( ) > is false. The sensible thing to do is ban it, and additionally ban use? ( ) inside || and ^^ (if we've not done that already...). -- Ciaran McCreesh
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On sob, 2017-07-08 at 18:58 +0100, Ciaran McCreesh wrote: > On Sat, 8 Jul 2017 16:39:29 +0200 > Alexis Ballier wrote: > > > As much as I hate the weird || ( use ? ( ) ) and empty block rules, > > > it would be worse to have them apply in some situations but not > > > others. > > > > Indeed, makes sense. Would it also make sense to have some more > > logical meaning in a future EAPI ? I mean, in every context I've ever > > seen, applying a rule to the empty set is the neutral of that rule, > > so that it preserves associativity. > > That'd mean: || ( ) is false, && ( ) is true, ^^ ( ) is false, ?? ( ) > > is false. > > The sensible thing to do is ban it, and additionally ban use? ( ) > inside || and ^^ (if we've not done that already...). > My GLEP bans the latter. As someone already pointed out, it didn't concern the case of empty groups, so I can either ban them or describe them. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On sob, 2017-07-08 at 12:26 +0200, Ulrich Mueller wrote: > > > > > > On Sat, 08 Jul 2017, Michał Górny wrote: > > The pre-GLEP for review is here: > > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse > > On first glance: > > Section "Processing algorithm": > > > 2. Check whether the REQUIRED_USE constraint matches restrictions > > set in #Restrictions on REQUIRED_USE format. If it does not, report > > a REQUIRED_USE mismatch and abort. > > Why is this needed? This case should never occur if the REQUIRED_USE > syntax doesn't allow it. The syntax is restricted from the one allowed by the PMS. The algorithm doesn't cover the weird deep nesting cases. Unless we want to retroactively change PMS to disallow them as well, the spec needs to clearly establish the acceptable input for the algorithm presented. Of course, you are free to omit this step if you implement algorithm that covers all the PMS-defined cases. However, this goes beyond the basic scope which this GLEP is concerned about. > Section "REQUIRED_USE verification algorithm": > > > * An any-of group (||) evaluates to true if at least one of the > > items in it evaluates to true. > > * An exactly-one-of group (^^) evaluates to true if exactly one of > > the items in it evaluates to true, and all the remaining items > > evaluate to false. > > * An at-most-one-of group (??) evaluates to true if at most one of > > the items in it evaluates to true. > > It should be added that any empty group (|| or ^^ or ??) evalutates to > true, because that's what PMS specifies: > https://projects.gentoo.org/pms/6/pms.html#x1-780008.2 > Indeed you are correct. However, considering what Ciaran wrote it might be a reasonable alternative to ban them as well. I'm going to wait a while and see what others say. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On sob, 2017-07-08 at 16:12 +0200, Alexis Ballier wrote: > On Sat, 08 Jul 2017 11:43:39 +0200 > Michał Górny wrote: > > > Hi, everyone. > > > > I think the affairs have settled enough and I've finished filling > > in the pre-GLEP for REQUIRED_USE auto-enforcing. It's got all > > the algorithms, rationale and separated reference implementation. > > > > If there are no major concerns raised, I will soon start working > > on writing an optimized implementation for pkgcore/pkgcheck > > and integrating the verification algos with the CI. > > > > The pre-GLEP for review is here: > > > > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse > > > Constraint group reordering algorithm > > I really think we should only consider REQUIRED_USE with forced/masked > useflags instantiated there. And ban (in repoman) REQUIRED_USE that > contain some "False": "a? ( b )" with 'a' free and 'b' masked is > perfectly ok now but it hides a serious problem in the package/profile. > Instantiating this would give: "a? ( False )" and be an error > just like we have depend.bad & co. This is independent of auto > solving or not, it's already wrong. As I've already explained you multiple times, this obtains *exactly the same* effect. However, it's much simpler when it's done like this because it makes it possible to reuse the already defined algorithms instead of having to precisely define how to preprocess REQUIRED_USE for this and cover all the corner cases. > Reordering is a dangerous path as we've already seen since it can > create unexpected loops for the solver. Freeform reordering is dangerous, and I've removed that already. Reordering restricted to immutables can not cause any issues that any other solution wouldn't cause. > Working on instantiated REQUIRED_USE constraints would probably > simplify quite a bit your GLEP too: you already have the "Verifying > that the constraints do not alter immutable flags" part that roughly > does the same thing as instantiating, except if you assume it's already > true you can skip the reordering. Except that the reordering can be described in 2 points, and so can be the immutability verification. Please prove that you can provide a simpler explanation that doesn't fail in any of the corner cases. > Concept for transforming REQUIRED_USE into implications > > Ok, now I probably understand better the concept of common prefix. I'm > definitely biased here, but I would feel better with a more recursive > presentation of it. Assume we have 'validate(list of clauses)'; > basically, the common prefix idea is that for an implication 'foo? > ( consequences = list of clauses )' you first validate the consequences > as if it were a REQUIRED_USE (so that the subtree rooted at foo is > not self-conflicting) and then consider the whole thing as a clause. > The idea would then be to have similar checks as to what you've written > but working on trees (ASTs) instead of flattened clauses. This would > avoid having to deal with unique identities (these would come for free) > and IMHO would be easier to understand. > I'm not sure how to do this though, I'll ping you when I have some idea. Well, the problem of common prefix is quite complex, and I'm not even sure if it's really worth more consideration. After all, we're prettych much talking about doing: a? ( !a ... ) which has extremely low usability and even lower likeness of occurring. > One big advantage of working on ASTs is that it becomes trivial to > suggest a proper reordering. Reordering is never a trivial problem. Unless I'm missing something, it is technically possible that a 'reordering' will actually require a sub- item being moved out of the containing group. And to be honest, I find the output of the verification script in this regard quite useful. That is, it saying 'X affects Y, so it needs to go before it' is quite clear to me. I don't think most developers would actually need to script to pinpoint a specific location for every single constraint. > --- > > Restrictions on REQUIRED_USE format > > I still fail to see the point here. One can simply apply the rewriting > you suggest below and be done with it. Rationale is not very convincing > to me: > > - avoiding unpredictable results of automatic flag adjustments: > A deterministic algorithm is, by definition, predictable. s/unpredictable/surprising/? The goal is for it do something that the developer *not reading the spec* could reasonably predict happening. > - improving readability of REQUIRED_USE constraints: > No need for a restriction for that. If people want to shoot > themselves in the foot, it is not a PMS problem. I see that > like proposing death penalty for those who commit suicide :) This is not PMS. This is a GLEP which serves both the purpose of a technical specification with extended rationale and a policy document. > - keeping the specification and implementation relatively simple: > You already define everythi
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
> On Sat, 08 Jul 2017, Michał Górny wrote: > On sob, 2017-07-08 at 12:26 +0200, Ulrich Mueller wrote: >> Section "Processing algorithm": >> >> > 2. Check whether the REQUIRED_USE constraint matches restrictions >> > set in #Restrictions on REQUIRED_USE format. If it does not, report >> > a REQUIRED_USE mismatch and abort. >> >> Why is this needed? This case should never occur if the REQUIRED_USE >> syntax doesn't allow it. > The syntax is restricted from the one allowed by the PMS. The > algorithm doesn't cover the weird deep nesting cases. Unless we want > to retroactively change PMS to disallow them as well, the spec needs > to clearly establish the acceptable input for the algorithm > presented. Sorry, but that makes no sense. Why would we introduce automatic solving with the next EAPI, but at the same time not restrict REQUIRED_USE syntax to the one the solver can handle? Ulrich pgp7n76dDaKcN.pgp Description: PGP signature
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
> On Sat, 8 Jul 2017, Ciaran McCreesh wrote: > On Sat, 8 Jul 2017 16:39:29 +0200 > Alexis Ballier wrote: >> Indeed, makes sense. Would it also make sense to have some more >> logical meaning in a future EAPI ? I mean, in every context I've ever >> seen, applying a rule to the empty set is the neutral of that rule, >> so that it preserves associativity. >> That'd mean: || ( ) is false, && ( ) is true, ^^ ( ) is false, I have no strong opinion about this. Is it worth the effort of changing the spec? >> ?? ( ) is false. I think ?? ( ) aka at-most-one-of should be true if empty. > The sensible thing to do is ban it, and additionally ban use? ( ) > inside || and ^^ (if we've not done that already...). Right. We have to check if this will break any eclass generated dependencies, though. Ulrich pgp1k8IEIi3sC.pgp Description: PGP signature
[gentoo-dev] [PATCH] *.eclass: Include GNOME2_ECLASS_ICONS condition in postrm as well
The original GNOME2_ECLASS_ICONS patch has moved the condition from
gnome2_icon_cache_update to postinst phases of functions using
the preinst/postinst logic but accidentally omitted postrm. Include it
there as well to restore the old behavior.
---
eclass/gnome2.eclass| 4 +++-
eclass/kde4-base.eclass | 4 +++-
eclass/kde5.eclass | 4 +++-
eclass/xfconf.eclass| 4 +++-
4 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/eclass/gnome2.eclass b/eclass/gnome2.eclass
index feb6301221ea..cb233e747605 100644
--- a/eclass/gnome2.eclass
+++ b/eclass/gnome2.eclass
@@ -359,7 +359,9 @@ gnome2_pkg_postinst() {
# Handle scrollkeeper, GSettings, Icons, desktop and mime database updates.
gnome2_pkg_postrm() {
xdg_pkg_postrm
- gnome2_icon_cache_update
+ if [[ -n ${GNOME2_ECLASS_ICONS} ]]; then
+ gnome2_icon_cache_update
+ fi
gnome2_schemas_update
gnome2_scrollkeeper_update
diff --git a/eclass/kde4-base.eclass b/eclass/kde4-base.eclass
index bbdcf8db20f4..494e79518da9 100644
--- a/eclass/kde4-base.eclass
+++ b/eclass/kde4-base.eclass
@@ -936,7 +936,9 @@ kde4-base_pkg_postinst() {
kde4-base_pkg_postrm() {
debug-print-function ${FUNCNAME} "$@"
- gnome2_icon_cache_update
+ if [[ -n ${GNOME2_ECLASS_ICONS} ]]; then
+ gnome2_icon_cache_update
+ fi
fdo-mime_desktop_database_update
fdo-mime_mime_database_update
buildsycoca
diff --git a/eclass/kde5.eclass b/eclass/kde5.eclass
index 3b6810526a0a..7843f5eace78 100644
--- a/eclass/kde5.eclass
+++ b/eclass/kde5.eclass
@@ -748,7 +748,9 @@ kde5_pkg_postinst() {
kde5_pkg_postrm() {
debug-print-function ${FUNCNAME} "$@"
- gnome2_icon_cache_update
+ if [[ -n ${GNOME2_ECLASS_ICONS} ]]; then
+ gnome2_icon_cache_update
+ fi
xdg_pkg_postrm
}
diff --git a/eclass/xfconf.eclass b/eclass/xfconf.eclass
index b91d0fe1eda3..4cbcb8fa9a4e 100644
--- a/eclass/xfconf.eclass
+++ b/eclass/xfconf.eclass
@@ -151,5 +151,7 @@ xfconf_pkg_postrm() {
debug-print-function ${FUNCNAME} "$@"
fdo-mime_desktop_database_update
fdo-mime_mime_database_update
- gnome2_icon_cache_update
+ if [[ -n ${GNOME2_ECLASS_ICONS} ]]; then
+ gnome2_icon_cache_update
+ fi
}
--
2.13.2
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 8 Jul 2017 21:05:57 +0200 Ulrich Mueller wrote: > > On Sat, 8 Jul 2017, Ciaran McCreesh wrote: > > > On Sat, 8 Jul 2017 16:39:29 +0200 > > Alexis Ballier wrote: > >> Indeed, makes sense. Would it also make sense to have some more > >> logical meaning in a future EAPI ? I mean, in every context I've > >> ever seen, applying a rule to the empty set is the neutral of that > >> rule, so that it preserves associativity. > >> That'd mean: || ( ) is false, && ( ) is true, ^^ ( ) is false, > > I have no strong opinion about this. Is it worth the effort of > changing the spec? > > >> ?? ( ) is false. > > I think ?? ( ) aka at-most-one-of should be true if empty. Maybe; this one is annoying I think since it is not associative per definition: ?? ( true ?? ( false false ) ) -> ?? ( true true ) -> false ?? ( ?? ( true false ) false ) -> ?? ( true false) -> true > > The sensible thing to do is ban it, and additionally ban use? ( ) > > inside || and ^^ (if we've not done that already...). > > Right. We have to check if this will break any eclass generated > dependencies, though. That's probably the best course of action indeed. Alexis.
[gentoo-dev] [PATCH] gnome2*.eclass: Move the preinst conditional out of gnome2_schemas_update
Move the GNOME2_ECLASS_GLIB_SCHEMAS conditional from
gnome2_schemas_update straight into the implementation of gnome2.eclass
postinst/postrm.
This variable is set in preinst to indicate whether any files were
installed. However, the updater itself does not use the list in any way
and updates all the schemas anyway.
Therefore, avoid requiring the ebuilds to explicitly define
preinst/postinst when it is known that the package installs schemas,
and instead let gnome2_schemas_update called in postinst/postrm update
schemas unconditionally.
---
eclass/gnome2-utils.eclass | 5 -
eclass/gnome2.eclass | 8 ++--
2 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/eclass/gnome2-utils.eclass b/eclass/gnome2-utils.eclass
index a89b9885c391..9b4296c11fad 100644
--- a/eclass/gnome2-utils.eclass
+++ b/eclass/gnome2-utils.eclass
@@ -381,11 +381,6 @@ gnome2_schemas_update() {
return
fi
- if [[ -z ${GNOME2_ECLASS_GLIB_SCHEMAS} ]]; then
- debug-print "No GSettings schemas to update"
- return
- fi
-
ebegin "Updating GSettings schemas"
${updater} --allow-any-name "$@"
"${EROOT%/}/usr/share/glib-2.0/schemas" &>/dev/null
eend $?
diff --git a/eclass/gnome2.eclass b/eclass/gnome2.eclass
index cb233e747605..d2b45ad560b3 100644
--- a/eclass/gnome2.eclass
+++ b/eclass/gnome2.eclass
@@ -340,7 +340,9 @@ gnome2_pkg_postinst() {
if [[ -n ${GNOME2_ECLASS_ICONS} ]]; then
gnome2_icon_cache_update
fi
- gnome2_schemas_update
+ if [[ -z ${GNOME2_ECLASS_GLIB_SCHEMAS} ]]; then
+ gnome2_schemas_update
+ fi
gnome2_scrollkeeper_update
gnome2_gdk_pixbuf_update
@@ -362,7 +364,9 @@ gnome2_pkg_postrm() {
if [[ -n ${GNOME2_ECLASS_ICONS} ]]; then
gnome2_icon_cache_update
fi
- gnome2_schemas_update
+ if [[ -z ${GNOME2_ECLASS_GLIB_SCHEMAS} ]]; then
+ gnome2_schemas_update
+ fi
gnome2_scrollkeeper_update
if [[ ${#GNOME2_ECLASS_GIO_MODULES[@]} -gt 0 ]]; then
--
2.13.2
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On Sat, 08 Jul 2017 20:44:24 +0200 Michał Górny wrote: > On sob, 2017-07-08 at 16:12 +0200, Alexis Ballier wrote: > > On Sat, 08 Jul 2017 11:43:39 +0200 > > Michał Górny wrote: > > > > > Hi, everyone. > > > > > > I think the affairs have settled enough and I've finished filling > > > in the pre-GLEP for REQUIRED_USE auto-enforcing. It's got all > > > the algorithms, rationale and separated reference implementation. > > > > > > If there are no major concerns raised, I will soon start working > > > on writing an optimized implementation for pkgcore/pkgcheck > > > and integrating the verification algos with the CI. > > > > > > The pre-GLEP for review is here: > > > > > > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse > > > > > > Constraint group reordering algorithm > > > > I really think we should only consider REQUIRED_USE with > > forced/masked useflags instantiated there. And ban (in repoman) > > REQUIRED_USE that contain some "False": "a? ( b )" with 'a' free > > and 'b' masked is perfectly ok now but it hides a serious problem > > in the package/profile. Instantiating this would give: "a? ( False > > )" and be an error just like we have depend.bad & co. This is > > independent of auto solving or not, it's already wrong. > > As I've already explained you multiple times, this obtains *exactly > the same* effect. However, it's much simpler when it's done like this > because it makes it possible to reuse the already defined algorithms > instead of having to precisely define how to preprocess REQUIRED_USE > for this and cover all the corner cases. Simpler??? I don't think so. What I wrote clearly pinpoints that: When you'll write the algorithm for "Verifying that the constraints do not alter immutable flags" you'll notice this is exactly that and can be put as a preprocessing step and then you can drop all the corner cases considerations for immutable flags. I never understood why you're insisting that much on immutables: they're really not natural, not simple, not standard, and carrying them all over seems to be a burden to me. > > Reordering is a dangerous path as we've already seen since it can > > create unexpected loops for the solver. > > Freeform reordering is dangerous, and I've removed that already. > Reordering restricted to immutables can not cause any issues that any > other solution wouldn't cause. You're very likely right there. Any proof? Esp. any proof that the checker still guarantees the existence of a solution in all cases? I'm not asking for a formal proof, but simply a bit more details than just an assertion saying it's fine. > > Working on instantiated REQUIRED_USE constraints would probably > > simplify quite a bit your GLEP too: you already have the "Verifying > > that the constraints do not alter immutable flags" part that roughly > > does the same thing as instantiating, except if you assume it's > > already true you can skip the reordering. > > Except that the reordering can be described in 2 points, and so can be > the immutability verification. Please prove that you can provide > a simpler explanation that doesn't fail in any of the corner cases. Except reordering is an invention and immutable checking is simply applying boolean logic rules to your implication and check that no "False" can appear. You can simply start by applying boolean logic and forget about reordering. > > Concept for transforming REQUIRED_USE into implications > > > > Ok, now I probably understand better the concept of common prefix. > > I'm definitely biased here, but I would feel better with a more > > recursive presentation of it. Assume we have 'validate(list of > > clauses)'; basically, the common prefix idea is that for an > > implication 'foo? ( consequences = list of clauses )' you first > > validate the consequences as if it were a REQUIRED_USE (so that the > > subtree rooted at foo is not self-conflicting) and then consider > > the whole thing as a clause. The idea would then be to have similar > > checks as to what you've written but working on trees (ASTs) > > instead of flattened clauses. This would avoid having to deal with > > unique identities (these would come for free) and IMHO would be > > easier to understand. I'm not sure how to do this though, I'll ping > > you when I have some idea. > > Well, the problem of common prefix is quite complex, and I'm not even > sure if it's really worth more consideration. After all, we're > prettych much talking about doing: > > a? ( !a ... ) > > which has extremely low usability and even lower likeness of > occurring. Hmm. I think you came up with more valid cases. Can't remember a precise one though. > > One big advantage of working on ASTs is that it becomes trivial to > > suggest a proper reordering. > > Reordering is never a trivial problem. Unless I'm missing something, > it is technically possible that a 'reordering' will actually require > a sub- item being moved out of the containing group. Not
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On sob, 2017-07-08 at 20:58 +0200, Ulrich Mueller wrote: > > > > > > On Sat, 08 Jul 2017, Michał Górny wrote: > > On sob, 2017-07-08 at 12:26 +0200, Ulrich Mueller wrote: > > > Section "Processing algorithm": > > > > > > > 2. Check whether the REQUIRED_USE constraint matches restrictions > > > > set in #Restrictions on REQUIRED_USE format. If it does not, report > > > > a REQUIRED_USE mismatch and abort. > > > > > > Why is this needed? This case should never occur if the REQUIRED_USE > > > syntax doesn't allow it. > > The syntax is restricted from the one allowed by the PMS. The > > algorithm doesn't cover the weird deep nesting cases. Unless we want > > to retroactively change PMS to disallow them as well, the spec needs > > to clearly establish the acceptable input for the algorithm > > presented. > > Sorry, but that makes no sense. Why would we introduce automatic > solving with the next EAPI, but at the same time not restrict > REQUIRED_USE syntax to the one the solver can handle? > Nobody said anything about the next EAPI. The GLEP doesn't say a word about introducing it in a future EAPI. We're adding this as an optional (default off) FEATURE into Portage and we'll see how it works. As far as I'm concerned, we can enable it for all EAPIs without touching PMS at all. In fact, the GLEP points out that the PMS does not specifically define how PMs are supposed to deal with ensuring that REQUIRED_USE is satisfied. Failing with poor error messages is just established historical Portage behavior. But if we get good test results and majority agreement, I see no reason why we couldn't start enabling it by default and eventually relying on it. After all, it wouldn't be the first non-PMS extension that we accept for user convenience yet do not require strictly. Of course, if you think it should be made obligatory or at least accounted for in a future EAPI, I have no problem with that. Ciaran might, however. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On sob, 2017-07-08 at 22:34 +0200, Alexis Ballier wrote: > On Sat, 08 Jul 2017 20:44:24 +0200 > Michał Górny wrote: > > > On sob, 2017-07-08 at 16:12 +0200, Alexis Ballier wrote: > > > On Sat, 08 Jul 2017 11:43:39 +0200 > > > Michał Górny wrote: > > > > > > > Hi, everyone. > > > > > > > > I think the affairs have settled enough and I've finished filling > > > > in the pre-GLEP for REQUIRED_USE auto-enforcing. It's got all > > > > the algorithms, rationale and separated reference implementation. > > > > > > > > If there are no major concerns raised, I will soon start working > > > > on writing an optimized implementation for pkgcore/pkgcheck > > > > and integrating the verification algos with the CI. > > > > > > > > The pre-GLEP for review is here: > > > > > > > > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse > > > > > > > > > Constraint group reordering algorithm > > > > > > I really think we should only consider REQUIRED_USE with > > > forced/masked useflags instantiated there. And ban (in repoman) > > > REQUIRED_USE that contain some "False": "a? ( b )" with 'a' free > > > and 'b' masked is perfectly ok now but it hides a serious problem > > > in the package/profile. Instantiating this would give: "a? ( False > > > )" and be an error just like we have depend.bad & co. This is > > > independent of auto solving or not, it's already wrong. > > > > As I've already explained you multiple times, this obtains *exactly > > the same* effect. However, it's much simpler when it's done like this > > because it makes it possible to reuse the already defined algorithms > > instead of having to precisely define how to preprocess REQUIRED_USE > > for this and cover all the corner cases. > > Simpler??? I don't think so. What I wrote clearly pinpoints that: > When you'll write the algorithm for "Verifying that the constraints do > not alter immutable flags" you'll notice this is exactly that and can > be put as a preprocessing step and then you can drop all the corner > cases considerations for immutable flags. I never understood why you're > insisting that much on immutables: they're really not natural, not > simple, not standard, and carrying them all over seems to be a burden > to me. I wrote the algorithms, and they're simple. This specific check is the combination of three simple steps: a. reordering the groups based on immutables, b. transforming the AST into flat form, c. verifying each flat constraint. The first step is trivial -- it's basically 'move true to front, false to back'. The second step is more complex but it's needed anyway, and quite well-defined, especially with the assumption that all the groups always have at least one flag inside. The third step is trivial again because it's just checking the conditions and constraints against a list. The alternative to reordering is altering the groups. Altering means we need to have separate logic for every type of group while sorting works the same in all of them. Altering means we need to explicitly special case forcing 1 and >1 items, and masking all items, for each group. Again, sorting does not need to be concerned about that because the check following it (also trivial) will catch it anyway. Of course, you could say that you will get a little better error message, like 'all flags inside || are masked' instead of '!b -> a' will alter immutable flag. But that's purely an implementation detail. It's not worth making the reference algorithms longer. > > > Reordering is a dangerous path as we've already seen since it can > > > create unexpected loops for the solver. > > > > Freeform reordering is dangerous, and I've removed that already. > > Reordering restricted to immutables can not cause any issues that any > > other solution wouldn't cause. > > You're very likely right there. Any proof? Esp. any proof that the > checker still guarantees the existence of a solution in all cases? > I'm not asking for a formal proof, but simply a bit more details than > just an assertion saying it's fine. The case for checker is just the same as with any other kind of immutability processing. We need to run the reordering, transform and verification separately for every possible combination of immutable flags. The reordering explicitly alters the results of the transform, and with the trivial implication form of the flattened constraints the verification stage checks will find any problems that may arise from it, just like they would find any problem from doing a similar thing verbatim. > > > > Working on instantiated REQUIRED_USE constraints would probably > > > simplify quite a bit your GLEP too: you already have the "Verifying > > > that the constraints do not alter immutable flags" part that roughly > > > does the same thing as instantiating, except if you assume it's > > > already true you can skip the reordering. > > > > Except that the reordering can be described in 2 points, and so can be > > the immutability verification. P
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On 07/08/2017 11:31 PM, Michał Górny wrote: > Nobody said anything about the next EAPI. The GLEP doesn't say a word > about introducing it in a future EAPI. > > We're adding this as an optional (default off) FEATURE into Portage > and we'll see how it works. As far as I'm concerned, we can enable it > for all EAPIs without touching PMS at all. With that in mind, does it really need a GLEP? Isn't this something that can be done within the package manager as a feature anyways without mandating changes? If anything it seems like it'd be an informational GLEP and not a standards track if going down that route. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Need GitHub snapshot hash verification failure samples
For anyone interested in such, I opened a feature request bug for allowing use of sets in profile packages. https://bugs.gentoo.org/show_bug.cgi?id=624300 -- William L. Thomson Jr. pgp04RTMoMwAV.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On sob, 2017-07-08 at 23:56 +0200, Michał Górny wrote: > On sob, 2017-07-08 at 22:34 +0200, Alexis Ballier wrote: > > Unless I'm missing something, rationale seems more about cases rejected > > by the restricted syntax. Numbers I'm talking about is the # of rejected > > constraints vs accepted (and assumed solvable) ones. > > I'll crunch some fresh numbers based on today's repository. Ebuild counts follow. count % of requse % of total Total ebuilds:38344 --- -- Ebuilds w/ REQUIRED_USE: 9945 --- -- Good REQUIRED_USE: 9727 97.8% 25.37% Invalid syntax: 160.2%0.04% Conflicting enforcements:490.5%0.13% Requiring >1 pass: 1531.5%0.40% So realstically saying, right now we're talking about 'rejecting' 0.04% of ebuilds, conditionally failing for 0.13% of them (i.e. with bad flag combinations) and conditionally requiring >1 iteration for 0.4% of them. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Sets vs Meta ebuilds
For anyone interested in such, I opened a feature request bug for allowing use of sets in profile packages. https://bugs.gentoo.org/show_bug.cgi?id=624300 P.S. Miss posted on wrong thread... thus duplicate, sorry! -- William L. Thomson Jr. pgpkQZ6BpgeJj.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On 07/08/2017 02:43 AM, Michał Górny wrote: > Hi, everyone. > > I think the affairs have settled enough and I've finished filling > in the pre-GLEP for REQUIRED_USE auto-enforcing. It's got all > the algorithms, rationale and separated reference implementation. > > If there are no major concerns raised, I will soon start working > on writing an optimized implementation for pkgcore/pkgcheck > and integrating the verification algos with the CI. > > The pre-GLEP for review is here: > > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse > > TIA. > This has grown quite a bit since first recommended! Great job so far. Forgive me if I missed something, but wouldn't it be helpful to the user to let them know when automatically choosing for them? A single line in a logfile, einfo output, whatever, would be useful for people wondering how certain packages got pulled in. Users will continue to get errors if the constraints aren't met (or are wrong), but where will information go that indicates the automatic solver's choice? You and I can read an ebuild and guess from the dep spec, but what will a user look at? I searched the GLEP page for "log", "einfo", and "output" with no results. If I've missed something please let me know. Thanks for the work that's been put into this so far. ~zlg -- Daniel Campbell - Gentoo Developer OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net fpr: AE03 9064 AE00 053C 270C 1DE4 6F7A 9091 1EA0 55D6 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On sob, 2017-07-08 at 15:21 -0700, Daniel Campbell wrote: > On 07/08/2017 02:43 AM, Michał Górny wrote: > > Hi, everyone. > > > > I think the affairs have settled enough and I've finished filling > > in the pre-GLEP for REQUIRED_USE auto-enforcing. It's got all > > the algorithms, rationale and separated reference implementation. > > > > If there are no major concerns raised, I will soon start working > > on writing an optimized implementation for pkgcore/pkgcheck > > and integrating the verification algos with the CI. > > > > The pre-GLEP for review is here: > > > > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse > > > > TIA. > > > > This has grown quite a bit since first recommended! Great job so far. > Forgive me if I missed something, but wouldn't it be helpful to the user > to let them know when automatically choosing for them? A single line in > a logfile, einfo output, whatever, would be useful for people wondering > how certain packages got pulled in. Users will continue to get errors if > the constraints aren't met (or are wrong), but where will information go > that indicates the automatic solver's choice? You and I can read an > ebuild and guess from the dep spec, but what will a user look at? > > I searched the GLEP page for "log", "einfo", and "output" with no > results. If I've missed something please let me know. > > Thanks for the work that's been put into this so far. > Indeed I have entirely skipped the user output problem, and left it purely for package manager's design choice. Do you really feel like we need to explicitly specify it? I think it's best if package manager authors decide how to best fit it into whatever output the PMs already have. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Fri, Jul 7, 2017 at 10:21 PM, Michael Palimaka wrote: > > Bug #272488[0] proposed a PROPERTIES="set" feature to combine the power > of sets with the flexibility of ebuilds. > > 1: https://bugs.gentoo.org/show_bug.cgi?id=272488 > What do sets get us that packages do not? Why not move the other direction and just have packages instead of sets? One issue I see with sets is that as far as I can tell they aren't specified in PMS at all, so they can't go into the tree at all, and not all package managers may support them in the same way. Certainly this could be standardized, but I'm not sure what they actually get us. -- Rich
Re: [gentoo-dev] Pre-GLEP RFC: Automated enforcing of REQUIRED_USE constraints
On 07/08/2017 03:29 PM, Michał Górny wrote: > On sob, 2017-07-08 at 15:21 -0700, Daniel Campbell wrote: >> On 07/08/2017 02:43 AM, Michał Górny wrote: >>> Hi, everyone. >>> >>> I think the affairs have settled enough and I've finished filling >>> in the pre-GLEP for REQUIRED_USE auto-enforcing. It's got all >>> the algorithms, rationale and separated reference implementation. >>> >>> If there are no major concerns raised, I will soon start working >>> on writing an optimized implementation for pkgcore/pkgcheck >>> and integrating the verification algos with the CI. >>> >>> The pre-GLEP for review is here: >>> >>> https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse >>> >>> TIA. >>> >> >> This has grown quite a bit since first recommended! Great job so far. >> Forgive me if I missed something, but wouldn't it be helpful to the user >> to let them know when automatically choosing for them? A single line in >> a logfile, einfo output, whatever, would be useful for people wondering >> how certain packages got pulled in. Users will continue to get errors if >> the constraints aren't met (or are wrong), but where will information go >> that indicates the automatic solver's choice? You and I can read an >> ebuild and guess from the dep spec, but what will a user look at? >> >> I searched the GLEP page for "log", "einfo", and "output" with no >> results. If I've missed something please let me know. >> >> Thanks for the work that's been put into this so far. >> > > Indeed I have entirely skipped the user output problem, and left it > purely for package manager's design choice. Do you really feel like we > need to explicitly specify it? I think it's best if package manager > authors decide how to best fit it into whatever output the PMs already > have. > > I just considered it helpful, that's all. I hadn't considered the PMS vs. PM devs perspective. Do we plan to support some way for users to get such output from Portage? Thanks for clarifying. It does make more sense to leave it to PM devs. -- Daniel Campbell - Gentoo Developer OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net fpr: AE03 9064 AE00 053C 270C 1DE4 6F7A 9091 1EA0 55D6 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, 8 Jul 2017 18:34:55 -0400 Rich Freeman wrote: > > What do sets get us that packages do not? Why not move the other > direction and just have packages instead of sets? The blog entry I provided a link to I think made the best case example of usage of sets and their benefits. https://makuro.wordpress.com/2010/12/12/intro-to-portage-sets/ The biggest advantage is ability to re-emerge all without additional steps or arguments. Simple emerge @my_set just like emerge world, etc. Even more useful you can remove a set directly, without depclean. If you remove a meta package, you have to run depclean to remove the actual packages in the meta ebuild. Sets save you from this step. Sets are also used for package rebuilds, like x11-module-rebuild, live-rebuild, and others. > One issue I see with sets is that as far as I can tell they aren't > specified in PMS at all, so they can't go into the tree at all, and > not all package managers may support them in the same way. Certainly > this could be standardized, but I'm not sure what they actually get > us. world and system are sets we all have. Not sure about PMS. It is something portage has supported for some time. You likely have many sets already on your system emerge --list-sets https://wiki.gentoo.org/wiki/World_set_(Portage) https://wiki.gentoo.org/wiki/System_set_(Portage) https://wiki.gentoo.org/wiki//etc/portage/sets https://dev.gentoo.org/~zmedico/portage/doc/ch02s02.html -- William L. Thomson Jr. pgpwHyLbUNjFu.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, Jul 8, 2017 at 7:09 PM, William L. Thomson Jr. wrote: > On Sat, 8 Jul 2017 18:34:55 -0400 > Rich Freeman wrote: >> >> What do sets get us that packages do not? Why not move the other >> direction and just have packages instead of sets? > > The blog entry I provided a link to I think made the best case example > of usage of sets and their benefits. > https://makuro.wordpress.com/2010/12/12/intro-to-portage-sets/ > > The biggest advantage is ability to re-emerge all without additional > steps or arguments. Simple emerge @my_set just like emerge world, etc. > Even more useful you can remove a set directly, without depclean. I don't see why a package manager couldn't offer the same functionality for a meta package. As was pointed out the set behavior for unmerging isn't always desirable. > > world and system are sets we all have. Not sure about PMS. It is > something portage has supported for some time. You likely have many > sets already on your system Certainly. You just can't depend on them and so on without having them in PMS, because portage isn't the only package manager we "support." It just strikes me that we're probably better off picking one way of doing this and putting lots of support behind it, versus having two ways of doing this and some features work with one but not the other. Of the two meta packages seem like they're the most generic. -- Rich
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, Jul 8, 2017 at 4:09 PM, William L. Thomson Jr. wrote: > Sets are also used for package rebuilds, like x11-module-rebuild, > live-rebuild, and others. Usually there are better ways to trigger rebuilds. For example, slot operator dependencies for rebuilds due to subslot changes, and --newuse for USE changes. For live-rebuild, it would be much nicer to have a framework that automatically triggers rebuilds when upstream changes are detected, like smart-live-rebuild. We can add EAPI/PMS extensions that allow package managers to do what smart-live-rebuild does. -- Thanks, Zac
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, 8 Jul 2017 19:24:46 -0400 Rich Freeman wrote: > > I don't see why a package manager couldn't offer the same > functionality for a meta package. As was pointed out the set behavior > for unmerging isn't always desirable. Your missing that sets maybe made by the user, Making a meta ebuild is a bit more complex then dropping package names into a file for a set, no digest, etc. Sets seem to serve a different purpose. With regard to unmerging not being desirable. That is if someone does something stupid like putting a system package into a set. But as I mentioned that the package is part of another set, system or world. It would be pulled back into the system. There are warnings and other stuff that take place when a critical package is removed. A user can remove those now just as they could if they added it to a set. Which really makes that argument moot. Do dumb stuff, you will get undesired results. Like removing a system package, via any means, set, directly, meta dep, etc. > > > > world and system are sets we all have. Not sure about PMS. It is > > something portage has supported for some time. You likely have many > > sets already on your system > > Certainly. You just can't depend on them and so on without having > them in PMS, because portage isn't the only package manager we > "support." Not sure about other package managers, but I would think they have similar function. If not then anything listed under emerge --list-sets, would be portage specific. That would likely break other things. > It just strikes me that we're probably better off picking one way of > doing this and putting lots of support behind it, versus having two > ways of doing this and some features work with one but not the other. > Of the two meta packages seem like they're the most generic. The two ways are not the same, and there is a reason sets exist in the first place. People seem to be over looking that fact. I did not add sets. They are not new. I am simply trying to expand their use. If someone wants to "try" out some packages a set is an ideal way of doing such. If someone wants a subset of what is in a meta ebuild. Again a set is an ideal way to do that. If for no other reason, then if the user wants to remove them. They just emerge -C @my_set. I find sets very useful! Only limitation is the profile aspect. -- William L. Thomson Jr. pgpQ7noeVFu3a.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, 8 Jul 2017 16:35:34 -0700 Zac Medico wrote: > On Sat, Jul 8, 2017 at 4:09 PM, William L. Thomson Jr. > wrote: > > Sets are also used for package rebuilds, like x11-module-rebuild, > > live-rebuild, and others. > > Usually there are better ways to trigger rebuilds. Those have nothing to do with me, and I am not using sets for rebuild purposes. Sets in a profile per my interest and topic have nothing to do with rebuilding. Just pointing out others current usage of sets. > For example, slot operator dependencies for rebuilds due to subslot > changes, and --newuse for USE changes. Problem is portage does not catch all changes all the time. Many times I will update world with binaries, Just to find out if I run without binaries more packages need to be updated. However if there are no changes, and the user wants to rebuild. How do they do that? Sets make it very easy. > For live-rebuild, it would be > much nicer to have a framework that automatically triggers rebuilds > when upstream changes are detected, like smart-live-rebuild. Which would require some sort of check to upstream to detect changes on some interval. If the user wants to do that manually. How do they do that? Sets make it very easy. > We can add EAPI/PMS extensions that allow package managers to do what > smart-live-rebuild does. Will that allow a user to rebuild on demand for their own reasons with no changes to system that are "detectable"? Again rebuilding is not my interest in sets. That is others usage of such. However they seem like they could serve a purpose in such situations. Giving the user really fine grain control without having to list a set of packages or mess with making various meta ebuilds. -- William L. Thomson Jr. pgpQKu4C4tITh.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, 8 Jul 2017 19:39:33 -0400 "William L. Thomson Jr." wrote: > The two ways are not the same, and there is a reason sets exist in the > first place. People seem to be over looking that fact. I did not add > sets. They are not new. I am simply trying to expand their use. Sets exist because people keep saying "let's have sets!" without agreeing on what sets actually are or how they are to be used. Sets remain half-baked because it turns out they don't make consistent sense in different contexts when you give them a non-superficial amount of thought. -- Ciaran McCreesh
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sun, 9 Jul 2017 00:49:57 +0100 Ciaran McCreesh wrote: > On Sat, 8 Jul 2017 19:39:33 -0400 > "William L. Thomson Jr." wrote: > > The two ways are not the same, and there is a reason sets exist in > > the first place. People seem to be over looking that fact. I did > > not add sets. They are not new. I am simply trying to expand their > > use. > > Sets exist because people keep saying "let's have sets!" without > agreeing on what sets actually are or how they are to be used. Do they need to agree? Isn't Gentoo about choice? Maybe your use of sets is different from mine. Is that not acceptable to have choice? > Sets remain half-baked because it turns out they don't make > consistent sense in different contexts when you give them a > non-superficial amount of thought. Much of the world is half baked, but we manage despite such. There is much to be said about over thinking over engineering something. Few if anything in life is perfect. Many of the things we depend on are far from perfect. C'est la vie. This must be a Gentoo thing. Sets have nothing to do with me. Seems they have been around since ~2009. Someone coded in support for them etc. Seems some find use for them, and others have objection. Having been around Gentoo for some time, and just now coming across sets. I find them quite useful! Just ran into one limitation of them. -- William L. Thomson Jr. pgpgA_Jg_1aNe.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, 8 Jul 2017 19:58:13 -0400 "William L. Thomson Jr." wrote: > On Sun, 9 Jul 2017 00:49:57 +0100 > Ciaran McCreesh wrote: > > On Sat, 8 Jul 2017 19:39:33 -0400 > > "William L. Thomson Jr." wrote: > > > The two ways are not the same, and there is a reason sets exist in > > > the first place. People seem to be over looking that fact. I did > > > not add sets. They are not new. I am simply trying to expand > > > their use. > > > > Sets exist because people keep saying "let's have sets!" without > > agreeing on what sets actually are or how they are to be used. > > Do they need to agree? Isn't Gentoo about choice? Maybe your use of > sets is different from mine. Is that not acceptable to have choice? Well yes, they do need to agree, because otherwise when two different developers put sets in a profile expecting different effects, then at least two developers are going to end up confused, disappointed, and quite probably breaking user systems. -- Ciaran McCreesh
Re: [gentoo-dev] Sets vs Meta ebuilds
On Fri, Jul 07, 2017 at 01:07:57PM -0400, William L. Thomson Jr. wrote > On Fri, 7 Jul 2017 12:57:17 -0400 > Brian Evans wrote: > > > Beware of sets.. if you put toolchain packages in a set and later > > do 'emerge --unmerge @custom-set' , emerge will happily destroy > > your toolchain. > > That is not much different than removing a system package directly. > If you do foolish things you will run into such problems. That would > be a self inflicted issue. Likely done out of not knowing what you > are doing. I build Pale Moon to my own custom specs, as well as a contributed SSE-only build for older machines. At one point, I read the list of necessary stuff to do the build. I incorporated the following into /etc/portage/palemoon_build >=app-arch/zip-2.3 >=dev-lang/perl-5.6 >=dev-lang/python-2.7.3 >=dev-lang/yasm-1.0.1 >=dev-libs/glib-2.24 dev-vcs/git media-libs/fontconfig >=media-libs/freetype-2.1.0 media-libs/mesa =sys-devel/autoconf-2.13 sys-devel/gcc <=x11-libs/gtk+-3.0 x11-libs/libXt x11-themes/hicolor-icon-theme > Though I will have to see what happens if a package is listed in more > than one set. I think there is a hierarchy there. I tried "emerge -pv --unmerge @palemoon_build", and it was ready to delete all the stuff, including gcc, etc. > Not to mention if it was removed. I think the world or system set > would pull it back in. Kind of hard to "pull it back in" if gcc or glib or ncurses isn't present. This is rather dangerous. The problem is that, unlike an ebuild, "emerge --unmerge @set" removes all packages in the set, regardless of whether they're required by another package or not. I deleted /etc/portage/sets/palemoon_build, and the entry "@palemoon_build" from /var/lib/portage/world_sets. It turns out that all these packages are required anyways. hicolor-icon-theme was not required previously, but gtk seems to add more and more GNOME dependencies every time I update my system. Let's say I try to do this as a meta package. So in my overlay I create a category "meta-set" and a file "meta-set/pmbuild-0.ebuild" EAPI=5 SLOT="0" KEYWORDS="amd64 x86" DEPEND=" >=app-arch/zip-2.3 >=dev-lang/perl-5.6 >=dev-lang/python-2.7.3 >=dev-lang/yasm-1.0.1 >=dev-libs/glib-2.24 dev-vcs/git media-libs/fontconfig >=media-libs/freetype-2.1.0 media-libs/mesa =sys-devel/autoconf-2.13 sys-devel/gcc <=x11-libs/gtk+-3.0 x11-libs/libXt x11-themes/hicolor-icon-theme" Does each entry have to be detailed with configure, install, etc, stuff, or is this sufficient? -- Walter Dnes I don't run "desktop environments"; I run useful applications
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sun, 9 Jul 2017 01:10:11 +0100 Ciaran McCreesh wrote: > On Sat, 8 Jul 2017 19:58:13 -0400 > "William L. Thomson Jr." wrote: > > On Sun, 9 Jul 2017 00:49:57 +0100 > > Ciaran McCreesh wrote: > > > On Sat, 8 Jul 2017 19:39:33 -0400 > > > "William L. Thomson Jr." wrote: > > > > The two ways are not the same, and there is a reason sets exist > > > > in the first place. People seem to be over looking that fact. I > > > > did not add sets. They are not new. I am simply trying to > > > > expand their use. > > > > > > Sets exist because people keep saying "let's have sets!" without > > > agreeing on what sets actually are or how they are to be used. > > > > Do they need to agree? Isn't Gentoo about choice? Maybe your use of > > sets is different from mine. Is that not acceptable to have > > choice? > > Well yes, they do need to agree, because otherwise when two different > developers put sets in a profile expecting different effects, then at > least two developers are going to end up confused, disappointed, and > quite probably breaking user systems. Valid points, so basically need a set of guidelines or rules for sets used in profiles. Which should not be that complex, as usage is minimal. Offhand, likely could be more; - Sets used in profiles are "lists of packages" for users to emerge/re-emerge, and as such should be minimal list only. Similar to the contents of a profile/packages, less the * symbol. - Sets used in profiles cannot have use expansion, versions or anything beyond cat/pkg. - Sets should not have the same file listed, in that case inherit the other set if using overlapping packages or split into smaller -- William L. Thomson Jr. pgpPXW60ZeBxe.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Sets vs Meta ebuilds
On Sat, Jul 8, 2017 at 8:27 PM, Walter Dnes wrote: > > Let's say I try to do this as a meta package. So in my overlay I > create a category "meta-set" and a file "meta-set/pmbuild-0.ebuild" > > EAPI=5 > SLOT="0" > KEYWORDS="amd64 x86" > DEPEND=" >>=app-arch/zip-2.3 >>=dev-lang/perl-5.6 >>=dev-lang/python-2.7.3 >>=dev-lang/yasm-1.0.1 >>=dev-libs/glib-2.24 >dev-vcs/git >media-libs/fontconfig >>=media-libs/freetype-2.1.0 >media-libs/mesa >=sys-devel/autoconf-2.13 >sys-devel/gcc ><=x11-libs/gtk+-3.0 >x11-libs/libXt >x11-themes/hicolor-icon-theme" > > Does each entry have to be detailed with configure, install, etc, > stuff, or is this sufficient? > Setting aside QA stuff like description/etc (which wouldn't matter if you're just doing this in your own overlay) this should work fine. If you install this it would pull in all the other stuff, and then do nothing. If you intend for this stuff to stick around you probably want to put it in RDEPEND, otherwise it will just get cleaned at first opportunity. Ebuilds specify their dependencies, they don't have to replicate the ebuilds of those dependencies. If portage needs to install git it will go read the git ebuild and figure out how to install it. Most of the stuff under /usr/portage/virtual should give you a sense of what can be done. Virtuals are really only different from meta-packages in how they're used, not how they work. It is slightly more cruft than a set, but honestly not a great deal so. -- Rich
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, Jul 8, 2017 at 4:46 PM, William L. Thomson Jr. wrote: > On Sat, 8 Jul 2017 16:35:34 -0700 > Zac Medico wrote: > >> For live-rebuild, it would be >> much nicer to have a framework that automatically triggers rebuilds >> when upstream changes are detected, like smart-live-rebuild. > > Which would require some sort of check to upstream to detect changes on > some interval. What I imagine is an option like --newuse that rebuilds packages with upstream changes. I suppose you could also have an option to rebuild if some interval has expired since the last rebuild. -- Thanks, Zac
Re: [gentoo-dev] Sets vs Meta ebuilds
On Sat, 8 Jul 2017 20:27:38 -0400 "Walter Dnes" wrote: > > > Though I will have to see what happens if a package is listed in > > more than one set. I think there is a hierarchy there. > > I tried "emerge -pv --unmerge @palemoon_build", and it was ready to > delete all the stuff, including gcc, etc. Did you get any warnings? Your about to remove a system package, etc. > > Not to mention if it was removed. I think the world or system set > > would pull it back in. > > Kind of hard to "pull it back in" if gcc or glib or ncurses isn't > present. This is rather dangerous. The problem is that, unlike an > ebuild, "emerge --unmerge @set" removes all packages in the set, > regardless of whether they're required by another package or not. It is the same as doing emerge -C gcc. At the same time if you built and generated a binary package. On re-emerge you could pull in the gcc binary you made when it was installed the first time. I love binary packages, I make and use them all the time. Invaluable for re-emerging. If you are making sets, adding system packages, and removing those. I would assume you are doing that as some sort of testing. Which would want to re-emerge/install those packages. Depending on what your doing very likely would want to make and use binaries in that process. Surely if removing system packages :) > I deleted /etc/portage/sets/palemoon_build, and the entry > "@palemoon_build" from /var/lib/portage/world_sets. It turns out that > all these packages are required anyways. Meaning little was removed after you did emerge --depclean world ? -- William L. Thomson Jr. pgpBF_KcKMHrU.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, 8 Jul 2017 18:30:10 -0700 Zac Medico wrote: > On Sat, Jul 8, 2017 at 4:46 PM, William L. Thomson Jr. > wrote: > > On Sat, 8 Jul 2017 16:35:34 -0700 > > Zac Medico wrote: > > > >> For live-rebuild, it would be > >> much nicer to have a framework that automatically triggers rebuilds > >> when upstream changes are detected, like smart-live-rebuild. > > > > Which would require some sort of check to upstream to detect > > changes on some interval. > > What I imagine is an option like --newuse that rebuilds packages with > upstream changes. I suppose you could also have an option to rebuild > if some interval has expired since the last rebuild. That could be useful for all live packages without requiring a set. It could also be used for packages that are part of a set. Like if you have a set of live ebuilds, plus some others one your system emerge --live world Updates all live ebuilds, in a set or not. That would be useful. I tend to avoid emerging live ebuilds due to having to always re-emerge them. Sets would help there. But a emerge option would be even better!!! Could have cron run that, and then an interval in portage is not necessary. I was more thinking some sort of hook to upstream to see if there have been any commits or changes. No reason to rebuild a live package if nothing changed :) -- William L. Thomson Jr. pgpk_2ZygI7q8.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Sets vs Meta ebuilds
On Sat, Jul 8, 2017 at 6:39 PM, William L. Thomson Jr. wrote: > On Sat, 8 Jul 2017 18:30:10 -0700 > Zac Medico wrote: > >> On Sat, Jul 8, 2017 at 4:46 PM, William L. Thomson Jr. >> wrote: >> > On Sat, 8 Jul 2017 16:35:34 -0700 >> > Zac Medico wrote: >> > >> >> For live-rebuild, it would be >> >> much nicer to have a framework that automatically triggers rebuilds >> >> when upstream changes are detected, like smart-live-rebuild. >> > >> > Which would require some sort of check to upstream to detect >> > changes on some interval. >> >> What I imagine is an option like --newuse that rebuilds packages with >> upstream changes. I suppose you could also have an option to rebuild >> if some interval has expired since the last rebuild. > > That could be useful for all live packages without requiring a set. It > could also be used for packages that are part of a set. Like if you > have a set of live ebuilds, plus some others one your system > > emerge --live world > > Updates all live ebuilds, in a set or not. That would be useful. I tend > to avoid emerging live ebuilds due to having to always re-emerge them. > Sets would help there. But a emerge option would be even better!!! Yeah, but it's not going to happen without an EAPI/PMS extension. There needs to be a standard way for the package manager to check if there has been an upstream change for a given package since the last time it was rebuilt. > Could have cron run that, and then an interval in portage is not > necessary. Doing updates via cron is inappropriate for most scenarios. I was thinking that we'd have an option to rebuild if an interval has expired *and* there has been an upstream change. That way, you can limit the rate of rebuilds if upstream is changing very frequently. > I was more thinking some sort of hook to upstream to see if > there have been any commits or changes. No reason to rebuild a live > package if nothing changed :) Yeah, that's what I was thinking all along. That's why we need an EAPI/PMS extension, in order to implement behavior like smart-live-rebuild. -- Thanks, Zac
[gentoo-dev] Last rites: dev-ruby/modernizr
# Hans de Graaff (09 Jul 2017) # Upstream has removed the code and the published gem. # Removal in 30 days. dev-ruby/modernizr signature.asc Description: This is a digitally signed message part
