Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Léo Le Bouter]

On Tue, 2021-03-16 at 23:28 +, Christopher Baines wrote:
> I did spot these patches
> https://patches.guix-patches.cbaines.net/project/guix-patches/list/?series=7298

Awesome!! I did not see them earlier!

> I think the Guix Data Service could run the "refresh" code from Guix
> and
> store relevant data, although I'm also thinking along the lines of
> trying to generate patches, like you go on to below...

Yes :-D

> So, one thing I'm hoping to start work on in the coming weeks/months
> is
> the long awaited service for providing subscriptions/notifications
> for
> the Guix Data Service (+ other services).
> 
> My current plan is to use a WebSub like pattern, but this could
> easily
> be adapted to RSS/email/...

That sounds very great! I know you've been working on this, but I also
wanted (through my email) try to make more people aware and draw
attention to the subject of automation in GNU Guix.

> Yeah, one problem with the current automated patch review stuff I've
> got
> going at the moment is that there's no feedback when the Guix Data
> Service finds out that things do/don't build.

Yes, and because of that the Guix Data Service and guix-build-
coordinator-agent thing you are running goes mostly unused (even for
me!).

> However, as I also set out above, there's been a plan and design for
> making that possible for years, it just needs implementing.
> 
> One thing I'm hoping to do once it's possible to subscribe to Guix
> Data
> Service data is make the checks in Patchwork actually reflective of
> results (like 4 packages broken by these changes), rather than just
> providing links, and someone having to figure out what information is
> hidden within.
> 
> Those same subscriptions could be used to prompt people to look at
> patches for package updates that don't introduce breakages (following
> what you set out above).

+1000

> The pieces are slowly coming together for this, at least with the way
> I
> would approach it. For example, it's possible to get commits in to
> data.guix-patches.cbaines.net now by just pushing to the Git
> repository,
> so to set out something similar to what you describe above:
> 
>  - Some service watches for new releases (through the guix refresh
>code), and then makes commits, and pushes them to a Git repo
> 
>  - There's a Guix Data Service reading from that Git repository, it
>starts processing the changes
> 
>  - Something (probably the "service" above") subscribes to find out
> when
>relevant information is available (like build successes/failures)
> 
>  - Builds happen via the Guix Build Coordinator, which feeds
> information
>back to the Guix Data Service
> 
>  - That "service" gets the notifications that the commits are good,
> and
>prompts someone to review them

All awesome! Can't wait! As soon as I can I will make sure to help, I
am reading through the GNU Guile manual, need to find more
concentration time.

> I hope some of that makes sense,
> 
> Chris

Thanks a lot Chris for all this!!


signature.asc
Description: This is a digitally signed message part

Guix moving too fast?

[zimoun]

Hi,

Thanks Mark for your words.  Interestingly, using the angle of
scientific software, I agree with your general analysis.


On Tue, 16 Mar 2021 at 19:49, Leo Famulari  wrote:
> On Tue, Mar 16, 2021 at 07:19:59PM -0400, Mark H Weaver wrote:

>> Ultimately, I gave up.  In my opinion, Guix has never achieved usability
>> as a desktop system on non-Intel systems.  Therefore, the Guix community
>> is unable to attract many developers who want a distro that supports
>> non-Intel systems well.  Our community has thus become dominated by
>> Intel users, and there's unsufficient political will to adopt policies
>> that would enable us to provide a usable system for non-Intel users.
>> 
>> What do you think?
>
> Thanks, as always, for your well-reasoned message. Your story of your
> experience here, and what it means for Guix, computing security, and
> free software in general, is really valuable. I still think it's really
> unfortunate for the rest of us that you gave up, but I don't see how it
> could have gone differently.

Moving less fast? :-)


> I agree with you that Guix moves too fast to support non-Intel
> architectures in its current state. My hope is that, within the next two
> years, there will be workstation aarch64 machines more widely available
> at comparable prices to Intel/AMD, and this will translate into more
> developer support for aarch64 in the years after that. Time will tell.

Moving too fast, i.e., pushing a lot of changes, has various other
consequences: a lot of rebuilds, even if the build farm is really
improving these days, there is a high probability that “guix pull”
intensively computes––hopefully ’channel-with-substitutes-available’ [1]
avoids that for limited resource machines––then “guix install” right
after generally builds the package locally.  For example, the package
’gmsh’ [2], there is no substitute for 373c7b5 (pulled on March 13th
couple of days ago) but builds fine, and this ’gmsh’ packages had been
last updated on Oct 8th 2020.

If you look at the Data Service [2], there is 7 different “versions” for
the same 4.6.0.  Therefore, it had been implied by some unrelated
changes.  That’s fine and that’s why Guix is so great: 4.6.0 is not
enough for the binary versioning.

This can be really worse, for instance the package ’openfoam’ [3].  It
is a complex package and for the same 4.1, the output version changes
every 5 days on average.  How is it possible that the build farm follows
such rate?

Another example, the package ’freecad’ [4].

Even worse, what happens if an unrelated change breaks the package?  The
time spent at fixing it is not spent at adding other nice-to-have
packages or fixing bugs.  Or we prefer to add/update other packages or
add features and this case, because we do not care, this broken package
should be removed: from a user perspective, nothing is worse that broken
packages and from the build farm perspective, it saves resources.

Currently, there is ~5% (~1000 packages) of broken packages.  My guess
is, considering the same rate of changes and an increase of the number
of packages, this percentage would be the same, i.e., the number of
broken packages would be higher.

Well, because scientific software are often complex, meaning with a huge
dependency graph, it makes hard to maintain them with such change rate.

And I am not speaking about third-party channels.  They are also hard to
maintain.

To that, let multiply by the number of architectures.


All in all, maybe the 3 branches model does not scale well.  It should
not be possible that broken packages are in the default branch (now
master).  It is unavoidable with the current model.  Chris initiated
discussions and works about QA with patchwork, see [5,6].  Somehow, what
is “production” should be distinguished from what is “test”.  It is not
currently the case, every updates is pushed to master and we cross the
fingers—–it works well most of the time though, the rare broken cases
are just full annoyance and too visible to my taste.


The default branch could be “stable”, which receives only security
fixes, i.e., grafts, also bug fixes and patches touching ’guix/’.  All
other patches, i.e., touching ’gnu/’, could go to “next”.  And massive
rebuilds could go to “massive”.  Each time a graft is added, it is also
ungrafted to “next” (or “massive” if it is a real deep change),
therefore, each release becomes (almost) ungrafted.

The branches “stable” and “next” are continuously built whereas
“massive” only built before the merge.

Every 3 months, “next” is merged to “stable”.
Every 3 months, “massive” is merged to “next”.
Every 6 months, “stable” is released.

Like a metronome.  For instance, if the “massive” merge is missed for
whatever reason, it is delayed to 3 months but “next” is still merged on
time and the release is on time too.

I do not have a clear view about other architectures than x86_64, but
since “stable” is moving slower, it could be discussed at the “next”
merge time some changes speci

Re: Security-czar needed? WAS: Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[zimoun]

Hi,

On Wed, 17 Mar 2021 at 07:24, Léo Le Bouter  wrote:

> I think we can handle this without granting us any special powers, I
> like it that we don't have roles actually!
>
> We can discuss, debate, agree to common goals, I don't think we are
> going to enter into conflict, we hear each other, we communicate, I
> think that's a really good thing in GNU Guix :-D
>
> Lots of other communities enter into conflict fast and stop
> communicating, GNU Guix is not that, there's a spirit of goodwill of
> everyone and that's really pleasing to live as a contributor and user.

I agree and am aligned with these words. (Without saying there is de
facto hats. :-))

The downside is that sometimes things are stalling.  Examples:
core-updates unmerged since ~10 months, patches that fall in the crack,
old bugs never closed, etc. Pick any non fun stuff. :-)

Hard topic about collective work in general: is it possible to scale
with only implicit hats and no explicit ones? :-)  Hat meaning feel in
charge and do the job to make it happen.


Cheers,
simon

Re: Are gzip-compressed substitutes still used?

[Ludovic Courtès]

Hi,

Ludovic Courtès  skribis:

> From that, we could deduce that about 1% of our users who take
> substitutes from ci.guix are still using a pre-1.1.0 daemon without
> support for lzip compression.
>
> I find it surprisingly low: 1.1.0 was released “only” 9 months ago,
> which is not a lot for someone used to the long release cycles of
> “stable” distros.

(See

for the initial message.)

Here’s an update, 1.5 month later.  This time I’m looking at nginx logs
covering Feb 8th to Mar 17th and using a laxer regexp than in the
message above, here are the gzip/lzip download ratio for several
packages:

--8<---cut here---start->8---
ludo@berlin ~$ ./nar-download-stats.sh /tmp/sample3.log 
   gtk%2B-3: gzip/lzip ratio: 37/3255 1%
glib-2: gzip/lzip ratio: 97/8629 1%
coreutils-8: gzip/lzip ratio: 81/2306 3%
python-3: gzip/lzip ratio: 120/7177 1%
r-minimal-[34]: gzip/lzip ratio: 8/302 2%
openmpi-4: gzip/lzip ratio: 19/236 8%
hwloc-2: gzip/lzip ratio: 10/43 23%
gfortran-7: gzip/lzip ratio: 6/225 2%
--8<---cut here---end--->8---

(Script attached.)

The hwloc/openmpi outlier is intriguing.  Is it one HPC web site running
an old daemon, or several of them?  Looking more closely, it’s 22 of
them on 8 different networks (looking at the first three digits of the
IP address):

--8<---cut here---start->8---
ludo@berlin ~$ grep -E 
'/gzip/[[:alnum:]]{32}-(hwloc-2|openmpi-4)\.[[:digit:]]+\.[[:digit:]]+ ' < 
/tmp/sample3.log | cut -f1 -d- | sort -u | wc -l
22
ludo@berlin ~$ grep -E 
'/gzip/[[:alnum:]]{32}-(hwloc-2|openmpi-4)\.[[:digit:]]+\.[[:digit:]]+ ' < 
/tmp/sample3.log | cut -f1 -d- | cut -f 1-3 -d. | sort -u | wc -l
8
--8<---cut here---end--->8---

Conclusion?  It still sounds like we can’t reasonably remove gzip
support just yet.

I’d still like to start providing zstd-compressed substitutes though.
So I think what we can do is:

  • start providing zstd substitutes on berlin right now so that when
1.2.1 comes out, at least some substitutes are available as zstd;

  • when 1.2.1 is announced, announce that gzip substitutes may be
removed in the future and invite users to upgrade;

  • revisit this issue with an eye on dropping gzip within 6–18 months.

Thoughts?

Ludo’.

#!/bin/sh

if [ ! "$#" = 1 ]
then
echo "Usage: $1 NGINX-LOG-FILE"
exit 1
fi

set -e

sample="$1"
items="gtk%2B-3 glib-2 coreutils-8 python-3 r-minimal-[34] openmpi-4 hwloc-2 
gfortran-7"

for i in $items
do
# Tweak the regexp so we don't catch ".drv" substitutes as these
# usually compress better with gzip.
lzip="$(grep -E "/lzip/[[:alnum:]]{32}-$i\\.[[:digit:]]+(\\.[[:digit:]]+)? 
" < "$sample" | wc -l)"
gzip="$(grep -E "/gzip/[[:alnum:]]{32}-$i\\.[[:digit:]]+(\\.[[:digit:]]+)? 
" < "$sample" | wc -l)"
echo "$i: gzip/lzip ratio: $gzip/$lzip $(($gzip * 100 / $lzip))%"
done

armhf-linux substitutes

[Ludovic Courtès]

Hi,

Leo Famulari  skribis:

> On Mon, Mar 15, 2021 at 05:55:21PM +0100, Ludovic Courtès wrote:
>> > The architecture armf will not be included.
>> 
>> Wait wait, I missed that.  What happened?  I think we should include it,
>> even if substitute availability remains low.
>
> I had asked about the status of the armhf branch on #guix when a few of
> us were trying to "tie up loose ends" for this release.
>
> I don't have an opinion one way or the other, but since we are not
> building substitutes for it at all, we should include in the release
> announcement a clear description of the level of support that users can
> expect.

Right, we could adjust the text in the “GNU Distribution” node of the
manual (we did that before when mips64el-linux was supported without
substitutes.)

Mathieu, what’s preventing us from doing armhf-linux builds again?  We
could use the OverDrives for that (with an upper bound though), along
with the SBCs in machines-for-berlin.scm.

That won’t be enough to keep up, so perhaps we’ll have to restrict
armhf-linux builds to the “core” subset or the release-manifest.scm.

Thoughts?

Ludo’.

Why [bug#47081] Remove mongodb?

[zimoun]

Hi Léo,

On Fri, 12 Mar 2021 at 01:56, Léo Le Bouter  wrote:

> mongodb 3.4.10 has unpatched CVEs and mongodb 3.4.24 has some files in the
> release tarball under the SSPL, therefore we cannot provide mongodb while
> upholding to good security standards.

[...]

>  doc/guix.texi  |  28 -
>  gnu/packages/databases.scm | 252 -
>  gnu/services/databases.scm |  88 -
>  gnu/tests/databases.scm|  83 
>  4 files changed, 451 deletions(-)

Could you wait more than 4 days between the patch submission and
effectively pushing it?

Well, you updated mongodb from 3.4.10 to 3.4.24 on the March 10th,
submitted a patch series for the removal on the March 12th and pushed on
the March 16th.  In the meantime, the update has been reverted on the
March 11th because of license issue, IIUC.


If the removal for security reasons had been discussed on IRC, it could
be nice to point the discussion here.  Otherwise, open a discussion on
the topic on guix-devel or bug-guix.  The full removal is a radical
solution (especially, it should be done with 2 commits: service+doc and
then package; well another story).


All the best,
simon

Re: Are gzip-compressed substitutes still used?

[Léo Le Bouter]

Just as a reminder siding with vagrantc here:

We must ensure the Debian 'guix' package can still work and upgrade
from it's installed version, so ensure that removing gzip doesnt break
initial 'guix pull' with it.


signature.asc
Description: This is a digitally signed message part

Re: Why [bug#47081] Remove mongodb?

[Léo Le Bouter]

On Wed, 2021-03-17 at 17:56 +0100, zimoun wrote:
> If the removal for security reasons had been discussed on IRC, it
> could
> be nice to point the discussion here.  Otherwise, open a discussion
> on
> the topic on guix-devel or bug-guix.  The full removal is a radical
> solution (especially, it should be done with 2 commits: service+doc
> and
> then package; well another story).

https://issues.guix.gnu.org/47081 - some of it there: 
https://logs.guix.gnu.org/guix/2021-03-12.log#001752

Efraim, Cbaines, Lfam was involved there and shown no big objections

> 
> Well, you updated mongodb from 3.4.10 to 3.4.24 on the March 10th,
> submitted a patch series for the removal on the March 12th and pushed
> on
> the March 16th.  In the meantime, the update has been reverted on the
> March 11th because of license issue, IIUC.
> 

The security update was reverted, then the revert was reverted due to
debate on licensing which turns out reverting the revert was actually
wrong because some specific files were under SSPL, at that point we
were shipping SSPL code which is nonfree, so the removal is also that.

Nonfree code + security issue made it kind of stressful

Léo


signature.asc
Description: This is a digitally signed message part

Re: [bug#47163] Using package transformations declaratively

[Ludovic Courtès]

Hi,

zimoun  skribis:

> There is several ways to have package transformations at the manifest
> level.  One is:
>
> (use-modules (guix transformations))
>
> (define transform1
>   (options->transformation
> '((with-c-toolchain . "hello=gcc-toolchain@8"
>
> (packages->manifest
>   (list (transform1 (specification->package "hello"

On this topic, do not miss this section of the fine manual:
.
:-)

Ludo’.

Re: Why [bug#47081] Remove mongodb?

[Léo Le Bouter]

Sorry for duplicated email,

On Wed, 2021-03-17 at 17:56 +0100, zimoun wrote:
> If the removal for security reasons had been discussed on IRC, it
> could
> be nice to point the discussion here.  Otherwise, open a discussion
> on
> the topic on guix-devel or bug-guix.  The full removal is a radical
> solution (especially, it should be done with 2 commits: service+doc
> and
> then package; well another story).

Another thing is that openssl 1.1.1 on non-SSPL mongodb doesnt work and
we are working on removal of openssl 1.0.x which will removed all it's
dependents and mongodb is one so it was inevitably going to be removed
anyway.

> All the best,
> simon


signature.asc
Description: This is a digitally signed message part

Re: [bootstrappable] Re: Can Guile be bootstrapped from source without psyntax-pp.scm?

[Ludovic Courtès]

Hi Michael,

Michael Schierl  skribis:

> Am 15.03.2021 um 18:09 schrieb Ludovic Courtès:
>> Woow, this is great news!  I think it would be great towards importing
>> it in Guile proper.
>>
>> To do that, I think we should first get Andy’s opinion on the approach.
>
> I don't think upstream is very interested in having psyntax-pp.scm
> bootstrappable. In Guile 3.0.3 they broke even the `make
> ice-9/psyntax-pp.scm.gen` target, and did not repair it even in Guile
> 3.0.5, that's why I used 3.0.2 for the bootstrap. But I included a patch
> to repair it in 3.0.5 in case you really want to bootstrap that version
> (psyntax-pp.scm has not changed there). OTOH, from the git log it seems
> like psyntax is currently being overhauled for the next release, so
> probably my code would need some updates for the next version.

Andy made it clear that it was a bug.  There’s an interest in providing
a good bootstrapping story for Guile; that’s why there’s an interpreter
written in C, for example.  I’m sure we’d be happy to address psyntax
bootstrapping as well!

> Also, in the last 15 years I avoided directly contributing to "GNU
> projects" (with FSF as copyright holder in the license headers), reasons
> below. But if anyone else takes my code and upstreams it, I won't object.

Sure, we can discuss the details of how to integrate your code.  (Guile
incorporates code not initially written for Guile, such as sxml or
(ice-9 match), and the policy is to not require copyright assignment for
such code.)

> Regardless, even when not part of Guile, I believe this code is very
> useful for both the live-bootstrap project and Guix to get their Guile
> bootstrapped. And even if nobody ever updates it for 3.0.6+, you can
> always bootstrap the later versions from an earlier Guile. And maybe a
> variation of it lands in GNU Mes, too.

Yes, that would be great.  We’ll have to take a closer look, but I’m
under the impression that a fruitful approach for Guile would be to have
it maintained in-tree; that would immediately benefit all distros.

Thanks!

Ludo’.

PS: Not all GNU packages require copyright assignment.  Guix doesn’t,
for instance, and we’d be happy to get your contributions.  :-)

Finding the store path of a package

[Konrad Hinsen]

Dear Guix experts,

I wonder if there is a straightforward way to find the store path
corresponding to a package, assuming that the package actually is in the
store. I don't care if it's done via the CLI or via Guile code.

Use case: Looking at the files inside a package. What I do now is "ls
/gnu/store/**", but that usually lists many variants of
the package, and I don't know which of them actually is the current one.

I came up with some Guile code that does the job:

   (define (store-path specification)
 (let*-values (((package output)
(specification->package+output specification))
   ((entry)
(package->manifest-entry package output))
   ((l-entry)
(with-store store
  (run-with-store store
(lower-manifest-entry entry (%current-system))
   (manifest-entry-item l-entry)))

but it also downloads/builds the package if it's not yet in the store,
which is not what I want. In fact, I don't care what happens then the
package is not in the store. Returning a non-existing path is fine,
as is raising an error or returning #f.

Another attempt is "guix package –-list-installed", but this works only
for packages installed in a profile. I am shifting more and more to
on-the-fly environments, meaning that many packages in my store belong
to no profile.

Cheers,
  Konrad.

Rust and parametric packages

[raingloom]

I'm re-reading the threads about Rust packaging and I realized there
might be a solution to the build artifact reuse problem.

As I understand it, the problem is that crates can be compiled with any
number of features enabled or disabled. If this and the compiler
version are truly the only variables to consider, we could just lift
the features to the package level, right?

https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00269.html

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Ludovic Courtès]

Hi,

Léo Le Bouter  skribis:

> It seems Fedora has automated infrastructure and feeds to monitor new
> upstream releases so that maintainers can get notifications on them.
>
> https://release-monitoring.org/

Established distros have great tools and services for that.

Guix has ‘guix refresh’, which predates some of the trendy release/CVE
monitoring services and actually works well.  It’s not perfect, but it’s
good, so my advice would be to work on improving it.

If you look at Guix’s design and history, you’ll find that much is built
in a way that avoids relying on external services when we can.

> We could even go as far as preparing a patch (similar to guix refresh
> -u  && etc/committer.scm) and trying to build it and all it's
> dependents and if it doesnt cause any new failures we could send an
> automated patch contribution on guix-patches directly and tag it with
> "ci-update-ok" or something so a maintainer just has to double-check
> quickly and push (very efficient workflow). 

I definitely agree.  We could build upon the infrastructure we already
have to write a tool that roughly runs ‘guix refresh -u’ on individual
packages, attempts to build it and its dependents, and publishes a
ready-to-apply patch—or a failure notification.

Whether it’s a task for the Guix Data Service or some other service, I
don’t know, but it would be most welcome!

Ludo’.

Re: Are gzip-compressed substitutes still used?

[Vagrant Cascadian]

On 2021-03-17, Léo Le Bouter wrote:
> Just as a reminder siding with vagrantc here:
>
> We must ensure the Debian 'guix' package can still work and upgrade
> from it's installed version, so ensure that removing gzip doesnt break
> initial 'guix pull' with it.

... and I would expect this version to ship in Debian for another ~3-5
years, unless it gets removed from Debian bullseye before the upcoming
(real soon now) release!

But if lzip substitutes are still supported, I *think* guix 1.2.0 as
packaged in Debian still supports that, at least.

Dropping both gzip and lzip would be unfortunate; I don't think it would
be trivial to backport the zstd patches to guix 1.2.0, as it also
depends on guile-zstd?


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Léo Le Bouter]

On Wed, 2021-03-17 at 18:35 +0100, Ludovic Courtès wrote:
> Established distros have great tools and services for that.
> 
> Guix has ‘guix refresh’, which predates some of the trendy
> release/CVE
> monitoring services and actually works well.  It’s not perfect, but
> it’s
> good, so my advice would be to work on improving it.
> 
> If you look at Guix’s design and history, you’ll find that much is
> built
> in a way that avoids relying on external services when we can.

Just to clarify, I also get behind that very much I was suggesting the
functional part more so than the service kind, even though I think
services (like Guix Data Service) are OK as long as datasets are easily
exportable and the service itself can easily be re-deployed through GNU
Guix as well.

It seems GNU Guix takes a generic approach to updates while Debian or
Fedora seems to look at specialized rules for each package, I was
thinking we could import those already existing rules (in Fedora's or
Debian's) into GNU Guix, I find it a superior approach for reliability
of notifications even though generic updaters are also very
complementary.


signature.asc
Description: This is a digitally signed message part

Re: Why [bug#47081] Remove mongodb?

[zimoun]

On Wed, 17 Mar 2021 at 18:09, Léo Le Bouter  wrote:
> On Wed, 2021-03-17 at 17:56 +0100, zimoun wrote:
>> If the removal for security reasons had been discussed on IRC, it
>> could
>> be nice to point the discussion here.  Otherwise, open a discussion
>> on
>> the topic on guix-devel or bug-guix.  The full removal is a radical
>> solution (especially, it should be done with 2 commits: service+doc
>> and
>> then package; well another story).
>
> https://issues.guix.gnu.org/47081 - some of it there: 
> https://logs.guix.gnu.org/guix/2021-03-12.log#001752
>
> Efraim, Cbaines, Lfam was involved there and shown no big objections

Thanks.


>> Well, you updated mongodb from 3.4.10 to 3.4.24 on the March 10th,
>> submitted a patch series for the removal on the March 12th and pushed
>> on
>> the March 16th.  In the meantime, the update has been reverted on the
>> March 11th because of license issue, IIUC.
>> 
>
> The security update was reverted, then the revert was reverted due to
> debate on licensing which turns out reverting the revert was actually
> wrong because some specific files were under SSPL, at that point we
> were shipping SSPL code which is nonfree, so the removal is also that.

AFAIT, 3.4.10 is released under GNU AGPL 3.0 and Apache 2.0.  This
version had been released before the October 16th, 2018.  Could you
point which code is non-free?

IMHO, this claim about non-free code is wrong.  The last versions with
an acceptable license seem 4.0.3 or 4.1.4, I guess.

I am not against removing MongoBD.  I am just saying that the removal
deserves at least a message on guix-devel and maybe a --news entry.

Other said, it deserves more than 6 days between the “oh there is
security vulnerabilities” and the full removal.  When one uses a version
from 2017 as 3.4.10 is, one knows that it can have security
vulnerabilities.

I am not complaining about the commit itself, but I am complaining by
the way of doing the thing.


All the best,
simon

Re: Are gzip-compressed substitutes still used?

[zimoun]

Hi,

On Wed, 17 Mar 2021 at 18:12, Ludovic Courtès  wrote:

> I’d still like to start providing zstd-compressed substitutes though.
> So I think what we can do is:
>
>   • start providing zstd substitutes on berlin right now so that when
> 1.2.1 comes out, at least some substitutes are available as zstd;
>
>   • when 1.2.1 is announced, announce that gzip substitutes may be
> removed in the future and invite users to upgrade;
>
>   • revisit this issue with an eye on dropping gzip within 6–18 months.

Sounds reasonable.  The full removal could be announced for the 1.4
release.  Even if we do not know when it will happen. ;-)  So people
know what to expect.

Cheers,
simon

Re: Why [bug#47081] Remove mongodb?

[Léo Le Bouter]

On Wed, 2021-03-17 at 18:56 +0100, zimoun wrote:
> AFAIT, 3.4.10 is released under GNU AGPL 3.0 and Apache 2.0.  This
> version had been released before the October 16th, 2018.  Could you
> point which code is non-free?
> 
> IMHO, this claim about non-free code is wrong.  The last versions
> with
> an acceptable license seem 4.0.3 or 4.1.4, I guess.

It's not wrong, look at 2f9132e2e0b1e01398a01a32972e87f45ec2f7a6, we
were shipping 3.4.24 before the removal, not 3.4.10.

> I am not against removing MongoBD.  I am just saying that the removal
> deserves at least a message on guix-devel and maybe a --news entry.
> 
> Other said, it deserves more than 6 days between the “oh there is
> security vulnerabilities” and the full removal.  When one uses a
> version
> from 2017 as 3.4.10 is, one knows that it can have security
> vulnerabilities.
> 
> I am not complaining about the commit itself, but I am complaining by
> the way of doing the thing.

I agree, will do differently in the future, no one mentionned it during
all discussions, but if it was I would've, 3-4 days did not give you
time to comment so I'll wait longer maybe re-re-revert the revert to
restore 3.4.10 instead so we get rid of the non-free code issue. Does
anyone actually use MongoDB on GNU Guix? Some people don't look at
versions or when they were released and just trust GNU Guix.

> 
> All the best,
> simon

Léo


signature.asc
Description: This is a digitally signed message part

Re: Are gzip-compressed substitutes still used?

[Jonathan Brielmaier]

On 17.03.21 18:12, Ludovic Courtès wrote:

(See

for the initial message.)

Here’s an update, 1.5 month later.  This time I’m looking at nginx logs
covering Feb 8th to Mar 17th and using a laxer regexp than in the
message above, here are the gzip/lzip download ratio for several
packages:

--8<---cut here---start->8---
ludo@berlin ~$ ./nar-download-stats.sh /tmp/sample3.log 
   gtk%2B-3: gzip/lzip ratio: 37/3255 1%
glib-2: gzip/lzip ratio: 97/8629 1%
coreutils-8: gzip/lzip ratio: 81/2306 3%
python-3: gzip/lzip ratio: 120/7177 1%
r-minimal-[34]: gzip/lzip ratio: 8/302 2%
openmpi-4: gzip/lzip ratio: 19/236 8%
hwloc-2: gzip/lzip ratio: 10/43 23%
gfortran-7: gzip/lzip ratio: 6/225 2%
--8<---cut here---end--->8---


Interesting findings...


Conclusion?  It still sounds like we can’t reasonably remove gzip
support just yet.

I’d still like to start providing zstd-compressed substitutes though.
So I think what we can do is:

   • start providing zstd substitutes on berlin right now so that when
 1.2.1 comes out, at least some substitutes are available as zstd;

   • when 1.2.1 is announced, announce that gzip substitutes may be
 removed in the future and invite users to upgrade;


My personal substitution servers runs with lzip + zstd, so no gzip. It
works fine, but I didn't had any "legacy" users. Although zstd is only
0,6% of lzip in regards to total downloads over the last months...

Re: Rust and parametric packages

[Léo Le Bouter]

On Wed, 2021-03-17 at 18:30 +0100, raingloom wrote:
> I'm re-reading the threads about Rust packaging and I realized there
> might be a solution to the build artifact reuse problem.
> 
> As I understand it, the problem is that crates can be compiled with
> any
> number of features enabled or disabled. If this and the compiler
> version are truly the only variables to consider, we could just lift
> the features to the package level, right?
> 
> https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00269.html
> 

I advise you look there also: 
https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/rlib-intermediate-object-reuse/near/225653640


signature.asc
Description: This is a digitally signed message part

Re: Finding the store path of a package

[zimoun]

Hi Konrad,

On Wed, 17 Mar 2021 at 18:55, Konrad Hinsen  wrote:

> I wonder if there is a straightforward way to find the store path
> corresponding to a package, assuming that the package actually is in the
> store. I don't care if it's done via the CLI or via Guile code.

does “guix build  -n” fit your use-case?


> but it also downloads/builds the package if it's not yet in the store,
> which is not what I want. In fact, I don't care what happens then the
> package is not in the store. Returning a non-existing path is fine,
> as is raising an error or returning #f.

Well, ’package-output’ in (guix packages) is what you need, I guess.

--8<---cut here---start->8---
$ guix gc -D $(guix build hello)
0.1 MB will be downloaded:
   /gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10
substituting /gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10...
downloading from 
https://ci.guix.gnu.org/nar/lzip/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10 ...
 hello-2.10  51KiB  
454KiB/s 00:00 [##] 100.0%

finding garbage collector roots...
[0 MiB] deleting '/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10'
deleting `/gnu/store/trash'
deleting unused links...
note: currently hard linking saves 20885.94 MiB

$ guix repl
GNU Guile 3.0.5
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,use(gnu packages base) ; hello
scheme@(guix-user)> ,use(guix packages) ; package-output
scheme@(guix-user)> ,use(guix store) ; with-store
scheme@(guix-user)> (with-store store (package-output store hello))
$1 = "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10"
scheme@(guix-user)> ,q

$ ls /gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10
ls: cannot access '/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10': No 
such file or directory
--8<---cut here---end--->8---



Hope that helps,
simon

Re: Why [bug#47081] Remove mongodb?

[zimoun]

On Wed, 17 Mar 2021 at 19:16, Léo Le Bouter  wrote:
> On Wed, 2021-03-17 at 18:56 +0100, zimoun wrote:
>> AFAIT, 3.4.10 is released under GNU AGPL 3.0 and Apache 2.0.  This
>> version had been released before the October 16th, 2018.  Could you
>> point which code is non-free?
>> 
>> IMHO, this claim about non-free code is wrong.  The last versions
>> with
>> an acceptable license seem 4.0.3 or 4.1.4, I guess.
>
> It's not wrong, look at 2f9132e2e0b1e01398a01a32972e87f45ec2f7a6, we
> were shipping 3.4.24 before the removal, not 3.4.10.

It is exactly what I am complaining!  It is not possible to follow.

The version before the March 10th is 3.4.10.  This version is free and
from 2017; with security vulnerabilities but everything is fine.

Then less than 6 days later, the package is updated to 3.4.24 which is a
non-free version.  So reverted to 3.4.10.  So re-reverted to 3.4.24.
And last, removed.

It shows exactly my point.  The correct and polite way of doing the
thing is first to examine the issue at hand (3.4.10 is old with security
vulnerabilities), then propose a fix (e.g., the removal), wait feedback,
and complete.

Whatever, now it is done.  And as I said, I am not against the removal.


All the best,
simon

Re: Why [bug#47081] Remove mongodb?

[Léo Le Bouter]

The issue with 3.4.24 / 3.4.10 is that Efraim reverted the commit then
it was briefly discussed on IRC and Efraim thought I was right about
the licensing being fine on 3.4.24 and reverted their revert commit,
after some actual checking in the tarball grepping for license headers
I found out I was wrong and instead of reverting the revert of the
revert of Efraim the next change was removal because of other reasons.

Besides the openssl issue I think the commit message laid out these
things quite well.


signature.asc
Description: This is a digitally signed message part

Re: armhf-linux substitutes

[zimoun]

Hi,

On Wed, 17 Mar 2021 at 18:28, Ludovic Courtès  wrote:

> Mathieu, what’s preventing us from doing armhf-linux builds again?  We
> could use the OverDrives for that (with an upper bound though), along
> with the SBCs in machines-for-berlin.scm.

We should start to build ASAP…

> That won’t be enough to keep up, so perhaps we’ll have to restrict
> armhf-linux builds to the “core” subset or the release-manifest.scm.

…to have the time to effectively build and probably fix this at least
“core” subset.


Naive question, how can I emulated without X this architecture?


Cheers,
simon

Re: Why [bug#47081] Remove mongodb?

[Léo Le Bouter]

On Wed, 2021-03-17 at 19:51 +0100, zimoun wrote:
> It shows exactly my point.  The correct and polite way of doing the
> thing is first to examine the issue at hand (3.4.10 is old with
> security
> vulnerabilities), then propose a fix (e.g., the removal), wait
> feedback,
> and complete.

Actually we did not know pushing a security fix with 3.4.24 was not
fine, from quick auditing I have made 3.4.24 would still be under AGPL
so it would be fine to upgrade, turns out not since some files inside
are under SSPL but that was discovered way later, even when Efraim had
doubt and reverted my commit we had a debate and Efraim bought my
arguing even though I was wrong and they were right, if for every
security issue I have to ask feedback I may not ship them in a timely
manner, so that's also why they tend to be pushed faster than usual..
we may want to establish a clear process here. I usually create issues
for things I need help on, if I can do it myself and feel confident, I
just push, I can be wrong of course and always sorry for issues, I fix
them shortly in next commits if any.


signature.asc
Description: This is a digitally signed message part

Re: Why [bug#47081] Remove mongodb?

[zimoun]

On Wed, 17 Mar 2021 at 20:11, Léo Le Bouter  wrote:
> On Wed, 2021-03-17 at 19:51 +0100, zimoun wrote:
>> It shows exactly my point.  The correct and polite way of doing the
>> thing is first to examine the issue at hand (3.4.10 is old with
>> security
>> vulnerabilities), then propose a fix (e.g., the removal), wait
>> feedback,
>> and complete.
>
> Actually we did not know pushing a security fix with 3.4.24 was not
> fine, from quick auditing I have made 3.4.24 would still be under AGPL
> so it would be fine to upgrade, turns out not since some files inside
> are under SSPL but that was discovered way later, even when Efraim had

Later means here only hours.

> doubt and reverted my commit we had a debate and Efraim bought my
> arguing even though I was wrong and they were right, if for every
> security issue I have to ask feedback I may not ship them in a timely
> manner, so that's also why they tend to be pushed faster than usual..

Haste is not speed.

> we may want to establish a clear process here. I usually create issues
> for things I need help on, if I can do it myself and feel confident, I
> just push, I can be wrong of course and always sorry for issues, I fix
> them shortly in next commits if any.

I really appreciate your valuable work. I have the impression you think
that you have to push as fast as you can, whatever if it is the right
fix.  If I might, first please avoid to burn out and second do not
worry, the world will not explode because of a security vulnerability in
Guix.  Maybe one day when Guix will dominate the world, soon! :-)

I am not convinced that the regular Guix user is upgrading their package
set twice a day; maybe once a week at best and more probably time to
time.  Guix is rooted in The Right Thing™ and sometimes it means delay
to think what the right thing really is.  Therefore, the process is
already clear: go via guix-patch for non-trivial changes and wait
feedback.

At the end, I cannot express better what Tobias wrote:

   

or Leo:

   
   

All the best,
simon

Re: Are gzip-compressed substitutes still used?

[zimoun]

Hi Vagrant,

On Wed, 17 Mar 2021 at 11:08, Vagrant Cascadian  wrote:

> ... and I would expect this version to ship in Debian for another ~3-5
> years, unless it gets removed from Debian bullseye before the upcoming
> (real soon now) release!

I could miss a point.  In 3-5 years, some people will be still running
Debian stable (or maybe oldstable or maybe this stable is LTS), so they
will “apt install guix” at 1.2.0, right?  But then there is no guarantee
that Berlin will still serve this 5 years old binary substitutes.  But
“guix install” fallback by compiling what is missing, right?  Then the
question will be: are the upstream sources still available?  Assuming
that SWH is still alive at this future, all the git-fetch packages will
have their source, whatever the upstream status.  For all the other
methods, there is no guarantee.

On the other hand, at this 3-5 years future, after “apt install guix”,
people will not do “guix install” but instead they should do “guix
pull”.  Therefore, the compression of substitutes does not matter that
much, right?

The only strong backward compatibility seems between “guix pull” rather
than all the substitutes themselves.  Isn’t it?  Other said, at least
keep all the necessary to have “guix pull” at 1.2.0 be able to complete.


Thanks for this opportunity to think at such time scale. :-)


Cheers,
simon