Re: merge wip-haskell?

[Jakub Kądziołka]

On Thu, Aug 06, 2020 at 10:13:46AM +0200, Ricardo Wurmus wrote:
> Number 4 is by far the ugliest change of them all.  In order to
> statically link packages we need to add all the “static” outputs of all
> Haskell inputs *and* the “static” outputs of *their* Haskell inputs.
> This is not easily accomplished, so I ended up using “package-closure”
> on all direct inputs, and then filtered the result to packages with
> names starting with “ghc-”.  If there was a more appropriate tool I’d
> use it, but I don’t think it exists.

Perhaps we should work on making propagated-inputs per-output? That way,
:static could propagate the :static output of the dependencies.

This would also be useful in other situations. For example, a package
might contain both a binary and a library, and the library must
propagate its dependencies to make the header files work.

I don't know what a good syntax for this would be.

Thoughts?

Regards,
Jakub Kądziołka


signature.asc
Description: PGP signature

Re: merge wip-haskell?

[John Soo]

Hi Jakub,

I could see splitting the static output being useful but I would rather wait 
until some evidence that the closure size would be too large. Also I’m not sure 
propagation is necessary for dependents to find libraries or use paths from an 
input.

Thoughts?

John

On Aug 7, 2020, at 8:04 AM, Jakub Kądziołka  wrote:

On Thu, Aug 06, 2020 at 10:13:46AM +0200, Ricardo Wurmus wrote:
> Number 4 is by far the ugliest change of them all.  In order to
> statically link packages we need to add all the “static” outputs of all
> Haskell inputs *and* the “static” outputs of *their* Haskell inputs.
> This is not easily accomplished, so I ended up using “package-closure”
> on all direct inputs, and then filtered the result to packages with
> names starting with “ghc-”.  If there was a more appropriate tool I’d
> use it, but I don’t think it exists.

Perhaps we should work on making propagated-inputs per-output? That way,
:static could propagate the :static output of the dependencies.

This would also be useful in other situations. For example, a package
might contain both a binary and a library, and the library must
propagate its dependencies to make the header files work.

I don't know what a good syntax for this would be.

Thoughts?

Regards,
Jakub Kądziołka

Re: merge wip-haskell?

[Jakub Kądziołka]

On Fri, Aug 07, 2020 at 08:12:36AM -0700, John Soo wrote:
> I would rather wait until some evidence that the closure size would be too 
> large. Also I’m not sure propagation is necessary for dependents to find 
> libraries or use paths from an input.

Ricardo already explained that this is indeed the case.


signature.asc
Description: PGP signature

Re: merge wip-haskell?

[Ricardo Wurmus]

Jakub Kądziołka  writes:

> On Fri, Aug 07, 2020 at 08:12:36AM -0700, John Soo wrote:
>> I would rather wait until some evidence that the closure size would be too 
>> large. Also I’m not sure propagation is necessary for dependents to find 
>> libraries or use paths from an input.
>
> Ricardo already explained that this is indeed the case.

Yes, and for the case of pandoc it’s significant.  The closure of
ghc-pandoc is >3GiB right now and with the changes it’s <200MiB.  This
affects lots of R packages that need Rmarkdown, and lots of bioinfo
packages.

There's no doubt that moving the static libs to their own output has a
significant impact on closure size.

-- 
Ricardo

Re: merge wip-haskell?

[John Soo]

Hi Ricardo and Jakub,

Ah ok. Sorry I had forgotten the point of the thread.  Sounds like a plan!

- John

On Aug 7, 2020, at 8:59 AM, Ricardo Wurmus  wrote:


Jakub Kądziołka  writes:

>> On Fri, Aug 07, 2020 at 08:12:36AM -0700, John Soo wrote:
>> I would rather wait until some evidence that the closure size would be too 
>> large. Also I’m not sure propagation is necessary for dependents to find 
>> libraries or use paths from an input.
> 
> Ricardo already explained that this is indeed the case.

Yes, and for the case of pandoc it’s significant.  The closure of
ghc-pandoc is >3GiB right now and with the changes it’s <200MiB.  This
affects lots of R packages that need Rmarkdown, and lots of bioinfo
packages.

There's no doubt that moving the static libs to their own output has a
significant impact on closure size.

-- 
Ricardo

Re: merge wip-haskell?

[Ricardo Wurmus]

Jakub Kądziołka  writes:

> On Thu, Aug 06, 2020 at 10:13:46AM +0200, Ricardo Wurmus wrote:
>> Number 4 is by far the ugliest change of them all.  In order to
>> statically link packages we need to add all the “static” outputs of all
>> Haskell inputs *and* the “static” outputs of *their* Haskell inputs.
>> This is not easily accomplished, so I ended up using “package-closure”
>> on all direct inputs, and then filtered the result to packages with
>> names starting with “ghc-”.  If there was a more appropriate tool I’d
>> use it, but I don’t think it exists.
>
> Perhaps we should work on making propagated-inputs per-output? That way,
> :static could propagate the :static output of the dependencies.
>
> This would also be useful in other situations. For example, a package
> might contain both a binary and a library, and the library must
> propagate its dependencies to make the header files work.
>
> I don't know what a good syntax for this would be.

The binary / library problem is particularly interesting to me as we
have a few Python bioinfo tools that also work as libraries.  When used
as a library we need propagation, but when uses as command line tools
propagation is not needed (and can be harmful due to the effects on
PYTHONPATH).

Syntax seems less important than verifying that this actually works,
though.  Alas, as someone who has committed and encouraged syntactic
crimes with excessive reader macros, I think my vote on syntax carries
no weight :)

-- 
Ricardo

Re: merge wip-haskell?

[Jakub Kądziołka]

On Fri, Aug 07, 2020 at 06:27:21PM +0200, Ricardo Wurmus wrote:
> 
> Jakub Kądziołka  writes:
> 
> > On Thu, Aug 06, 2020 at 10:13:46AM +0200, Ricardo Wurmus wrote:
> >> Number 4 is by far the ugliest change of them all.  In order to
> >> statically link packages we need to add all the “static” outputs of all
> >> Haskell inputs *and* the “static” outputs of *their* Haskell inputs.
> >> This is not easily accomplished, so I ended up using “package-closure”
> >> on all direct inputs, and then filtered the result to packages with
> >> names starting with “ghc-”.  If there was a more appropriate tool I’d
> >> use it, but I don’t think it exists.
> >
> > Perhaps we should work on making propagated-inputs per-output? That way,
> > :static could propagate the :static output of the dependencies.
> >
> > This would also be useful in other situations. For example, a package
> > might contain both a binary and a library, and the library must
> > propagate its dependencies to make the header files work.
> >
> > I don't know what a good syntax for this would be.
> 
> The binary / library problem is particularly interesting to me as we
> have a few Python bioinfo tools that also work as libraries.  When used
> as a library we need propagation, but when uses as command line tools
> propagation is not needed (and can be harmful due to the effects on
> PYTHONPATH).
> 
> Syntax seems less important than verifying that this actually works,
> though.

One still needs to pick a variant, even at random, to begin
implementation, which is hard when no options have been suggested :)

I suppose that we could add a list of outputs each propagated-input
applies to, putting it after the package and its output?

(this is unambiguous as we can check whether the third element of the
list is a string, or a list)

(propagated-inputs
 `(("foo-static" ,foo "static" ("static"))
   ("bar" ,bar ("static"

However, this has the issue of being somewhat ad-hoc, and other
extensions to the list of inputs might become troublesome. This
motivates my second proposal:

(propagated-inputs
 `(("foo-static" ,foo "static" #:propagate-to ("static"))
   ("bar" ,bar #:propagate-to ("static"

On the other hand, this variant feels somewhat over-engineered. After
all, we can always change the syntax if it becomes actually necessary.

There's also the option of adding another field to the package record:

(propagated-inputs-for-outputs ;; name up to bikeshedding :)
 `(("static"
(("foo-static" ,foo "static")
 ("bar ,bar)

> Alas, as someone who has committed and encouraged syntactic
> crimes with excessive reader macros, I think my vote on syntax carries
> no weight :)

Well, the number of reader macro crimes encouraged seems to correlate
with experience, so I would still like to ask for your opinion.

Regards,
Jakub Kądziołka


signature.asc
Description: PGP signature

Re: merge wip-haskell?

[Timothy Sample]

Hi John and Ricardo,

John Soo  writes:

> I believe there was also some work being done to de-duplicate flags
> sent to gcc sent by ghc (this was the only thing keeping stack from
> building).
> I hope that can make it in, too!

I agree, and sorry to you John for the delay.  I’m not using much
Haskell these days, so I’m having trouble keeping Haskell stuff on my
radar.  The last patch I sent to  is
about as good as I can come up with, so I will apply it to the branch
before the merge.

Ricardo, I will try and do this tonight or tomorrow, and will look over
your changes then as well.

> If there is any way we can bump the default ghc to 8.8.x, that would
> be excellent, too.  I think something like that makes sense on staging
> though.

I would love for this to happen, but I think it would needlessly delay
Ricardo’s changes.  Fortunately, one can build all of our Haskell
packages overnight on a modest computer, so it’s no big deal to merge
the branch now and then merge the GHC 8.8.x updates whenever they’re
ready.

Also, it looks like “wip-haskell-updates” is no longer being built by
the CI infrastructure.  Since the branch triggers a rebuild of all the
Haskell packages, it should be built before merging, right?


-- Tim

Re: merge wip-haskell?

[Ricardo Wurmus]

Timothy Sample  writes:

> Also, it looks like “wip-haskell-updates” is no longer being built by
> the CI infrastructure.  Since the branch triggers a rebuild of all the
> Haskell packages, it should be built before merging, right?

Yes, I’ll rebase it on top of “master” and add the specification to
ci.guix.gnu.org well before the merge.

-- 
Ricardo

Why does %build-inputs contain the transitive closure of inputs?!

[Jakub Kądziołka]

Consider this package:

(use-modules
  (guix packages)
  (guix build-system trivial)
  (gnu packages version-control))

(define foo
  (package
(name "foo")
(version "0")
(source #f)
(build-system trivial-build-system)
(arguments
 `(#:builder
   (begin
 (display %build-inputs)
 #t)))
(inputs
 `(("libgit2" ,libgit2)))
(synopsis "")
(description "")
(home-page "")
(license #f)))

When we try building it, it prints:

((libgit2 . /gnu/store/hlcfmg5n29ynf354x7s4s7bkxswrmkq4-libgit2-1.0.1) (zlib . 
/gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11) (pcre2 . 
/gnu/store/ckpchpflfwhag5qsfk8klvmihwzqg9rp-pcre2-10.35) (openssl . 
/gnu/store/dkzivzn17qilmqdfpyps62b395wxhshh-openssl-1.1.1f))

Why? I would expect only libgit2 to be present, and not all of its
dependencies. It seems to me that this has only downsides - the entire
transitive closure needs to be present during build, even when the
actually used dependencies don't reference it, and the names of inputs
used by packages are now public API, and changing them can break the
build of any transitively dependent package.

I've traced this back to bag->derivation's use of
bag-transitive-inputs, but unfortunately git blame didn't hold any hints
- it has been this way ever since bag->derivation was first implemented.

Am I missing something, or is this a bug?

Regards,
Jakub Kądziołka


signature.asc
Description: PGP signature

Re: Why does %build-inputs contain the transitive closure of inputs?!

[Carlo Zancanaro]

Hi Jakub,

On Sat, Aug 08 2020, Jakub Kądziołka wrote:
Why? I would expect only libgit2 to be present, and not all of 
its dependencies. ...


If you take a look at the definition for libgit2 in 
gnu/packages/version-control.scm you'll see that this isn't all 
the dependencies for libgit2, it's only the propagated-inputs. For 
instance, libssh2 and http-parser are both inputs for libgit2, but 
they don't appear in your list because they're not propagated.


... It seems to me that this has only downsides - the entire 
transitive closure needs to be present during build, even when 
the actually used dependencies don't reference it, and the names 
of inputs used by packages are now public API, and changing them 
can break the build of any transitively dependent package.


The way propagated inputs work is that they're installed alongside 
the package, so in some sense they are part of the public API of 
the package. They should be used with care, but as far as I can 
tell everything is working properly here.


Carlo

Re: Why does %build-inputs contain the transitive closure of inputs?!

[Jakub Kądziołka]

On Sat, Aug 08, 2020 at 10:26:08AM +1000, Carlo Zancanaro wrote:
> Hi Jakub,
> 
> On Sat, Aug 08 2020, Jakub Kądziołka wrote:
> > Why? I would expect only libgit2 to be present, and not all of its
> > dependencies. ...
> 
> If you take a look at the definition for libgit2 in
> gnu/packages/version-control.scm you'll see that this isn't all the
> dependencies for libgit2, it's only the propagated-inputs. For instance,
> libssh2 and http-parser are both inputs for libgit2, but they don't appear
> in your list because they're not propagated.
> 
> > ... It seems to me that this has only downsides - the entire transitive
> > closure needs to be present during build, even when the actually used
> > dependencies don't reference it, and the names of inputs used by
> > packages are now public API, and changing them can break the build of
> > any transitively dependent package.
> 
> The way propagated inputs work is that they're installed alongside the
> package, so in some sense they are part of the public API of the package.
> They should be used with care, but as far as I can tell everything is
> working properly here.

You're absolutely right. This even explains the original weirdness I
saw. Serves me right for hacking at 1 AM...

Sorry for the noise :/


signature.asc
Description: PGP signature