Re: Generating compiled scm (.go) files as part of LilyPond build
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/01/11 11:21, Andy Wingo wrote: > Hi Ian, > > [re-adding the list; please keep the list CC'd on all guile-related > topics] > > On Fri 28 Jan 2011 22:37, Ian Hulin writes: > >> Are you going to fix the double extension bug for defaulted/cached >> output names? I.e. compiling c++.scm will produce an output >> filename of c++.scm.go in the cache. > > As far as I'm concerned this isn't a bug. It only happens for writes > to the fallback path. Appending an extension instead of replacing > an extension makes the algorithm more robust (e.g, doesn't confuse > `foo' and `foo.scm'. > It may boil down to a matter of taste, but I find double and triple extensions on a filename intrinsically nasty. I've normally come across them before on Windows systems where a filename such as thing.htm.exe usually means there's malware or a trojan or suchlike on your system. See also comments below. >>> Basically compiled-file-name doesn't do any path searches. It >>> simply computes a place in ~/.cache in which to cache the result >>> of compiling FILE. >>> >> Yuk! This means we will need to have custom routines in our code to >> work out and specify the output path we want. > > There is not a DWIM answer, for various reasons -- but the > combinatorics between %load-path, %load-compiled-path, > %load-extensions, %load-compiled-extensions, and the actual `load' > procedure should give you an idea. The autocompilation strategy is > as good as we can get without you actually telling Guile where you > want your files. > >> It also means that autocompilation is a hindrance rather than a >> help, since it hi-jacks the compiled files into the cache. > > What is the problem with writing to the cache? It is a cache. It's > doing it's job. > The problem is knowing where the cache is. For Lilypond, we need to have a common root directory off of which we can hang the compiled files in scm/out. I've been able to find out from Guile V2.02 that the cache root can be controlled by an environment variable XDG_CACHE_HOME, which is defaulted by Guile to $HOME/.cache if not defined. The documentation also says that compiled files for will be placed in $XDG_CACHE_HOME/guile/ccache but this is what it actually does: ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD GNU Guile 2.0.2 Copyright (C) 1995-2011 Free Software Foundation, Inc. Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. This program is free software, and you are welcome to redistribute it under certain conditions; type `,show c' for details. scheme@(guile-user)> (load "testing.scm") ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0 ;;; or pass the --no-auto-compile argument to disable. ;;; compiling /home/ian/src/Guile/guile-2.0.2/testing.scm ;;; compiled /home/ian/src/Guile/guile-2.0.2/cache/guile/ccache/2.0-LE-4-2.0/home/ian/src/Guile/guile-2.0.2/testing.scm.go scheme@(guile-user)> The output filename looks like it's derived like this The first section of the path "/home/ian/src/Guile/guile-2.0.2/cache" is controlled by XDG_CACHE_HOME or defaulted, but instead of $HOME/.cache it defaults to $PWD/cache here. The first two sections of the path /home/ian/src/Guile/guile-2.0.2/cache/guile/ccache/2.0-LE-4-2.0 is controlled by %compile-fallback-path using the value derived or defaulted from $XDG_CACHE_HOME for the first section. The third section of the path /home/ian/src/Guile/guile-2.0.2/ is the absolute path of the incoming filename concatenated onto the first two sections. This is unhelpful for what we want to do in Lilypond. If the first two sections are of the path point to a cache base directory, at this point section three of the path should be derived from the *relative* path in the incoming filename. testing.scm.go Lose the .scm bit, it also is unhelpful. If the compiled-file-name used the incoming relative path for the third section we could do the following in our initialization code before calling guile ($LILY_DATADIR is the the root LilyPond directory). 1. Insert $LILY_DATADIR/scm at the head of %load-path and $LILY_DATADIR/scm/out at the head of %load-compiled-path 2. setenv $XDG_CACHE_HOME to $LILY_DATADIR 3. set %compile-fallback-path to $LILY_DATADIR If the double extension "feature" was also removed, Then the equivalent of (load "testing.scm") with auto-compilation on would generate $LILY_DATADIR/scm/testing.go It wouldn't give us _quite_ what we want as the compiled files would be in the same directory as the source files, but we could live with that if it allows us either to use auto-compile, or to rely in Guile's in-built check of the the source and compiled file's modified times. >>> Load (system base compile) then, no? >>> >> OK, this is another V1.8/V2.0 conditional kludge we need to put in >> the LilyPond initialization code. > > Compiling files is a 2.0 thing, of course... > It looks like (compiled-file-name) uses a hard-
Re: Generating compiled scm (.go) files as part of LilyPond build
Hi Ian,
On Tue 19 Jul 2011 15:18, Ian Hulin writes:
> It may boil down to a matter of taste, but I find double and triple
> extensions on a filename intrinsically nasty. I've normally come across
> them before on Windows systems where a filename such as thing.htm.exe
> usually means there's malware or a trojan or suchlike on your system.
> See also comments below.
Consider it an opaque key into a cache. We could change in the future
to default to the SHA1 of the whole path. That does have some
advantages regarding flattening the directory hierarchy, resulting in
fewer stat operations.
> ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD
This will set XDG_CACHE_HOME=${top_builddir}/cache.
> The problem is knowing where the cache is. For Lilypond, we need to have
> a common root directory off of which we can hang the compiled files in
> scm/out.
Why do you care? (Honest question.)
To me there are two cases:
1) You compile the files yourself. You either ensure that the .go
files end up in the right places, or adjust
GUILE_LOAD_COMPILED_PATH.
2) You rely on autocompilation. In this case there is no guarantee
about where the files go, besides residing in the
XDG_CACHE_DIR/guile.
Regards,
Andy
--
http://wingolog.org/
Re: Generating compiled scm (.go) files as part of LilyPond build
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Tue 19 Jul 2011 15:28:41 BST, Andy Wingo wrote:
> Hi Ian,
>
> On Tue 19 Jul 2011 15:18, Ian Hulin writes:
>
>> It may boil down to a matter of taste, but I find double and
>> triple extensions on a filename intrinsically nasty. I've normally
>> come across them before on Windows systems where a filename such as
>> thing.htm.exe usually means there's malware or a trojan or suchlike
>> on your system. See also comments below.
>
> Consider it an opaque key into a cache. We could change in the
> future to default to the SHA1 of the whole path. That does have
> some advantages regarding flattening the directory hierarchy,
> resulting in fewer stat operations.
>
Sorry, but how does this benefit LilyPond as a user of the Guile API?
I'm used to viewing file extensions as an indication of the format of
the file content. In our simple view scheme sources are *.scm, and
compiled scheme files are *.go.
Guile documentation implies something similar (Chap 4 Environment
Variable sections on GUILE_AUTO_COMPILE and GUILE_LOAD_COMPILED_PATH)
"If a compiled (‘.go’) file corresponding to a ‘.scm’ file is not found
or is not newer than the ‘.scm’ file, the ‘.scm’ file will be compiled
on the fly, and the
resulting ‘.go’ file stored away. An advisory note will be printed on
the console."
"GUILE_LOAD_COMPILED_PATH
This variable may be used to augment the path that is searched for
compiled Scheme files (‘.go’ files) when loading"
Neither of these sections leads the reader to expect an actual file name
generated to be
;;; compiled
/long/path/name/testing.scm.go
^^^
>> ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD
>
> This will set XDG_CACHE_HOME=${top_builddir}/cache.
>
The Guile Reference Manual Chapter 4 p 38 specifically says this
defaults to $HOME/cache
>> The problem is knowing where the cache is. For Lilypond, we need to
>> have a common root directory off of which we can hang the compiled
>> files in scm/out.
>
> Why do you care? (Honest question.)
>
(Honest answer)
Because if Lilypond is run up with the Guile default of --auto-compile
it will check for and generate compiled files in a different place to
where we want to have produced them at build-time. Also once for a
project like LilyPond these files would need to be in a shared directory
on the file system. Relying on the default compiled file spec would have
a set of cached files somewhere on each user's cache in their home
directory.
> To me there are two cases:
>
> 1) You compile the files yourself. You either ensure that the .go
> files end up in the right places, or adjust
> GUILE_LOAD_COMPILED_PATH.
>
But GUILE_LOAD_COMPILED_PATH setting won't prevent the vanilla
compiled-file-name from checking in the wrong place for at compile time.
OK we can brew our own expectation of where the file will be, and
provide our own ly:load and intelligence to pick up the compiled file,
but we have to do this in conjunction with a
scm_putenv("GUILE_AUTO_COMPILE=0") call in our initialization code.
> 2) You rely on autocompilation. In this case there is no guarantee
> about where the files go, besides residing in the
> XDG_CACHE_DIR/guile.
>
Looks like this is a no-no. We can't control what the files are called,
or if they'd be recognized once they've been moved to their location
after /make install/ for LilyPond. Also there's the issue of embedded
Scheme statements in Lily source files (which could contain their own
(load "") statements).
It's a real shame because it looks like auto-compile is a feature
designed to make life easy for Guile users (including layered projects
like LilyPond). Unfortunately its default settings prevent us from using
it. :-(.
I've also got a question re the (include) procedure which Ludovic
mentioned in a post on guile-users. I'll start a different thread for
that one.
Thanks for the help so far,
Cheers,
Ian
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJOJg5cAAoJEBqidDirZqASMj0H/2jn/jiNG+MtT+YEc762baxL
8pMC4ijNMwWEnMwzuWm1LHhHiJqG5roxvinPoe5bjeC8RBNSMOW7SKGkJ+ao5pTK
vYfBm55WqdpAvAVQA9OaDkL6J33TEa6z1ioNYxj2yFOHHsj0HHVnhUM29RDNYgmw
9duK9lKtv37AGT0W9aj6bGx8FRGFUOFgRUnHMCtn7usE1DboXQgjaY3IQ46XYs6n
XGfuFz1JBXqhKEjjK7Fbu5M44A0eD2yWGJ01K3eVQqB1OmIrz7zy7wU495Wqd4im
/eop+BCJRnpRAdHIR0y/LgrrlPSpD+mulBk9BOuoAglXM6N8spaQQY/mj8MkrKU=
=igTg
-END PGP SIGNATURE-
Guile V2.0 (include) procedure.
In another post Ludo mentioned an as yet undocumented (include) procedure. How does it differ from (load)? Could LilyPond add a stub for use with V1.8 (as we did for (module_export_all!) )? What's the calling interface? How does it interact with %load-path and %load-compiled-path? How is it affected by GUILE_AUTO_COMPILE? Cheers, Ian Hulin
