On Mon, May 9, 2011 at 2:57 PM, Ondrej Certik <ond...@certik.cz> wrote: > On Mon, May 9, 2011 at 9:11 AM, Robert Bradshaw > <rober...@math.washington.edu> wrote: >> On Mon, May 9, 2011 at 3:05 AM, Ondrej Certik <ond...@certik.cz> wrote: >>> On Sat, May 7, 2011 at 9:41 PM, Robert Bradshaw >>> <rober...@math.washington.edu> wrote: >>>> On Fri, May 6, 2011 at 2:21 PM, Ondrej Certik <ond...@certik.cz> wrote: >>>>> On Fri, May 6, 2011 at 2:19 PM, Ondrej Certik <ond...@certik.cz> wrote: >>>>>> On Fri, May 6, 2011 at 11:26 AM, Robert Bradshaw >>>>>> <rober...@math.washington.edu> wrote: >>>>>> [...] >>>>>>> Were I to design the system from scratch, I'd put >>>>>>> all our code (devel/scripts/...) in a single repo, along with the >>>>>>> top-level files, and a list of dependencies (spkgs). Building sage >>>>>>> would fetch (locally or remotely) the dependencies listed and build >>>>>>> them in such a way that changing the list of dependencies and >>>>>>> re-building would easily and cheaply reversible. I would probably >>>>>>> still build my own Python, but may require it (flexible version) as a >>>>>>> bootstrapping prerequisite. Whether the non-upstream parts of an spkg >>>>>>> belong in the spkgs or the main repo, I'm not sure, but I'd rather >>>>>>> *everything* be expressed as commit to a single repository (possibly >>>>>>> moving a pointer to some new, vanilla upstream source, rather than >>>>>>> putting all upstream sources in our repo). >>>>>>> >>>>>>> If others have similar views, maybe we could move in that direction. >>>>>> >>>>>> I actually developed such a system from scratch (it's called Qsnake: >>>>>> http://qsnake.com) and pretty much followed your paragraph above, so I >>>>>> pushed all the repos at github: >>>>>> >>>>>> https://github.com/qsnake >>>> >>>> Cool. >>>> >>>>>> for example the Cython repo is here: >>>>>> >>>>>> https://github.com/qsnake/cython >>>>>> >>>>>> it is actually a fork of the official Cython repo. And then Qsnake is >>>>>> clever enough, that when it is fetching the sources, and if there is >>>>>> setup.py and no spkg-install, it creates spkg-install automatically >>>>>> with "setup.py install" (and other default stuff), and then saves the >>>>>> cython.spkg package into the spkg/standard/ directory. >>>>>> >>>>>> Having all spkg-install scripts in the main repository (so in my case >>>>>> in this repository: https://github.com/qsnake/qsnake) is a valid idea, >>>>>> that I have been bouncing around too. I decided not to, to keep things >>>>>> localized, as sometimes nontrivial modifications are needed to the >>>>>> upstream packages, and the best way to do such modifications is to >>>>>> simply create couple git patches. Also for testing, if there is >>>>>> spkg-install file, committed, I can checkout the git repo for a >>>>>> particular package and >>>>>> >>>>>> qsnake install . >>>>>> >>>>>> and it will install the package. And I can debug it easily. >>>> >>>> Is it a reversible install? >>> >>> No, currently it's just like in Sage. For reversible install, one >>> would have to redo every single package to be able to take things from >>> SPKG_LOCAL (=SAGE_LOCAL in Sage), but install into something like >>> SPKG_INSTALL, so that one can package it and keep track of files. >> >> I could see this done with paths, with every spkg installing into its >> one versioned directory, and the python/shell/include/library paths >> set up to have the list of all currently enabled spkgs. > > I was thinking about this a lot yesterday, and there are a lot more > issues to resolve, than it seems at first sight. In particular, some > packages like Python is doing some recompiling of modules (I think, > but maybe I am wrong) and some other stuff to the place where it is > installed. Some other packages (setuptools?) are modifying some stuff > as well (at least I read it somewhere). > > Pretty much, as long as the "installation" is just copying of files, > then it should work. But if you also need to modify some stuff after > installing it (post install script in Debian/Ubuntu), then things > become more complex. > > With our current approach, one is free to do any kind of necessary > tweaks in $SPKG_LOCAL (=SAGE_LOCAL) to make things work. Usually by > the build system of the package itself.
True, but I can't think of anything in Sage where one needs to modify the environment any more than put a file somewhere that it's accessible (though such a thing could be possible). I agree a general solution is much more subtle. >>> Hand in hand with this go "binary packages". If one can do >>> "uninstall", then immediately one can start creating automatic binary >>> packages. That would be super cool. >>> >>> I am currently undecided whether to go with this or not. No doubt it >>> would be useful. If Sage goes this way, than surely we'll follow too. >>> Otherwise probably not, as compatibility is important. People who know >>> Sage should find it quite easy to play with Qsnake and vice versa. >>> Learning a completely new system and how it works is an obstacle. >>> >>> Adding uninstall and binary packages will make everything more complex >>> (imagine installing the wrong binary into incompatible base >>> install....). Right now it's all just source packages, and possibly >>> one binary for the whole thing (for each platform), which is >>> manageable. >>> >>>> What about concurrent versions? >>> >>> Only if you rename the package. In this I have the same (or similar) >>> vision as Sage, that is to create a well tested scientific environment >>> with just one tested version of each package, that works with >>> everything. >> >> Yes, that's what I'm thinking, but it'd be nice to be able to, e.g., >> try out a new spkg without hosing the entire install (in a possibly >> irreversible manner). >> >>> If there are two incompatible versions, then one should >>> change the name of the package, e.g. python -> python2 + python3. Then >>> it can live side by side. >> >> Or python2.6p10. >> >>>> What about >>>> dependencies (e.g. if the version of Cython was updated, would all the >>>> stuff depending on Cython get re-compiled? >>> >>> No. Only the other way round -- if you want to install "phaml", that >>> uses Cython to wrap Fortran, it will also pull in "cython" >>> automatically (as well as all the other packages). However, since the >>> dependency tree is known, we can add this feature as well, to >>> recompile all "rdepends" (reverse dependencies, to use Debian >>> terminology). >> >> That would be useful to get the known-stable environment, regardless >> of the order you install packages. That's one of the things that's so >> appealing about http://nixos.org/nix/ >> >>>>> Oh, and then people can simply send pull requests to the respective >>>>> packages directly. So I think it solves lots of the problems raised in >>>>> this thread. For sage, it would have to move to github though, so it >>>>> might not be an option. >>>> >>>> Probably not. Is it really tied to github, or could any dvcs be plugged in? >>> >>> I just made two releases last couple days, one can download it from here: >>> >>> https://github.com/qsnake/qsnake/archives/master >>> >>> and that is just one big source tarball, and installs completely >>> locally, no git is needed (it will actually install its own >>> automatically --- but it is not needed for the build system). Git is >>> only used to create the big source tarball using "qsnake -d". It can >>> be easily changed to "hg" in the build system (and then one needs to >>> move all the packages, which is simple but tedious). >> >> I was thinking of something even simpler, where the dependent packages >> would just be tarballs (at least as an option), so no revision control >> is needed by the build system at all. > > You can download tarballs from github, for example for the Qsnake's > cython package: > > https://github.com/qsnake/cython/archives/master > > without having git installed. So in principle the build system can use > it too. Ah, yes. > I just stick with git for now. Nothing against github (in fact I really like it), it's just that I'm wary of making my infrastructure heavily dependent on someone else's for some things. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
