On Mon, Feb 20, 2012 at 6:45 AM, Keshav Kini <keshav.k...@gmail.com> wrote: > Robert Bradshaw <rober...@math.washington.edu> writes: >> On Fri, Feb 17, 2012 at 1:09 AM, Keshav Kini <keshav.k...@gmail.com> wrote: >>> Robert Bradshaw <rober...@math.washington.edu> writes: >>> In the category of "glue code" I meant to include everything in >>> $SAGE_LOCAL/bin/sage-*. I see much of that stuff as more related to >>> maintenance of the entire distribution of software we ship than to the >>> actual Python/Cython code in the Sage library. Of course, if we do >>> switch to Prefix or something like it, most of it will become >>> unnecessary, or can actually be fit into the portage data tree as >>> "Tools", so I see your point. >> >> Yeah, tools describes it well (or at least what remains after the >> package management stuff is gone, like testing and coverage and >> stuff). > > Well, I meant that there is a "Tools" directory in sage-on-gentoo. I'm > not sure if this is a standard thing in the directory structure of a > Gentoo Prefix tree, or, if it is, what sort of stuff is supposed to go > there, but I see some utility scripts, presumably written by François or > Christopher, in there and was suggesting we might be putting our sage-* > scripts there too. But I don't know enough about Prefix to be sure > whether that makes sense. > >>>> I think Sage will be monolithic and Windows be VM for the near future >>>> at least, with a larger percentage of people using a Sage install "in >>>> the cloud" on a university or otherwise hosted server for the near-mid >>>> term future. But as you said who knows... >>> >>> Yes, I agree. That seems like the most likely future, at the moment. >>> However, William has asked me to write a Sage Enhancement Proposal for >>> switching to git, and I think we concluded on IRC that it might make >>> sense to make a long term timeline for other big changes as well, such >>> as what we're talking about now - or at least a proposal for such a >>> timeline :) >> >> +1, which is why we should be having this discussion now. Switching to >> git could probably happen a lot faster than revamping the build >> system, but at least we could consolidate some of the repositories and >> re-seat it at a higher level in the directory structure so moving to a >> new build system is "just a patch" :). > > Absolutely. Switching to git is much easier than revamping the build > system. The only reason I'm mentally grouping them together is that > consolidating SPKGs, which would be part of remaking the build system as > I've described, is also related to version control. > > But that actually doesn't make a lot of sense now that I think about it. > We would basically be throwing away the old SPKGs' files, so I don't > know if it's even really worth preserving their history in the portage > tree of Prefix, assuming we move to Prefix - we could easily preserve > their history somewhere else instead. > > Of course, even if it's not really related to the switch to git, I'd > definitely like to keep a build system revamp on the horizon!
Yep, my thoughts exactly. >> Thanks for filling me in. I was aware of the sage-on-gentoo effort, >> but never looked into it technically. The more I see of it the better >> it sounds. This seems to have precisely the flexibility we need. > > I'm very glad to hear you think so too! Spread the word! :P > > Just to be clear, sage-on-gentoo itself has no goal of being a way to > distribute Sage in general. It is simply a port of Sage and its SPKGs to > the Gentoo package management system, using the "overlay" architecture > the Gentoo distro has (similar to Personal Package Archives on Ubuntu, > if you're familiar with those). Basically you just put something in a > config file on your Gentoo system which says "augment the package > repository with ebuilds from sage-on-gentoo", and you can now install > Sage system-wide. > > It is Burcin's lmonade project which combines sage-on-gentoo with Gentoo > Prefix, which is a portable version of the Gentoo package management > system which can be run in a folder on any system, and more importantly, > can be customized to supply any set of packages, not necessarily the > entire set of officially sanctioned packages that comprise the Gentoo > Linux distribution. As the name suggests, it's similar to how you might > use `./configure --prefix=/some/dir` in order to install a program to > /some/dir/bin rather than /usr/bin, say. > > Obviously, this is pretty necessary for Sage proper, which likes to sit > in its own folder and have control over its own private copy of various > things. To do otherwise would be to abandon doing our own packaging > altogether, which I think is basically impossible. So if we were to > adapt anything, it would be lmonade, not pure sage-on-gentoo. Yes. >> One thing seems a bit unclear: are the sage-specific changes pushed >> into some global repository for, e.g. gap, or are they local to the >> sage install? If they are, how is it proposed that versioning be >> handled (will there be a sage-5.0 and sage.5.1 tag, or sage-r1 and >> sage-r2, ?) Or is this repository of Sage use metadata something that >> we maintain? > > I'm not really sure what you mean by a "global repository" or "local to > the sage install". Versioning will be basically the same as how we do > SPKGs right now, namely that we'll take the upstream version of the > package and add on a "Sage patch version", which in portage must be > represented as a suffix of the form -r1, -r2, -r3, etc. So for example > what is now sympow-1.018.1.p11.spkg would become, in the new system, a > possible set of patches, plus sympow-1.018.1-r11.ebuild, containing > (among other things) a URL from which to download the sympow 1.018.1 > source tarball. > > Here is a highly abbreviated directory tree of lmonade, as it appears in > the installation I made the other day, for reference: > > lmonade/ > | dist package management / build system > | | distfiles/ downloaded upstream source tarballs > | | log/ build logs > | ` portage/ ebuilds, patches, etc. > | ` sci-mathematics/ > | ` gap/ > | | files/ > | | | gap-4.4.12-sage-and-steve-lintons-itanium.patch > | | ` gap-4.4.12-sage-strict_aliasing.patch > | | gap-4.4.12-r2.ebuild > | ` metadata.xml > ` local/ software is installed here > | bin/, etc/, include/, lib/, libexec/, sbin/ > ` share/ > ` sage/ > ` devel/ corresponds to $SAGE_ROOT/devel today > > So the "package repository" would be dist/portage, and the "sage library > repository" would be kept at local/share/sage/devel/sage-main (or > hopefully just ...devel/sage, once we get rid of `sage -clone`). I don't > actually know my way around this directory structure very well, but > those are the main takeaways, I think. > > You can see the contents of gap-4.4.12-r2.ebuild here: > https://github.com/cschwan/sage-on-gentoo/blob/83deaae5/sci-mathematics/gap/gap-4.4.12-r2.ebuild > > There are some seeming oddities there, like the USE flag "emacs" which > pulls in Emacs as a dependency and then installs the Emacs mode for GAP > into it - clearly not something we want to do, and indeed it's not > enabled by default in lmonade. It's there because, as I said, > sage-on-gentoo is meant to allow you to install Sage into a full Gentoo > system, where it *would* make sense to do that. > > I guess to avoid extra maintenance load we'll just have to leave that > extra stuff in, both in sage-on-gentoo and on our own packaging repo. > (And who knows, maybe someone *does* want a local copy of Emacs or > whateverinside their Sage installation...) I doubt we'll be able to > actually use sage-on-gentoo directly as our packaging repo, but at least > we can keep them similar (?). > > By the way, it looks like only three packages in sage-on-gentoo actually > have a "sage" USE flag defined (PolyBoRi, Python, and LinBox). It seems > that the others, including GAP, just install Sage's patches whether you > want them or not. I'm not sure why this is - maybe it's just because > it's painful to specially create a "sage" USE flag for every package, or > maybe it's because sage-on-gentoo wants to get Sage's patches included > in Gentoo proper as default patches, or maybe there's some other reason. > > I'm getting way out of my depth here, as you can tell, so I'm CCing > Burcin and François, in case they haven't been reading this exchange and > have something to add or remove from what I've been saying! I'm sure > Burcin in particular has thought a lot more about where all this could > go than I have. Of course, feel free not to jump in if you're too busy. > > Well, actually I'm going to send this message through Gmane, so maybe > they won't be CC'd after all. We'll see :) > >>> I think we can get a pretty good alternative to this with the split >>> system I'm describing (nimble package management repo for everyone and >>> carefully reviewed sage repo for creating releases from), if I >>> understand correctly. We would include the ebuild for Sage itself inside >>> the Sage repository, and export it to our package repository when we >>> make a release of Sage. >> >> By "our package repository" are you thinking of something maintained >> by us, or by gentoo? > > Gentoo's involvement with sage-on-gentoo and lmonade ends at providing > the package manager software, and in the case of sage-on-gentoo, putting > sage-on-gentoo on the official list of overlays, so that you can attach > sage-on-gentoo to your Gentoo system with a simple command, `layman -a > sage-on-gentoo`. > > So our package repository would be maintained by us. Of course, we would > only need to actively maintain the ebuilds and patches for programs > which we actually patch. No more would it be necessary to create an SPKG > for a program which needs no Sage-specific patches, as long as that > package is already available on Gentoo - we can just steal the ebuild > file from Gentoo's portage tree. > >> But yes, this sounds like a great idea! So the >> collection of .ebuild files (and their supporting patches, etc.) is in >> the Sage repository, but periodically exported somewhere central >> (owned by us? Gentoo?) that people pull from, right? > > Er, nope. The ebuild corresponding to the Sage library itself would be > stored in the Sage repository, and periodically (upon the release of a > new Sage version) be exported into our package repository. All other > ebuilds and patches would be put directly into that package repository, > which would be hosted live somewhere, say on github. Our package > repository would look very similar to the current sage-on-gentoo > repository, i.e. be a portage tree providing ebuilds. The other > repository would be the Sage repository. I understand the purity of keeping ebuild files separate from the source itself, keeping (exact) dependencies and metadata under a single revision control, and viewing the Sage library itself as just another package, this still means one can't do git checkout trac-12345 # or pull from remote, etc. local/bin/sage -b for tickets spanning package changes. Perhaps something with git submodules (fake or otherwise) or overlaying one portage directory over the other could make this possible. Upgrading to the latest alpha (or even more find-grained versions) would be "git pull" which is something that's also really lacking in our current workflow. All of this requires a repository. Of course, there's also the question of bootstrapping itself. Would lmonade itself be shipped with Sage? What about versioning this piece of software? - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
