I thought this would be a simple question about integrating xdeb with sbuild but as it turns out there are other things here that could benefit from being discussed with a bigger audience, so I'm CCing linaro-dev.
On Wed, 2010-11-10 at 21:30 +0000, Wookey wrote: > +++ Guilherme Salgado [2010-11-10 17:49 -0200]: > > Hi Wookey, > > > > As I've mentioned on IRC, we want to have sbuild do cross builds using > > xdeb so that we can use it in > > <https://wiki.linaro.org/Platform/Infrastructure/Specs/PackageDevelopmentTools>. > > Oh, that looks interesting. I'm very keen to have a cross-autobuilder > churning through things to find out how much stuff actually > cross-builds at the moment. > > In fact reading that work thing you seem to be wanting to build > exactly what I want to build: a system that repeatably builds > packages in pristine chroots. I thought sbuild didn't normally do the > pristine chroot thing? (which is why I've been using pbuilder to > supply that functionality to date). Yeah, I don't think sbuild itself provides that, so I'm using schroot's type=file chroots for that. You just point schroot to a tarball containing your chroot and it will (by default) create a new session (with a pristine chroot) every time you 'schroot -c <name>'. If you want to change the contents of the tarball you can easily do that with 'schroot -c <name>-source'. All you need for that to work is the following config: [natty] description=Natty chroot type=file file=/var/chroots/natty.tar location=/natty > > If you look at my (Still developing) cross-building spec: > https://wiki.linaro.org/Platform/Foundations/Specs/CrossBuilding > you'll see that I have very similar ideas, but I want to have tools to > do the various aspects of this job (satisfy-cross-deps, upload things, > download things, do build-ordering), rather than the unhelpfully > monolithic xdeb. It's certainly a good idea to have separate tools (or maybe even libraries?) for the various aspects, and although at first that didn't seem necessary to reach our goals in PackageDevelopmentTools, I think it's clearly becoming more important now as we find opportunities to collaborate. > > Do you realise that pdebuild-cross (in maverick) already makes a chroot to do > cross-building in with a single command: 'pdebuild-cross-create'? I didn't know about that; I'm still finding my way around all these tools for cross-building debs. :) > It installs a base system plus crosstoolchains > plus build-essential. (defaults to -aarmel). Currently there is just a > config for maverick, but we can add some more if we want to have natty > and natty+ppearses flavoured toolchains. That 'make me a cross-chroot' > stuff hides the bits you detail in "Notes on using sbuild+xdeb to do > cross compilation" on having to make sure source lines are put in and > cross-tools installed and so on. It's all done already :-) That sounds quite nice, and I'm all for not reinventing the wheel, so I'll have a look at pdebuild-cross. I just ran pdebuild-cross-create and that seems to create the chroot and tar it up, so even if we go with sbuild I should be able to use pdebuild-cross to create the chroots. > > That actually uses multistrap to do the hard work of setting up the > chroots with the right stuff. This is useful because debootstrap can't > take a base system from here and tool from there - everything has to > come from one place (although we could probably fix that). multistrap > has a neat cascading config scheme for making chroots so you can say > 'maverick + armel-cross-build + this-toolchain'. This is very handy in > the real world where the parts you need are not necessarily (yet) in > the distro. > > > After learning and experimenting a bit, this is what I came up with. I > > have some specific questions further down, but any feedback you might > > have is appreciated. > > I am nervous about sbuild+xdeb. It depends exactly what you are > trying to do, but xdeb's caching and recursive build behaviour, whilst > useful in some ways, also means that builds are not repeatable unless > you either go to special effort to clean things our or do the builds > in a disposable chroot. (pbuilder gives you the disposable thing for > free, at the expense of the time it takes to unpack and install deps > each time). I want a cross-builder to the same thing each time I ask > it to build a package. xdeb doesn't help at all with that as currently > designed. Ah, I see you want that too and propose schroot > snapshots+sessions. That's cool. I've used schroot a lot for managing > cross-build chroots in the past, but that was persistent ones, not > snapshotted pristine ones. You'll have to show me how that works. We > appear to agree completely on the principles of the thing :-) And I hope we can agree on the tools as well. The reason why I went with sbuild is because I was told it'd be more suitable for this job than pbuilder, and one thing I liked about it is that it's got most of its functionality packaged in a library (although it's written in perl) rather than a single tool. Is pbuilder like that as well? > > I still haven't worked out why everyone is so enamoured of xdeb, and > keep trying to design things round it. It has some useful features, I can't answer that, but I can speculate: - it does the job - most Linaro docs I've seen about cross building use it - we have in-house expertise on the tool - it's written in python > but it was designed as a bootstraping-from-local-sources tool, which > is not the same thing at all as a tool for cross-building individual > packages in a repository context. The clever bit of having heuristics > to supply deps is better dealt with by actually fixing the package > metadata, and for packages where that's not yet been done (arguably) That's what MultiArch is about, right? MultiArch seems to be our long-term plan, but in the meantime we have to use something else. That's where xdeb becomes useful, AIUI. > to just brute-force install everything native _and_ cross, which works > just fine and is much easier to understand. (it is slower though - I'm > not sure how much in practice, or how much it really matters). > > Why 'larc' BTW? This tool is expected to grow into something that will help developers manage .deb archives as well, so it was named Linaro Archive Creator. https://wiki.ubuntu.com/Specs/M/ARMDeveloperEnvironment has more details about what the tool should do. > > > We don't necessarily expect this functionality to be desirable upstream, in > > which case we'd need a plan to apply them at runtime so that we don't have > > to > > maintain a fork of sbuild. If you think the functionality may be wanted > > upstream, I'm sure we'll work together with them to figure out the best > > way to integrate it. Otherwise, following is one way we could approach it > > > > sbuild's build function (lib/Sbuild/Build.pm, part of libsbuild-perl) > > delegates the package building to dpkg-buildpackage and does a lot of sanity > > checks that we might not need or may not make sense for our use cases, so we > > could replace the whole function with a custom implementation which > > delegates the (cross-)building to xdeb. > > - Is that ok? > > - Should xdeb be used only for cross builds? > > - Is it possible to replace (monkey-patching or similar?) that > > function at runtime with our own implementation of it? (I realize > > this is more of a perl question, but I know close to nothing about > > perl) > > I have never looked at sbuild so have no idea what the right way to > do this is. Err, I thought you would know because Steve L. told me to talk to you if I wanted to discuss implementation details. Anyway, it's good that I did because otherwise we wouldn't have found this overlap in our work. > > > Also, sbuild uses a (configurable) dependency resolver to figure out what > > build deps to install, but AIUI xdeb handles that, so I'd change sbuild to > > skip that step completely (by using a new dependency resolver that does > > nothing). > > This is exactly the problem I have with trying to jam xdeb in for > these jobs. It's too high-level. We should have a cross-dep-resolving > tool that _just_ does that and make sbuild use it. (xapt is a > simple-minded one, xdeb contains an implementation of a cleverer one > but you can't use it on it's own). This would result in simple, > maintainable, tools that can be used in various contexts. xdeb seems to be factored into small functions already, so it should be easy to move that functionality into a library or a separate tool. I suspect it'd be the same for the rest of xdeb as well. > > > Are there any other steps currently performed by sbuild that can be > > avoided in our case? Or anything that I've overlooked? Or a better way > > of doing it? > > I think there is a better way, as largely elaborated above: give > sbuild a cross-dep resolver, use apt to install things of the > appropriate arch, passed through dpkg-cross if no multiarch version of > package available. And just issue dpkg-buildpackage -a<arch> > to do cross-builds. That sounds fine as well, and it's definitely more appealing given that we should be able to share at least parts of the new tools/libraries we develop. > > I could be wrong and there is a reason why all this won't fit > together, but I haven't seen one yet. > > > http://paste.ubuntu.com/529453/ is a diff against maverick's sbuild, > > along the lines of what I described above (although dirty and full of > > hard-coded bits). This diff allowed me to use sbuild to cross build > > expat for armel. > > We seem to be on very much the same page, which is great. Quite a lot > of what I've been putting in my spec is actually about how to > implement your spec properly, it seems. Both of us seem keen on the > 'use existing tools by sticking them together' where each tool does > one job well. I think if we fish out the cross-dep resolver and > installer logic into a separate tool then we can do all sorts of > things in a satisfying way. We also need cross-dep metadata in > packages for any tool to get it right. And if we had stage-build Do you mean to fish them both out of xdeb or do you see the installer logic coming from somewhere else? I could have a look at extracting them from xdeb, just to have an idea of how much work it'd involve. > metadata in packages then we could use xdeb's build-ordering logic to > bootstrap new flavours or even ports. Exactly how that works needs a > bit more thought, but it'd be good to keep it in mind. > > Does that all sounds reasonable to you? I'm trying to work out what the It does! > optimal path is and where to derive the code from because there is > strong pressure to do this quickly as well as properly :-) -- Guilherme Salgado <https://launchpad.net/~salgado>
signature.asc
Description: This is a digitally signed message part
_______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
