+++ Guilherme Salgado [2010-11-11 17:32 -0200]: > I thought this would be a simple question about integrating xdeb with > sbuild but as it turns out there are other things here that could > benefit from being discussed with a bigger audience, so I'm CCing > linaro-dev.
Good idea. I've been planning to send a 'the state of cross building at the end of this cycle' mail very soon (once specs are done with), to say what we have so far, what works and what doesn't. (before we mess with things again :-) > On Wed, 2010-11-10 at 21:30 +0000, Wookey wrote: > > +++ Guilherme Salgado [2010-11-10 17:49 -0200]: > > > Hi Wookey, > > > > That sounds quite nice, and I'm all for not reinventing the wheel, so > I'll have a look at pdebuild-cross. > > I just ran pdebuild-cross-create and that seems to create the chroot and > tar it up, so even if we go with sbuild I should be able to use > pdebuild-cross to create the chroots. OK, glad that works for you too. I guess if we decide not to use pbuilder at all but use schroot+sbuild instead then we should call the relevant chroot-creation foo (which is really multistrap config files and a command like multistrap -f /usr/share/ubuntu/armel.conf) from somewhere else (if you look pdebuild-cross-create is very thin veneer). This is a good example of why tools to do each part well is good - because it becomes dead easy to support the use of pbuilder or sbuild, as required. > > We appear to agree completely on the principles of the thing :-) > > And I hope we can agree on the tools as well. The reason why I went > with sbuild is because I was told it'd be more suitable for this job > than pbuilder, and one thing I liked about it is that it's got most of > its functionality packaged in a library (although it's written in perl) > rather than a single tool. Is pbuilder like that as well? pbuilder is all shell, but seems quite well documented and put together. I don't have strong opinions about it - we used it because it was there and it was easy to do, and I was familiar with its operation mode. it does have minus points: One disadvantage it has is that a 'make clean' is done outside the chroot in order to generate a pristine source package, and thus you need any deps that task needs installed outside. That doesn't seem easily automable (because the package doesn't tell you which of its deps are used by make clean, although in practice a small number of packages (10?) seems to cover many cases. The other thing I haven't yet found out how to do is to choose which tools are used inside the chroot. You can have different hook-scripts easily for different tools (e.g. building with xdeb or building with xapt+dpkg-buildpackage), but it always wants to run all of them and I don't see an easy mechanism for saying 'just run these this time' (without copying different subsets in yourself). No idea if sbuild makes this any easier or not. In actual use you don't need this sort of choice anyway if you have a tool-set that works, but it's handy for experimenting. > > I still haven't worked out why everyone is so enamoured of xdeb, and > > keep trying to design things round it. It has some useful features, > > I can't answer that, but I can speculate: > - it does the job > - most Linaro docs I've seen about cross building use it > - we have in-house expertise on the tool > - it's written in python That last is a big minus point from my POV :-) I've had to learn python to change anything in it and I'm still rubbish at it, which doesn't help speed development. I'm much happier messing with things in perl and shell. I'm not 'ist' about these languages - it's just much easier actually get things done in ones you already know. > > but it was designed as a bootstraping-from-local-sources tool, which > > is not the same thing at all as a tool for cross-building individual > > packages in a repository context. The clever bit of having heuristics > > to supply deps is better dealt with by actually fixing the package > > metadata, and for packages where that's not yet been done (arguably) > > That's what MultiArch is about, right? Not exactly, but a feature of the way it is implemented provides the metadata in dependencies that we need. And handily we can add this without actually converting the package to multiarch so we can do a whole pile of that easily. > MultiArch seems to be our > long-term plan, but in the meantime we have to use something else. > That's where xdeb becomes useful, AIUI. It is, although my tests so far show that the tens-of-times-simpler xapt seems to work just as well (slightly better in fact because it doesn't barf on some packages) for cross-dependency satisfaction. > > I have never looked at sbuild so have no idea what the right way to > > do this is. > > Err, I thought you would know because Steve L. told me to talk to you if > I wanted to discuss implementation details. Anyway, it's good that I > did because otherwise we wouldn't have found this overlap in our work. Indeed. And I've been meaning to look at sbuild for ages anyway as well as this job, so this is as good a time as any. > > > Also, sbuild uses a (configurable) dependency resolver to figure out what > > > build deps to install, but AIUI xdeb handles that, so I'd change sbuild to > > > skip that step completely (by using a new dependency resolver that does > > > nothing). > > > > This is exactly the problem I have with trying to jam xdeb in for > > these jobs. It's too high-level. We should have a cross-dep-resolving > > tool that _just_ does that and make sbuild use it. (xapt is a > > simple-minded one, xdeb contains an implementation of a cleverer one > > but you can't use it on it's own). This would result in simple, > > maintainable, tools that can be used in various contexts. > > xdeb seems to be factored into small functions already, so it should be > easy to move that functionality into a library or a separate tool. I > suspect it'd be the same for the rest of xdeb as well. I certainly hope so, and that's the general plan. The important thing to get right is the interface points so that things fit together properly, and use the right (default) configs. I'm still working that out. > > We seem to be on very much the same page, which is great. Quite a lot > > of what I've been putting in my spec is actually about how to > > implement your spec properly, it seems. Both of us seem keen on the > > 'use existing tools by sticking them together' where each tool does > > one job well. I think if we fish out the cross-dep resolver and > > installer logic into a separate tool then we can do all sorts of > > things in a satisfying way. We also need cross-dep metadata in > > packages for any tool to get it right. And if we had stage-build > > Do you mean to fish them both out of xdeb or do you see the installer > logic coming from somewhere else? I could have a look at extracting > them from xdeb, just to have an idea of how much work it'd involve. Fishing them out of xdeb seems the most obvious course so that's what's planned, (but I reserve the right to decide that it's actually easier to use other things than separate out parts of xdeb without breaking it for its existing uses). I'd be very happy to input on that from someone with better python-foo. It does make a lot of sense to have a load of library functions and data structures that the various aspects can use. In all of this I am also conscious that both xdeb and pdebuild-cross+xapt do already work tolerably well for some of things we want to do, and I'd actually like to get some of that stuff done (ALIP flavoured builds, basic multiarch cross-dep support) _before_ we take a big chainsaw to the tools and break everything again. Hopefully we can do both in parallel with a bit of care. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ _______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
