Hi, On Fri, Nov 23, 2012 at 03:48:04AM +0000, peter green wrote: > >Since yesterday, my tools can now finally turn the whole dependency > >graph > Does this "whole dependency graph" include the implicit > build-dependency every package has on build-essential?
For source packages for native compilation, yes. What is not included is Build-Depends-Indep. I have been told that there are many problems with Build-Depends-Indep but not considering those dependencies tremendously helps to reduced the amount of cyclic dependency hell, so they should be made working for better bootstrappability. For binary packages, the implicit dependency on priority:essential is kept. For source packages for cross compilation, afaik wookey currently uses an implicit dependency on crossbuild-essential-$arch but this is not yet policy of course and has to be discussed still. > >The above case for example has no alternative solution as the cycle > >is of length two and has no other way of braking it than building > >pkg-config without libglib2.0-dev. Since this is unlikely to be > >possible > I don't see why it would be impossible to hack up the glib source > package to not rely on pkg-config. Whether that is a good idea or > not is another matter. It is not a dependency loop between the pkg-config source package and the glib2.0 source package. It is a dependency loop between src:pkg-config (we use the src: prefix to never confuse source package names with binary package names) and libglib2.0-dev. There is a loop because libglib2.0-dev has a binary dependency on pkg-config but the source package that pkg-config builds from (src:pkg-config) cannot be built without libglib2.0-dev. So even if the glib2.0 source package can be built and therefor make libglib2.0-dev available, libglib2.0-dev can never be installed because it has a binary dependency on pkg-config. pkg-config builds from src:pkg-config but src:pkg-config can never be compiled because it has a build dependency on libglib2.0-dev, which cannot be installed -> loop. The assumption that this specific loop cannot trivially (without cross compilation) be broken depends on the following two assumptions: - binary dependencies should never be modified - src:pkg-config cannot be built without libglib2.0-dev > > and since the assumption is that only > >build dependencies might be dropped when necessary but not binary > >dependencies, a possible solution might be cross compilation. > It seems pretty clear to me that there is a "core" of software that > will need to be cross-built as the first stage of bootstrapping a > port. Obviously "essential" and "build-essential" fall into this > category but while i'm sure there are ways one could hack away the > cycles and make things like pkg-config and debhelper natively > bootstrapable I don't think there is much point in doing so. It is always the call of the user to decide what is easier: - to bootstrap package A natively by braking cycles or - to make all source packages that need to be cross compiled to have package A installable cross compile Both tasks (finding out if the necessary source packages can be built with reduced build dependencies and checking how easy it is to make them cross compilable) can only be solved by a human. My tools help the user in that task by allowing to analyze the dependency graph situation for native compilation and also allow to investigate the list of source packages that have otherwise to be cross compiled to make the package installable. Comparing these two options allows the user to make a better call when he has to decide for either of those two choices. But right now, debhelper is indeed taken as part of the packages to be cross compiled to avoid a horribly messy initial dependency graph. > What i'd ideally like to see is for a tool to be able to generate a > directed acyclic graph of "build jobs" (some cross, some native, there > should be an option in the tool as to whether to preffer native or > cross-build jobs) that takes the user from having no packages for the > target architecture to having a set of bootstrap packages that can be > used to seed the "regular" building process. This is the final goal of my project. What is currently done is the native part of the above. Given a minimal build system (build-essential, priority:essential) it can devise said order of "build jobs" by braking the graph into a directed acyclic graph (DAG). The tool helps the user to find build dependencies that make most sense to be broken and uses a list of brake-able build dependencies supplied by the user to break the graph into DAG with a close to minimal number of source packages that have to be built with reduced build dependencies. Once Debian understands "build profiles" (see my last email) those will of course be taken into account as well. Until this is the case, my tool can help to identify those source packages that make sense to have build dependencies dropped. In a state where Debian understands "build profiles", my current code can be used to find useful build profiles and to check if the available build profiles are enough to bootstrap Debian and if not, point out which source packages should best be modified to make it bootstrappable again. Coming back to the cross case. My tool is using dose3 for dependency resolution. Since my summer project, dose3 can also be used to resolve cross build dependencies. The thing that is currently stopping me from generating a cross build dependency graph and brake it into a DAG just as I do for native compilation, is the current status of Multi-Arch. For multiarch cross building to successfully resolve cross build dependencies, more binary packages must be properly multiarch-ed. The last time I checked the status of multiarch was a month ago. I heard from wookey that since then he made considerable progremms with his arm64 port, so I will look at the cross build situation again. Once all necessary packages have sufficient multiarch properties, it will be possible to build a cross build dependency graph and brake it using the exact same algorithms that I already developed and successfully tested for the native case. Then, it can be done what you outline above, to devise a build order that goes from zero to everything that is needed to build all of Debian fully. You also raise the point of having the option to "prefer" native or cross build jobs. Sometimes (with our current assumptions) a native build dependency cycle cannot be broken other than by cross compiling source packages. For the other cycles there exist both options: braking build dependencies or cross compilation. I think if the algorithm knows (using build profiles) that enough build dependencies can be broken, native compilation should usually be preferred, right? On the other hand, if the current set of build profiles doesnt provide enough droppable dependencies for a cycle to be broken, the algorithm (this is is what it already can do) will present the user with the choice of either braking some build dependencies (suggestions are given) or by cross compiling a number of packages. What would be very nice: if my algorithm had a way of knowing which source packages are supposed to be cross compilable. But there is currently no way to mark a package as being cross-compilable. This makes sense because Debian packages are supposed to be compiled natively. On the other hand, in many bootstrapping scenarios there is no other choice than native compilation (if one does not want to involve another distribution like Gentoo or Openembedded). So maybe Debian will want to include such a flag in the source package description at some point so that future bootstrappers have an easier time to pick packages for cross compilation. Additionally, the packages needed for a minimal build system (priority:essential + build-essential) should require this flag to be set to true. And additionally if a source package has set this flag to true, it should be considered as a bug if the source package does not cross compile. The flag should then be removed (if it is not part of the minimal system) or the problem be fixed. My algorithm could then make suggestions like: "here is a native build dependency cycle that cannot be broken with the current knowledge I have about droppable build dependencies, but I see that the source packages required to break the cycle at a specific point are all supposedly cross compilable, so you might want to mark them to be cross compiled" Thoughts? cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20121123083644.GA7589@hoothoot
