On 13/03/2009, Cyril Brulebois <k...@debian.org> wrote: > Luca Favatella <slacky...@gmail.com> (12/03/2009): >> Can you please tell me if >> - someone already wrote patches for debootstrap to support more than a >> suite (at the same time, i.e. sid and unreleased)? > > http://lists.debian.org/debian-bsd/2007/08/msg00005.html
Why didn't this code go in debootstrap svn (just curious)? This git link seems not working. git://alioth.debian.org/~kibi-guest/debootstrap.git I refreshed (against current debootstrap svn) the patches at http://users.alioth.debian.org/~kibi-guest/kfreebsd/ I am attaching the "svn diff". I hope I did no mistakes. The most difficult things to decide for me have been " characters. I tried to run kbsd-debootstrap.sh (the refreshed one by me). I got: /lib/freebsd/mount: error while loading shared libraries: libbsd.so.0: cannot open shared object file: No such file or directory After chrooting in sandbox, I installed (with dpkg) libc0.1, libgcc1 and libbsd0 (all packages where already downloaded in /var/cache/...). After that, I tried: mount -t devfs devfs /dev and I got no error. It seems to me that in debootstrap: -or libbsd0 (and other packages) are downloaded but not extracted -or mount is called too early Can you please consider commenting this results? Can you also consider replying to at least some of the following questions? Is there any possibility to merge these refreshed patches in debootstrap svn? Is there any collection of tests to ensure that no regressions is introduced in debootstrap? Is there any rewriting effort of debootstrap underway? Should I consider improving crosshurd, or is it deprecated in favour of debootstrap? I noted that ging (Debian GNU/kFreeBSD live cd) is based upon crosshurd: if I play with ging, should I consider improving crosshurd or porting it to debootstrap? Thanks in advance, Luca Favatella
Index: kbsd-debootstrap.sh =================================================================== --- kbsd-debootstrap.sh (revision 0) +++ kbsd-debootstrap.sh (revision 0) @@ -0,0 +1,28 @@ +#!/bin/sh + +# First guess for needed "unreleased" packages: +# +# grep-dctrl -F Priority 'required' -s Package \ +# < /var/cache/apt/lists/ftp.debian-ports.org_debian_dists_unreleased_main_binary-kfreebsd-i386_Package \ +# | awk '{print $2}' | xargs ./scripts/porters/extra-dependencies.pl --extra-only module-init-tools net-tools +# +# "module-init-tools" has been added manually after a first run, due to an alternative +# "kfreebsd-image" has not been added since it is a virtual package +# +# "net-tools" has been added as Depends of packages from "unstable" +# +# +# Needed "unstable" packages: +# +# "libbsd0" and "ufsutils" have been added as Depends of packages from "unstable" +# +./debootstrap \ + --include="libbsd0 ufsutils" \ + --keep-debootstrap-dir \ + --extra-include="bsdutils freebsd-hackedutils freebsd-utils kfreebsd-loader libfreebsd0 libgeom0 libipx2 libkiconv3 libkvm0 libsbuf0 module-init-tools net-tools util-linux" \ + --extra-suite "unreleased" \ + --extra-mirror http://ftp.debian-ports.org/debian \ + sid \ + ../sandbox \ + http://ftp.debian-ports.org/debian \ + scripts/debian/sid Index: debootstrap =================================================================== --- debootstrap (revision 57816) +++ debootstrap (working copy) @@ -9,6 +9,9 @@ if [ "$DEBOOTSTRAP_DIR" = "" ]; then if [ -x /debootstrap/debootstrap ]; then DEBOOTSTRAP_DIR=/debootstrap + elif [ -x $(pwd)/debootstrap ]; then + echo "Warning: Make sure that 'fakeroot make devices.tar.gz' was done" + DEBOOTSTRAP_DIR=$(pwd) else DEBOOTSTRAP_DIR=/usr/share/debootstrap fi @@ -93,6 +96,8 @@ (requires --second-stage) --boot-floppies used for internal purposes by boot-floppies --debian-installer used for internal purposes by debian-installer + + --extra-* used by porters, see the manpage EOF } @@ -283,6 +288,26 @@ error 1 NEEDARG "option requires an argument %s" "$1" fi ;; + --extra-mirror) + if [ -n "$2" ] ; then + EXTRA_MIRROR="$2" + shift 2 + else + error 1 NEEDARG "option requires an argument %s" "$1" + fi + ;; + --extra-suite) + if [ -n "$2" ] ; then + EXTRA_SUITE="$2" + shift 2 + else + error 1 NEEDARG "option requires an argument %s" "$1" + fi + ;; + --extra-include*) + extra_debs="$(echo $1 | cut -f2 -d"="|tr , " ")" + shift 1 + ;; --*) error 1 BADARG "unrecognized or invalid option %s" "$1" ;; @@ -349,8 +374,12 @@ error 1 WHATARCH "Couldn't work out current architecture" fi -export ARCH SUITE TARGET +if [ "$extra_debs" != "" -a "$EXTRA_SUITE" = "" ]; then + EXTRA_SUITE="unreleased" +fi +export ARCH SUITE TARGET EXTRA_SUITE + if am_doing_phase first_stage second_stage; then if [ -x /usr/bin/id ] && [ `id -u` -ne 0 ]; then error 1 NEEDROOT "debootstrap can only run as root" @@ -430,8 +459,12 @@ fi fi -export MIRRORS +if [ "$EXTRA_MIRROR" = "" ]; then + EXTRA_MIRROR="$MIRRORS" +fi +export MIRRORS EXTRA_MIRROR + ok=false for v in $SUPPORTED_VARIANTS; do if doing_variant $v; then ok=true; fi @@ -445,7 +478,12 @@ if am_doing_phase finddebs; then if [ "$FINDDEBS_NEEDS_INDICES" = "true" ] || \ [ "$RESOLVE_DEPS" = "true" ]; then - download_indices + download_indices "$SUITE" "$MIRRORS" + # The test might be skipped if download_indices() can detect that + # it didn't receive two parameters + if [ "$extra_debs" != "" ]; then + download_indices "$EXTRA_SUITE" "$EXTRA_MIRROR" + fi GOT_INDICES=true fi @@ -492,9 +530,16 @@ if am_doing_phase dldebs; then if [ "$GOT_INDICES" != "true" ]; then - download_indices + download_indices "$SUITE" "$MIRRORS" + # The test might be skipped if download_indices() can detect that + # it didn't receive two parameters + if [ "$extra_debs" != "" ]; then + download_indices "$EXTRA_SUITE" "$EXTRA_MIRROR" + fi fi - download $all_debs + # Separate download needed: different suite/mirror + download "$SUITE" "$MIRRORS" $all_debs + download "$EXTRA_SUITE" "$EXTRA_MIRROR" $extra_debs fi if am_doing_phase maketarball; then @@ -502,6 +547,10 @@ tar czf - var/lib/apt var/cache/apt) >$MAKE_TARBALL fi +# Could be more fine-grained by handling $extra_debs until the end. +# Probably not needed, so merging $required and $extra_debs +required="$required $extra_debs" + if am_doing_phase first_stage; then # first stage sets up the chroot -- no calls should be made to # "chroot $TARGET" here; but they should be possible by the time it's @@ -542,12 +591,16 @@ rm -f "$TARGET/etc/apt/sources.list" fi if [ "${MIRRORS#http://}" != "$MIRRORS" ]; then - setup_apt_sources "${MIRRORS%% *}" - mv_invalid_to "${MIRRORS%% *}" + setup_apt_sources $SUITE "${MIRRORS%% *}" + mv_invalid_to $SUITE "${MIRRORS%% *}" else - setup_apt_sources "$DEF_MIRROR" - mv_invalid_to "$DEF_MIRROR" + setup_apt_sources $SUITE "$DEF_MIRROR" + mv_invalid_to $SUITE "$DEF_MIRROR" fi + if [ "$EXTRA_SUITE" != "" ]; then + setup_apt_sources $EXTRA_SUITE $EXTRA_MIRROR + mv_invalid_to $EXTRA_SUITE $EXTRA_MIRROR + fi if [ -e "$TARGET/debootstrap/debootstrap.log" ]; then if [ "$KEEP_DEBOOTSTRAP_DIR" = true ]; then Index: scripts/debian/sid =================================================================== --- scripts/debian/sid (revision 57816) +++ scripts/debian/sid (working copy) @@ -7,12 +7,12 @@ test "$FAKECHROOT" = "true" || error 1 FAKECHROOTREQ "This variant requires fakechroot environment to be started" fi -LIBC=libc6 -if [ "$ARCH" = "alpha" ] || [ "$ARCH" = "ia64" ]; then - LIBC="libc6.1" -elif [ "$ARCH" = "hurd-i386" ]; then - LIBC="libc0.3" -fi +case $ARCH in + alpha|ia64) LIBC6="libc6.1" ;; + kfreebsd-*) LIBC6="libc0.1" ;; + hurd-*) LIBC6="libc0.3" ;; + *) LIBC6="libc6" ;; +esac work_out_debs () { required="$(get_debs Priority: required)" Index: scripts/porters/extra-dependencies.pl =================================================================== --- scripts/porters/extra-dependencies.pl (revision 0) +++ scripts/porters/extra-dependencies.pl (revision 0) @@ -0,0 +1,116 @@ +#!/usr/bin/perl -w +# +# Cyril Brulebois <cyril.bruleb...@enst-bretagne.fr>, 2007 +# +# This program is free software; you can redistribute it and/or modify it +# under the same terms as Perl itself. + + +use strict; +use Getopt::Long; # Package: perl-base + +my $usage = << "END_OF_USAGE"; +Usage: + $0 [--extra-suite SUITE] [--extra-only] [--debug] package1 [...] packageN + +Default SUITE is "unreleased" +Others may be used, e.g. "experimental" +END_OF_USAGE + +# Option check +my ($extra_suite, $extra_only, $debug); +GetOptions ( + 'extra-suite=s' => \$extra_suite, + 'extra-only' => \$extra_only, + 'debug' => \$debug, +) + or die $usage; + +# Default setting +$extra_suite ||= 'unreleased'; + +# Parameter check +my (@packages) = @ARGV + or die $usage; + +# Loop on the packages +foreach my $package (@packages) { + print_if_debug("CONSIDERING: $package\n"); + + # Pick their dependencies + my @extras = get_dependencies($package); + + # Are they already on the list? If not, add them + foreach my $extra (@extras) { + if (not grep {$_ eq $extra} @packages) { + print_if_debug("ADDING: $extra\n"); + push @packages, $extra; + } + } +} + +# Print the final sorted list +print_if_debug("\n"); +print_if_debug("--[SUMMARY]--\n"); +print_if_debug("NEEDED: "); +print join(' ', sort @packages), "\n"; +print_if_debug("-------------\n"); + +# END + + + +# Calculate the dependencies, possibly only the experimental ones +sub get_dependencies { + my ($package) = @_; + + # Get the dependencies + my @items = (); + my @descriptions = `apt-cache show $package`; + foreach my $description (@descriptions) { + # Consider only Depends: and Pre-Depends: + push @items, (split /,/, $1) + if $description =~ /^(?:Pre-|)Depends: (.*)$/; + } + + # Analyze them + my @results = (); + foreach my $item (@items) { + # Trim + $item =~ s/^\s*//; + $item =~ s/\s*$//; + + # Check whether this is an alternative, warn & skip if needed + if ($item =~ /\|/) { + print("SKIPPING ALTERNATIVE: $item\n"); + next; + } + + # Unversion the dependency + $item =~ s/\s*\(.*?\)//; + + # Check whether it is a package from $extra_suite + if ($extra_only) { + my $origin = `apt-cache policy $item`; + if ($origin =~ m{((?:ftp|http)://.*?) (.*?)/main}ms) { + my ($url, $dist) = ($1, $2); + + # Skip if it is not the wanted + next + if $dist ne $extra_suite; + } + } + + # Add it finally (we don't care about the possible duplicates) + push @results, $item; + } + + return @results; +} + + +# Trivial helper function +sub print_if_debug { + print @_ + if $debug; +} Index: functions =================================================================== --- functions (revision 57816) +++ functions (working copy) @@ -374,12 +374,21 @@ download () { mk_download_dirs - "$DOWNLOAD_DEBS" $(echo "$@" | tr ' ' '\n' | sort) + local suite="$1" + local mirrors="$2" + shift 2 + "$DOWNLOAD_DEBS" "$suite" "$mirrors" $(echo "$@" | tr ' ' '\n' | sort) } download_indices () { mk_download_dirs - "$DOWNLOAD_INDICES" $(echo "$@" | tr ' ' '\n' | sort) + local suite="$1" + local mirrors="$2" + shift 2 + # It is never called with any additional parameters, why not just + # getting rid of the echo/tr/sort stuff? It might have been a + # copy/paste from the other function wrapper. + "$DOWNLOAD_INDICES" "$suite" "$mirrors" $(echo "$@" | tr ' ' '\n' | sort) } debfor () { @@ -443,16 +452,17 @@ } download_release_sig () { - local m1="$1" - local reldest="$2" - local relsigdest="$TARGET/$($DLDEST rel "$SUITE" "$m1" "dists/$SUITE/Release.gpg")" + local suite="$1" + local m1="$2" + local reldest="$3" + local relsigdest="$TARGET/$($DLDEST rel "$suite" "$m1" "dists/$suite/Release.gpg")" if [ -n "$KEYRING" ]; then progress 0 100 DOWNRELSIG "Downloading Release file signature" progress_next 50 - get "$m1/dists/$SUITE/Release.gpg" "$relsigdest" nocache || + get "$m1/dists/$suite/Release.gpg" "$relsigdest" nocache || error 1 NOGETRELSIG "Failed getting release signature file %s" \ - "$m1/dists/$SUITE/Release.gpg" + "$m1/dists/$suite/Release.gpg" progress 50 100 DOWNRELSIG "Downloading Release file signature" info RELEASESIG "Checking Release signature" @@ -465,24 +475,34 @@ } download_release_indices () { - local m1="${MIRRORS%% *}" - local reldest="$TARGET/$($DLDEST rel "$SUITE" "$m1" "dists/$SUITE/Release")" + local suite="$1" + local m1="$2" + shift 2 + + m1="${m1%% *}" + + local reldest="$TARGET/$($DLDEST rel "$suite" "$m1" "dists/$suite/Release")" progress 0 100 DOWNREL "Downloading Release file" progress_next 100 - get "$m1/dists/$SUITE/Release" "$reldest" nocache || - error 1 NOGETREL "Failed getting release file %s" "$m1/dists/$SUITE/Release" + get "$m1/dists/$suite/Release" "$reldest" nocache || + error 1 NOGETREL "Failed getting release file %s" "$m1/dists/$suite/Release" - TMPCOMPONENTS="$(sed -n 's/Components: *//p' "$reldest")" - for c in $TMPCOMPONENTS ; do - eval " - case \"\$c\" in - $USE_COMPONENTS) - COMPONENTS=\"\$COMPONENTS \$c\" - ;; - esac - " - done - COMPONENTS="$(echo $COMPONENTS)" + # Avoid component duplication (which is harmless anyway, components + # just appear several times in sources.list in this case) when + # download_release_indices() is called more than once. + if [ "$COMPONENTS" = "" ]; then + TMPCOMPONENTS="$(sed -n 's/Components: *//p' "$reldest")" + for c in $TMPCOMPONENTS ; do + eval " + case \"\$c\" in + $USE_COMPONENTS) + COMPONENTS=\"\$COMPONENTS \$c\" + ;; + esac + " + done + COMPONENTS="$(echo $COMPONENTS)" + fi if [ "$COMPONENTS" = "" ]; then mv "$reldest" "$reldest.malformed" @@ -490,7 +510,7 @@ fi progress 100 100 DOWNREL "Downloading Release file" - download_release_sig "$m1" "$reldest" + download_release_sig "$suite" "$m1" "$reldest" local totalpkgs=0 for c in $COMPONENTS; do @@ -519,7 +539,7 @@ progress 0 $totalpkgs DOWNPKGS "Downloading Packages files" for c in $COMPONENTS; do local subpath="$c/binary-$ARCH/Packages" - local path="dists/$SUITE/$subpath" + local path="dists/$suite/$subpath" local bz2md="`get_release_md5 "$reldest" "$subpath.bz2"`" local gzmd="`get_release_md5 "$reldest" "$subpath.gz"`" local normmd="`get_release_md5 "$reldest" "$subpath"`" @@ -538,8 +558,8 @@ md="${md:-$gzmd}" fi progress_next "$(($donepkgs + ${md#* }))" - for m in $MIRRORS; do - pkgdest="$TARGET/$($DLDEST pkg "$SUITE" "$c" "$ARCH" "$m" "$path")" + for m in $m1; do + pkgdest="$TARGET/$($DLDEST pkg "$suite" "$c" "$ARCH" "$m" "$path")" if get "$m/$path" "$pkgdest" $ext; then break; fi done if [ ! -f "$pkgdest" ]; then @@ -598,8 +618,12 @@ } download_release () { - local m1="${MIRRORS%% *}" + local suite="$1" + local m1="$2" + shift 2 + m1="${m1%% *}" + local numdebs="$#" local countdebs=0 @@ -610,8 +634,8 @@ for c in $COMPONENTS; do if [ "$countdebs" -ge "$numdebs" ]; then break; fi - local path="dists/$SUITE/$c/binary-$ARCH/Packages" - local pkgdest="$TARGET/$($DLDEST pkg "$SUITE" "$c" "$ARCH" "$m1" "$path")" + local path="dists/$suite/$c/binary-$ARCH/Packages" + local pkgdest="$TARGET/$($DLDEST pkg "$suite" "$c" "$ARCH" "$m1" "$path")" if [ ! -e "$pkgdest" ]; then continue; fi info CHECKINGSIZES "Checking component %s on %s..." "$c" "$m1" @@ -634,13 +658,17 @@ local dloaddebs=0 progress $dloaddebs $totaldebs DOWNDEBS "Downloading packages" - :>$TARGET/debootstrap/debpaths + # This function can be called several times. The first should be + # with $suite = $SUITE, only empty the file in this case. + if [ "$suite" = "$SUITE" ]; then + :>$TARGET/debootstrap/debpaths + fi pkgs_to_get="$*" for c in $COMPONENTS; do - local path="dists/$SUITE/$c/binary-$ARCH/Packages" - for m in $MIRRORS; do - local pkgdest="$TARGET/$($DLDEST pkg "$SUITE" "$c" "$ARCH" "$m" "$path")" + local path="dists/$suite/$c/binary-$ARCH/Packages" + for m in $m1; do + local pkgdest="$TARGET/$($DLDEST pkg "$suite" "$c" "$ARCH" "$m" "$path")" if [ ! -e "$pkgdest" ]; then continue; fi pkgs_to_get="$(download_debs "$m" "$pkgdest" $pkgs_to_get 5>&1 1>&6)" if [ "$pkgs_to_get" = "" ]; then break; fi @@ -654,15 +682,20 @@ } download_main_indices () { - local m1="${MIRRORS%% *}" + local suite="$1" + local m1="$2" + shift 2 + + m1="${m1%% *}" + progress 0 100 DOWNMAINPKGS "Downloading Packages file" progress_next 100 COMPONENTS=main export COMPONENTS - for m in $MIRRORS; do + for m in $m1; do for c in $COMPONENTS; do - local path="dists/$SUITE/$c/binary-$ARCH/Packages" - local pkgdest="$TARGET/$($DLDEST pkg "$SUITE" "$c" "$ARCH" "$m" "$path")" + local path="dists/$suite/$c/binary-$ARCH/Packages" + local pkgdest="$TARGET/$($DLDEST pkg "$suite" "$c" "$ARCH" "$m" "$path")" if [ -x /bin/gunzip ] && get "$m/${path}.gz" "${pkgdest}.gz"; then rm -f "$pkgdest" gunzip "$pkgdest.gz" @@ -675,15 +708,24 @@ } download_main () { - local m1="${MIRRORS%% *}" + local suite="$1" + local m1="$2" + shift 2 - :>$TARGET/debootstrap/debpaths + m1="${m1%% *}" + + # This function can be called several times. The first should be + # with $suite = $SUITE, only empty the file in this case. + if [ "$suite" = "$SUITE" ]; then + :>$TARGET/debootstrap/debpaths + fi + for p in "$@"; do for c in $COMPONENTS; do local details="" - for m in $MIRRORS; do - local path="dists/$SUITE/$c/binary-$ARCH/Packages" - local pkgdest="$TARGET/$($DLDEST pkg "$SUITE" "$c" "$ARCH" "$m" "$path")" + for m in $m1; do + local path="dists/$suite/$c/binary-$ARCH/Packages" + local pkgdest="$TARGET/$($DLDEST pkg "$suite" "$c" "$ARCH" "$m" "$path")" if [ ! -e "$pkgdest" ]; then continue; fi details="$($PKGDETAILS PKGS "$m" "$pkgdest" "$p")" if [ "$details" = "$p -" ]; then continue; fi @@ -709,6 +751,9 @@ ###################################################### deb choosing support get_debs () { + # Known to only operate on MIRRORS, that is: the default suite. + # Using the manpage instructions and the porter script should be + # sufficient. local field="$1" shift local m1 c @@ -771,26 +816,29 @@ } mv_invalid_to () { - local m="$1" + local suite="$1" + local m="$2" m="$(echo "${m#http://}" | tr '/' '_' | sed 's/_*//')" (cd "$TARGET/$APTSTATE/lists" - for a in debootstrap.invalid_*; do + for a in debootstrap.invalid_dists_${suite}_*; do mv "$a" "${m}_${a#*_}" done ) } setup_apt_sources () { + local suite="$1" + shift 1 mkdir -p "$TARGET/etc/apt" for m in "$@"; do local cs="" for c in $COMPONENTS; do - local path="dists/$SUITE/$c/binary-$ARCH/Packages" - local pkgdest="$TARGET/$($DLDEST pkg "$SUITE" "$c" "$ARCH" "$m" "$path")" + local path="dists/$suite/$c/binary-$ARCH/Packages" + local pkgdest="$TARGET/$($DLDEST pkg "$suite" "$c" "$ARCH" "$m" "$path")" if [ -e "$pkgdest" ]; then cs="$cs $c"; fi done - if [ "$cs" != "" ]; then echo "deb $m $SUITE$cs"; fi - done > "$TARGET/etc/apt/sources.list" + if [ "$cs" != "" ]; then echo "deb $m $suite$cs"; fi + done >> "$TARGET/etc/apt/sources.list" } setup_etc () { @@ -1009,6 +1057,8 @@ ##################################################### dependency resolution resolve_deps () { + # Known to only operate on MIRRORS, that is: the default suite. See + # the porter helper script to calculate the dependencies local m1="${MIRRORS%% *}" # XXX: I can't think how to deal well with dependency resolution and @@ -1316,4 +1366,4 @@ /usr/bin/ldd.REAL fakechroot" >> "$TARGET/var/lib/dpkg/diversions" -} +} \ No newline at end of file Index: debootstrap.8 =================================================================== --- debootstrap.8 (revision 57816) +++ debootstrap.8 (working copy) @@ -135,6 +135,30 @@ .IP "\fB\-\-debian\-installer\fP" Used for internal purposes by the debian-installer .IP +.SH "PORTER OPTIONS" +. +.PP +The following options should be useful only to porters whose arch has +not yet been integrating into the official archive, and who need to +download additional packages from a suite called \fIunreleased\fR or +similar. +.IP +.IP "\fB\-\-extra\-mirror EXTRA_MIRROR\fP" +Set the mirror for the extra packages, defaults to \fIMIRROR\fR. +.IP +.IP "\fB\-\-extra\-suite EXTRA_SUITE\fP" +Set the suite name to use for the extra packages, defaults to +\fIunreleased\fR. +.IP +.IP "\fB\-\-extra\-include=freebsd\-hackedutils,freebsd\-utils,...\fP" +Set the packages to pull from there. +.PP +Note that all dependencies have to be solved manually: the extra +included packages should be autosufficient (in \fIEXTRA_SUITE\fR); +and their dependencies in \fISUITE\fR have to be added using +\fB\-\-include\fP. A helper script is available in debootstrap's +sources, see \fIscripts/porters/\fR). +.IP .SH "EXAMPLE" . .PP