Hi, Apologies for the delayed reply.
On Thursday 25 February 2010 15:53:03 Ulrich Spörlein wrote: > On Thu, 25.02.2010 at 10:08:15 +0200, David Naylor wrote: > > Hi, > > > > As some may have noticed on -current I have been working on using > > stacked unionfs to implement a 'tinderbox' type build system. I have > > successfully used the scripts to build x11/xorg (and have compared the > > results to using the traditional approach using pkg_add). The build > > system is stable except for one nasty corner case: deadlocks. > > When I did this a couple of years ago, the major problems were failing > chdir(2) calls during ports build, etc. I'm hoping time and -current will solve this problem as it is beyond my ability to fix. > > To setup a compatible test environment requires: > > - recompile the kernel with `options KSTACK_PAGES=32`, otherwise the > > kernel will panic with a double fault. WITNESS options results in > > substantial performance degradation. > > - patch mtree (see below) [optional] > > - create the appropriate chroot environment (and reboot) [see below > > for corner case] > > > > A performance bottleneck in mtree was identified. This resulted in > > mtree (as run by port install) consuming ~20% of the build time. See > > bin/143732 for a patch and further details. > > Good work! > > > The normal tinderbox approach takes ~80% more time to install compared to > > the quick and dirty approach. The stacked unionfs approach takes ~170% > > more time (an increase of ~50% over the tinderbox approach). Some > > performance gains can be had if one uses memory backed storage (vs HDD > > in this case). > > Please explain: what is the quick and dirty approach and which one is > faster now? The quick and dirty is `make -C /usr/ports/x11/xorg install clean`. The stacked unionfs is still the slowest (even with a 20% improvement from patching mtree). If one is interested in performance in building ports in a sandbox then the tinderbox route is the way to go. > As your scripts did not make it through, perhaps you can upload them to > the wiki? What I did back then was using a clean base system as the > underlying unionfs store to avoid re-generating the clean base over and > over again. Nowadays, a ZFS clone would probably be the way to go. I've attached the scripts with a .txt suffix. This should hopefully make it past mailman. To which wiki page do you refer? You are welcome to add the scripts there. I skipped the base system and only unionfs /usr/local. I never cleaned the base system. I figured that was the easiest to implement when the script runs in a chroot. An area for further improvements? > I'm not sure if a recursive approach is feasible here, as you can have > only one underlying unionfs mount. But special casing, e.g., perl may > still give a massive speedup. So for each port that has perl as > dependancy, you would not pull in the clean base + pkg_add perl, but > instead grab the clean-base+perl directory as an underlying unionfs. Have you done any speed comparisons of running programs (such as perl) from a unionfs. If the disk cache is big enough I don't think there will be a big performance hit? It is certainly a viable approach. Regards, David
#!/bin/sh
BUILDDIR=${BUILDDIR:-/usr/build}
LOCALBASE=${LOCALBASE:-/usr/local}
PORTSDIR=${PORTSDIR:-/usr/ports}
PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports}
PKG_DBDIR=${PKG_DBDIR:-$LOCALBASE/db_pkg}
PACKAGES=${PACKAGES:-$BUILDDIR/packages}
ENV="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR
PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES"
MAKE="$ENV make"
PKG_ADD="$ENV pkg_add"
PKG_DELETE="$ENV pkg_delete"
set -e
mkdir -p $BUILDDIR $PACKAGES
port2name() {
echo $1 | sed 's|[/.-]|_|g'
}
port2pkg() {
local pkg_name=
local port=
port=$1; shift
eval pkg_name=PKG$(port2name $port)
eval pkg=\$$pkg_name
if [ -z "$pkg" ]
then
pkg=$($MAKE -C $port -V PKGNAME)
eval $pkg_name=$pkg
fi
}
depends() {
local depend=
local depends_name=
local _deps=
local name=
local port=
local type
type=$1
port=$2
eval depends_name=DEPEND_${type}_$(port2name $port)
eval deps=\"\$$depends_name\"
if [ -z "$deps" ]
then
echo "Getting $type dependancies for $port" > /dev/stderr
if [ "$type" = "build" ]
then
depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V
LIB_DEPENDS -V RUN_DEPENDS)"
else
depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)"
fi
for depend in $depend_list
do
name=$(echo $depend | cut -f 2 -d ':')
depends runtime $name
_deps="$_deps $deps $name"
done
deps=" "
for depend in $_deps
do
if [ -z "`echo "$deps" | grep " $depend "`" ]
then
deps="$deps$depend "
fi
done
[ "`echo $deps | tr ' ' '\n' | sort`" = "`echo $deps | tr ' ' '\n' | sort
-u`" ]
depends_name=$depends_name
eval $depends_name=\"$deps \"
fi
}
run() {
set +e
trap "echo Terminating..." INT TERM EXIT
"$@"
_status=$?
[ $status -ne 0 ] || status=$_status
trap - INT TERM EXIT
set -e
}
build() {
local _deps=
local dep=
local port=
port=$1
depends build $port
_deps="$deps"
for dep in $_deps
do
port2pkg $dep
if [ ! -f $PACKAGES/All/$pkg.tbz ]
then
if ! build $dep
then
echo "Port $port failed due to dependency $dep" > /dev/stderr
return 255
fi
fi
done
echo "Building port $port..."
mkdir -p $LOCALBASE $PKG_DBDIR
for pkg in $_deps
do
port2pkg $pkg
run $PKG_ADD $PACKAGES/All/$pkg.tbz
[ $status -eq 0 ] || break
done
[ $status -ne 0 ] || run $MAKE -C $port clean build -DNO_DEPENDS -DBATCH
if [ $status -eq 0 ]
then
run $MAKE -C $port install package -DNO_DEPENDS -DBATCH
fi
if [ $status -eq 0 -o -z "$NO_CLEANUP" ]
then
port2pkg $port
run $PKG_DELETE -f $pkg
for pkg in $(echo $_deps | sort -r)
do
port2pkg $pkg
run $PKG_DELETE -f $pkg
done
run rm -rf $LOCALBASE || (run chflags -R 0 $LOCALBASE; run rm -rf
$LOCALBASE)
fi
if [ $status -ne 0 ]
then
echo "Port $port failed to build" > /dev/stderr
else
run $MAKE -C $port clean
fi
return $status
}
build /usr/ports/x11/xorg
#!/bin/sh
BUILDDIR=${BUILDDIR:-/usr/build}
LOCALBASE=${LOCALBASE:-/usr/local}
PORTSDIR=${PORTSDIR:-/usr/ports}
PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports}
PKG_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_pkg}
PACKAGES=${PACKAGES:-$BUILDDIR/packages}
MAKE="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR
PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES make"
set -e
mkdir -p $BUILDDIR $LOCALBASE $PKG_DBDIR $PACKAGES
[ -n "$(kldstat -v | grep unionfs)" ] || kldload unionfs
[ ! -e $BUILDDIR/.installing_port ] || rm -r `cat $BUILDDIR/.installing_port`
$BUILDDIR/.installing_port
port2name() {
echo $1 | sed 's|[/.-]|_|g'
}
port2pkg() {
local pkg_name=
local port=
port=$1; shift
eval pkg_name=PKG$(port2name $port)
eval pkg=\$$pkg_name
if [ -z "$pkg" ]
then
pkg=$($MAKE -C $port -V PKGNAME)
eval $pkg_name=$pkg
fi
}
depends() {
local depend=
local depends_name=
local _deps=
local name=
local port=
local type
type=$1
port=$2
eval depends_name=DEPEND_${type}_$(port2name $port)
eval deps=\"\$$depends_name\"
if [ -z "$deps" ]
then
echo "Getting $type dependancies for $port" > /dev/stderr
if [ "$type" = "build" ]
then
depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V
LIB_DEPENDS -V RUN_DEPENDS)"
else
depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)"
fi
for depend in $depend_list
do
name=$(echo $depend | cut -f 2 -d ':')
depends runtime $name
_deps="$_deps $deps $name"
done
deps=" "
for depend in $_deps
do
if [ -z "`echo "$deps" | grep " $depend "`" ]
then
deps="$deps$depend "
fi
done
[ "`echo $deps | tr ' ' '\n' | sort`" = "`echo $deps | tr ' ' '\n' | sort
-u`" ]
depends_name=$depends_name
eval $depends_name=\"$deps \"
fi
}
run_make() {
set +e
trap "true" INT TERM EXIT
$MAKE "$@"
status=$?
trap - INT TERM EXIT
set -e
}
build() {
local _deps=
local dep=
local port=
port=$1
depends build $port
_deps="$deps"
for dep in $_deps
do
port2pkg $dep
if [ ! -d $BUILDDIR/$pkg ]
then
if ! build $dep
then
echo "Port $port failed due to dependency $dep" > /dev/stderr
return 255
fi
fi
done
echo "Building port $port..."
for pkg in $_deps
do
port2pkg $pkg
mount -t unionfs -r -o noatime $BUILDDIR/$pkg $LOCALBASE
done
run_make -C $port clean build -DNO_DEPENDS -DBATCH
if [ $status -eq 0 ]
then
port2pkg $port
mkdir -p $BUILDDIR/$pkg
mount -t unionfs -o noatime $BUILDDIR/$pkg $LOCALBASE
echo $BUILDDIR/$pkg > $BUILDDIR/.installing_port
run_make -C $port install package -DNO_DEPENDS -DBATCH
rm $BUILDDIR/.installing_port
[ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE
fi
if [ $status -eq 0 -o -z "$NO_CLEANUP" ]
then
for pkg in $(echo $_deps | sort -r)
do
umount $LOCALBASE
done
fi
if [ $status -ne 0 ]
then
echo "Port $port failed to build" > /dev/stderr
port2pkg $port
rm -rf $BUILDDIR/$pkg || (chflags -R 0 $BUILDDIR/$pkg; rm -rf
$BUILDDIR/$pkg)
else
$MAKE -C $port clean
fi
return $status
}
build /usr/ports/x11/xorg
signature.asc
Description: This is a digitally signed message part.
