Hello misc@,
[sorry this got much longer than I wanted it to]
I'm pretty sure many other people have already thought about, or even
used this, for faster compilation of kernels:
distcc
I wanted to wait with this message until I have everything together
concerning patches for distcc integration to bsd.port.mk and possibly
other parts of the tree. But since I read about the upcoming hackathon
and call for fast machines (I know they are still needed) I'm sending
this now. This gives people some time to this out (and improve it) in
advance to the hackathon.
The security problems assoziated with distrubed compilation DO NOT
matter in a secure/trusted environment such as a hackathon.
The time gained by using distcc should outweight the time to set it up
by far.
First some numbers using my private servers to compile GENERIC[.MP]:
# NOTE:
# These are no "clean" benchmarks since all hosts see some activity during
# testing, but it should not significantly influence them.
# X: make -jX or env MAKE_PARALLEL=X make
# time in MM:SS
GENERIC:
X time hosts
1 5:30 ahb64
4 4:26 ahb64 ahbabe
4 4:12 ahb64 ahbabe ahblaptop
6 3:12 ahb64 ahbabe ahblaptop
8 3:12 ahb64 ahbabe ahblaptop ahb1200
10 2:36 ahb64 ahbabe ahblaptop ahb1200
12 2:31 ahb64 ahbabe ahblaptop ahb1200
10 2:36 <group1>
12 2:30 <group1>
14 2:36 <group1>
groups:
-------
<group1>: ahb64 ahb1 ahblaptop ahb1200 router
<group2>: ahb64 ahblaptop ahb1200 router
hosts:
------
DISTCC_DIR=/tmp # Same for ALL hosts
# grep /tmp /etc/fstab
swap /tmp mfs rw,-s=204800,noexec,noatime,nosuid,nodev 0 0
sysctl hw.setperf=100 # apmd disabled
# dmesg | grep -e GENERIC -e cpu -e 'real mem' | grep -v mainbus
------------------------ ahb64 -----------------------------
OpenBSD 4.0-current (GENERIC) #1350: Fri Jan 19 16:42:39 MST 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Athlon(tm) 64 Processor 3000+ ("AuthenticAMD" 686-class, 512KB L2
cache) 1.81 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3
real mem = 2145873920 (2095580K)
cpu0: Cool'n'Quiet K8 1801 MHz: speeds: 1800 1000 MHz
OpenBSD 4.0-current (GENERIC) #1327: Mon Jan 1 17:15:26 MST 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Athlon(TM) XP 2600+ ("AuthenticAMD" 686-class, 512KB L2 cache) 1.92
GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE
real mem = 1073295360 (1048140K)
------------------------------------------------------------
------------------------ ahblaptop -------------------------
OpenBSD 4.0-stable (GENERIC) #1: Sat Jan 6 15:34:55 MST 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: mobile AMD Athlon(tm) XP 2500+ ("AuthenticAMD" 686-class, 512KB L2 cache)
1.87 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE
real mem = 536375296 (523804K)
OpenBSD 4.0-stable (GENERIC) #1: Sat Jan 6 15:34:55 MST 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Duron(tm) Processor ("AuthenticAMD" 686-class, 64KB L2 cache) 1.20 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE
real mem = 536440832 (523868K)
OpenBSD 4.0-stable (GENERIC) #1: Sat Jan 6 15:34:55 MST 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: VIA Esther processor 1500MHz ("CentaurHauls" 686-class) 1.50 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,CMOV,PAT,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,TM,SBF,SSE3,EST,TM2
cpu0: unknown Enhanced SpeedStep CPU, msr 0x08100f1308000f13
cpu0: using only highest and lowest power states
cpu0: Enhanced SpeedStep 1500 MHz (1004 mV): speeds: 1500, 800 MHz
cpu0: RNG AES AES-CTR SHA1 SHA256 RSA
real mem = 1006137344 (982556K)
OpenBSD 4.0-current (GENERIC) #1349: Tue Jan 16 16:55:56 MST 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) M processor 1.40GHz ("GenuineIntel" 686-class) 1.40
GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,TM,SBF,EST,TM2
real mem = 1063743488 (1038812K)
cpu0: Enhanced SpeedStep 1400 MHz (1116 mV): speeds: 1400, 1300, 1200, 1100,
1000, 900, 800, 600 MHz
------------------------------------------------------------
# COMPILE instructions
#---------------------
# Start distccd on ALL build hosts (as root) including "client"
# NOTE: For security reasons one might want to use IPSEC between hosts.
distccd --daemon --allow=127.0.0.1/24 --allow=10.2.0.0/24
# It is also possible to use distcc over an ssh link without distccd.
# build kernel
# FIXME should be picked up automatically after setting DISTCC_HOSTS
# XXX Order hosts from fastest -> slow
setenv DISTCC_HOSTS "127.0.0.1 rhost1 rhostn" # Order see above
make clean && make CC=distcc depend && make -jX CC=distcc
The hosts are all connected by 100Mb/s with very cheap/slow switches in
between and over a local IPSEC (I confirmed that added latency does not
hurt that much, compared to gained security).
Combining 4.0-release and current works very well.
IMHO an
improvement in build times of >100%
is quite nice to have, without any additional costs.
Using several fast machines connected by 1Gb/s this should improve quite
a bit more.
The best thing being that config generates a -jX save Makefile.
While this will not work for building the base system (which I consider
a bug, which should be fixed IMHO), it does not matter for a kernel
hackathon.
I know that distcc will never be included into base (GPL), but perhaps
somebody will fell the itch to write a sane/secure/fast BSD replacement
once base and ports can take full advantage from it.
-----
Some feedback on this TOPIC would be really terrific, because I think it
would speed up development (kernel/base/ports/X) and fun for everybody
involved in OpenBSD. At least everybody, who has more than 1 "fast" (see
my own hosts) computers of the same arch and "close" OpenBSD version.
-----
Since misc@ does not accept attachments a port of distcc (with a REALLY
UGLY workaround) can be found here (based on Fritz Elfert's port send to
ports@ on 2006-10-11):
http://bihlmaier.org/ports/devel_distcc.tgz
Regards,
ahb
SORRY, for the ugly, or at least kind of unsorted instructions, but
the hackathon is only a mere 2 weeks away.
This is not directly related to the hackathon, but just to show how easy
integration could be. (Yes I'd be willing to put time into adding
BROKEN_PARALLEL and MAKE_PARALLEL_MAX to a lot of ports)
# COMPILE instructions
# ports
setenv DISTCC_HOSTS "127.0.0.1 rhost1 rhostn"
env MAKE_PARALLEL=X make install # X number of ALL CPUs x 2 or 3;
Some numbers concerning ports, sorry not too many yet:
ports:
------
Primary (local) host is ahb64 (using 127.0.0.1 to address himself).
# cd port; /usr/bin/time <make> build |& tee -a build.log; grep
'real.*user.*sys' build.log
# <make>
# make: make
# make-j: make -jX
# distcc: setenv DISTCC_HOSTS 'list of hosts'; env MAKE_PARALLEL=X make
# hX means that used X is set as MAKE_PARALLEL_MAX and cannot be raised
# B = broken; P = needs patch
X distcc make-j make hosts
archivers/
bzip 8 2 2 1 4 <group2>
bzip2 8 7 7 7 4 <group2>
astro/
audio/
amarok 8 626 618 617 4 <group2>
audiacity #
BROKEN_PARALLEL="breaks built"
benchmarks/
biology/
books/
cad/
kicad 10 1406 2011 1990 5 <group1>
8 846 2071 2050 4 <group2> # Version:
2007-01-15
chinese/
comms/
converters/
databases/
postgresql 10 137 205 207 5 <group1>
mysql 10 509 966 966 5 <group1>
devel/
P nasm 10 22 28 27 5 <group1> # USE_GMAKE=Yes
distfiles/
editors/
B abiword #
BROKEN_PARALLEL="breaks built"
P vim 10 64 108 108 5 <group1> # USE_GMAKE=Yes
education/
emulators/
games/
wesnoth 8 1602 1012 1005 4 <group2>
geo/
graphics/
gimp 8 808 894 4 <group2>
10 857 969 948 5 <group1>
P libart 10 19 23 23 5 <group1> # USE_GMAKE=Yes
infrastructure/
inputmethods/
japanese/
java/
korean/
lang/
python 8 #
BROKEN_PARALLEL="breaks built"
ruby 8 101 110 92 4 <group2>
mail/
math/
misc/
multimedia/
net/
samba 8 #
BROKEN_PARALLEL="breaks built"
news/
packages/
palm/
plan9/
print/
P teTeX # USE_GMAKE=Yes
productivity/
russian/
security/
shells/
tcsh 10 27 24 19 5 <group1>
sysutils/
telephony/
textproc/
www/
P mozilla-firefox h4 1922 2695 2710 4 <group2> #
MAKE_PARALLEL_MAX=4
h4 1796 2877 2859 5 <group1>
x11/
kde/
base3 #
mplayer 10 281 309 306 5 <group1>
tellico 8 255 235 236 4 <group2>
B qt3 #
BROKEN_PARALLEL="breaks built"
# Patch against (current) ports/infrastructure/mk/bsd.port.mk
# or get it there:
# http://bihlmaier.org/patches/patch-infrastructure_mk_bsd_port_mk
#------------------------------------------------------------------------------#
--- infrastructure/mk/bsd.port.mk.orig Mon Jan 15 14:43:52 2007
+++ infrastructure/mk/bsd.port.mk Wed Jan 31 17:51:04 2007
@@ -78,7 +78,7 @@
_ALL_VARIABLES ?= HOMEPAGE DISTNAME BUILD_DEPENDS RUN_DEPENDS \
REGRESS_DEPENDS USE_GMAKE MODULES FLAVORS \
NO_BUILD NO_REGRESS SHARED_ONLY ONLY_FOR_ARCHS IS_INTERACTIVE \
- BROKEN MULTI_PACKAGES PSEUDO_FLAVORS \
+ BROKEN BROKEN_PARALLEL MULTI_PACKAGES PSEUDO_FLAVORS \
REGRESS_IS_INTERACTIVE DISTFILES DIST_SUBDIR \
PERMIT_DISTFILES_CDROM PERMIT_DISTFILES_FTP \
CONFIGURE_STYLE USE_LIBTOOL SEPARATE_BUILD \
@@ -291,6 +291,10 @@
DESTDIR ?= ${WRKINST}
MAKE_FLAGS ?=
+DISTCC_HOSTS?=
+MAKE_PARALLEL?=
+MAKE_PARALLEL_MAX?=
+BROKEN_PARALLEL?=
LIBTOOL_FLAGS ?=
.if !defined(FAKE_FLAGS)
FAKE_FLAGS = ${DESTDIRNAME}=${WRKINST}
@@ -498,6 +502,22 @@
PORTPATH ?=
${WRKDIR}/bin:/usr/bin:/bin:/usr/sbin:/sbin:${DEPBASE}/bin:${LOCALBASE}/bin:${X11BASE}/bin
+# distcc
+.if !empty(DISTCC_HOSTS)
+PORTPATH=${LOCALBASE}/libexec/distcc:${WRKDIR}/bin:/usr/bin:/bin:/usr/sbin:/sbin:${DEPBASE}/bin:${LOCALBASE}/bin:${X11BASE}/bin
+MAKE_ENV+=DISTCC_HOSTS=${DISTCC_HOSTS:Q}
+MAKE_ENV+=DISTCC_DIR=${TMPDIR}
+CONFIGURE_ENV+=DISTCC_HOSTS=${DISTCC_HOSTS:Q}
+CONFIGURE_ENV+=DISTCC_DIR=${TMPDIR}
+.endif
+.if !empty(MAKE_PARALLEL) && empty(BROKEN_PARALLEL)
+. if empty(MAKE_PARALLEL_MAX)
+MAKE_FLAGS+=-j${MAKE_PARALLEL}
+. else
+MAKE_FLAGS+=-j${MAKE_PARALLEL_MAX}
+. endif
+.endif
+
# Add any COPTS to CFLAGS. Note: programs that use imake do not
# use CFLAGS! Also, many (most?) ports hard code CFLAGS, ignoring
# what we pass in.
@@ -2061,6 +2081,31 @@
${_BUILD_COOKIE}: ${_CONFIGURE_COOKIE}
.if ${NO_BUILD:L} == "no"
@${ECHO_MSG} "===> Building for ${FULLPKGNAME}${_MASTER}"
+
+# distcc
+.if !empty(DISTCC_HOSTS) && empty(MAKE_PARALLEL)
+ @echo ""; \
+ echo "*** WARNING: MAKE_PARALLEL not set"; \
+ echo "*** Without only the first entry in DISTCC_HOSTS will be used."; \
+ echo "*** There is no general rule for setting MAKE_PARALLEL,"; \
+ echo " as a hint set it to twice the number of CPUs in cluster."; \
+ echo ""
+.endif
+
+.if !empty(BROKEN_PARALLEL)
+ @echo ""; \
+ echo "*** WARNING: Parallel building is broken:"; \
+ echo "*** ${BROKEN_PARALLEL}"; \
+ echo "*** Will ONLY built on FIRST host in DISTCC_HOSTS"; \
+ echo ""
+.elif !empty(MAKE_PARALLEL_MAX)
+ @echo ""; \
+ echo "*** WARNING: Parallel building ONLY works with"; \
+ echo "*** MAKE_PARALLEL <= ${MAKE_PARALLEL_MAX}"; \
+ echo " automatically using maximum allowed."; \
+ echo ""
+.endif
+
.if ${VMEM_WARNING:L} == "yes"
@echo ""; \
echo "*** WARNING: you may see an error such as"; \
#------------------------------------------------------------------------------#
# Benchmark procedure: (tcsh syntax)
setenv X 10
setenv REAL_DISTCC_HOST '127.0.0.1 ahb1.vpn.ahb.local ahblaptop.vpn.ahb.local
ahb1200.vpn.ahb.local router.vpn.ahb.local'
make depends && \
make clean && sleep 5 && make fetch &&
setenv DISTCC_HOSTS "$REAL_DISTCC_HOSTS" && \
/usr/bin/time env MAKE_PARALLEL=$X make build |& tee build.log && \
make clean && sleep 5 && \
unsetenv DISTCC_HOSTS && \
/usr/bin/time env MAKE_PARALLEL=$X make build |& tee -a build.log && \
make clean && sleep 5 && \
/usr/bin/time make build |& tee -a build.log && \
grep 'real.*user.*sys' build.log
