On Thu, Aug 23, 2012 at 09:13:33AM -0700, Dennis Glatting wrote: > ... > > The second primer in the committer handbook seems to indicate that it is > > difficult to run an SVN mirror. This appears to me to be the biggest > > drawback. I have been using CVS and perforce for years, but subversion > > is new to me. > > > > Thanks. > > I have several cases where I cannot have machines going to the Internet to > update for various reasons, including WAN loading and policies, but I can > have them hit local mirrors. I have three sites: > ...
I asked Dennis if a description of how I handle my private read-only SVN
mirrors might help, and his reaction was positive, so....
First: here's what I'm trying (and succeeding, AFAICT) to accomplish:
* I have a "build" machine at home, where I currently track stable/8,
stable/9, and head (on different slices) on a daily basis. I also
update the installed ports on that machine daily.
Periodically (by default, weekly), I also install the "Sunday"
snapshot of stable/8 on my 2 "production" machines (and update their
installed ports). [I expect to switch these machines to stable/9
soon. The build machine has already been building stable/9 kerenls
for them. Shortly after that switch, I expect to stop tracking
stable/8.]
Separate from the above, I also track stable/8, stable/9, and head on
a daily basis on my laptop; I also update its installed ports on a
daily basis. One of the reasons I do this is to gain actual
experience with the code I'm expecting to install on the "production"
machines: the build machine is used only for that, and when its job is
done for the day, I power it off.
Since the laptop spends some of the time it may be building FreeBSD
or ports disconnected from a network (e.g., while I'm commuting
to work), I prefer to just keep a local mirror of the SVN repo
on my laptop.
* So, I started with one of the recent "seed" repositories in each case,
and have my build machine (freebeast.catwhisker.org) use svnsync to
re-sync its local mirror with the repo on svn.freebsd.org.
* As somewhat of a relic of my experiences doing this with CVS mirrors,
I perform the re-sync in 2 stages overnight. After the 2nd one, I
update the local /usr/ports hierarchy (which used to be a CVS working
direwctory, and is now an SVN working copy).
The timing is intended so that during the daytime, my local mirror
is "stable": if I were to check out (say) head from that mirror
at 5AM, noon, 6PM, and 10PM, I would get the same result each
time (even if I didn't specify a GRN).
* As another relic of doing something similar with CVSup, I use the
login "cvsupin" to actually do this work.
* Somewhere, I acquired the perception that if, after setting up the
SVN mirror on freebeast, if I were to try to also use svnsync on my
laptop, that effort would end up talking back to svn.freebsd.org --
which is NOT what I want, for 2 reasons:
* I want to keep my mirrors in sync with one another, so I have
reasonable assurance that (kernel configs and other local
provisioning aside) I'm running the same code on my laptop that I
would be running on the others.
* I am also trying to avoid using more sv,freebsd.org resources tyhan
necessary.
Therefore, for re-syncing the SVN mirrors on my latop, I use rsync(1).
As a bit of an implementation detail, I consign my repositories to
separate file systems, and I have symlinks with names that depict the
function pointing to appropriate places in the file system -- e.g.:
d134(9.1-P)[13] df /repo
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/ada0s4f 40622796 22955728 14417246 61% /repo
d134(9.1-P)[14] ls -F /repo
cvs/ svn/
d134(9.1-P)[15] ls -F /repo/*
/repo/cvs:
local/
/repo/svn:
freebsd/ local/
d134(9.1-P)[16] ls -lT /{cvs,svn}
lrwxr-xr-x 1 root wheel 9 Sep 27 08:49:09 2010 /cvs -> /repo/cvs
lrwxr-xr-x 1 root wheel 9 Sep 27 08:49:09 2010 /svn -> /repo/svn
d134(9.1-P)[17] ls -F /svn/freebsd
doc/ ports/ src/
d134(9.1-P)[18]
I have attached a copy of the script I use to do this. It isn't really
set up for more general consumption -- it hard-codes the name of my
build machine, for example.
Here are some sample entries from /etc/crontab. First, for the build
machine:
# Local additions
30 01 * * * cvsupin /usr/local/etc/svn-repo -f
30 03 * * * cvsupin /usr/local/etc/svn-repo -f -w
/usr/ports
And from my laptop:
# Local additions
40 01 * * * cvsupin /usr/local/etc/svn-repo -l
40 03 * * * cvsupin /usr/local/etc/svn-repo -l -w
/usr/ports
# Copied from freebeast, for when I'm on the road....
# 30 01 * * * cvsupin /usr/local/etc/svn-repo -f
# 30 03 * * * cvsupin /usr/local/etc/svn-repo -f -w
/usr/ports
One other thing: I rather like to have the logs from the svnsync
handy, so the script copies those, to. There's a bit of sleight-of-hand
involved, as the user cvsupin can't create files in /var/log, and
I still want the log files there.
PS: Yes, I also rotate the svnsync log files. Sample entry from
/etc/newsyslog.conf:
/var/log/svnsync.log cvsupin: 644 12 * $M1D0 JC
Please feel free to make use of the ideas -- or even the code.
Peace,
david
--
David H. Wolfskill da...@catwhisker.org
Depriving a girl or boy of an opportunity for education is evil.
See http://www.catwhisker.org/~david/publickey.gpg for my public key.
#! /bin/sh
set -e
t_mode=""
mode=""
LOGFILE="svnsync.log"
RSYNC="/usr/local/bin/rsync"
RSYNC_SVN_ARGS="-aHAX"
RSYNC_SVN_SOURCE="rsync://freebeast/svn/freebsd"
RSYNC_SVN_TARGET="/svn/"
RSYNC_LOG_ARGS="-ptgoAX"
RSYNC_LOG_SOURCE="rsync://freebeast/svn_log/svnsync.log"
TMP_DIR="/tmp"
LOG_DIR="/var/log"
REPO_BASE="file:///svn/freebsd/"
REPO_LIST="src:src/base ports:ports doc:doc"
SVNSYNC="/usr/local/bin/svnsync"
SVN="/usr/local/bin/svn"
WORKING_COPIES=""
while getopts "flTw:" arg; do
case ${arg} in
f) mode="freebsd"; continue;;
l) mode="local"; continue;;
T) t_mode="echo "; continue;;
w) WORKING_COPIES="$OPTARG"; continue;;
?) echo "Usage: $0 -{f|l} [-T] [-w working_copy...]" >&2; exit 2;;
esac
done
if [ -n "$t_mode" ]; then
LOG_DIR="/dev"
LOGFILE="tty"
fi
if [ -z "$mode" ]; then
echo "$0: Need to specify -l (for rsync from local machine) or -f (for
svnsync from svn.freebsd.org)" >&2
exit 1;
fi
if [ "$mode" = "local" ]; then
cmd="$t_mode $RSYNC $RSYNC_SVN_ARGS $RSYNC_SVN_SOURCE $RSYNC_SVN_TARGET"
logit='$t_mode $RSYNC $RSYNC_LOG_ARGS $RSYNC_LOG_SOURCE $TMP_DIR && \
$t_mode /bin/cp -p $TMP_DIR/$LOGFILE $LOG_DIR && \
$t_mode /bin/rm $TMP_DIR/$LOGFILE'
elif [ "$mode" = "freebsd" ]; then
cmd="$t_mode $SVNSYNC sync --non-interactive \$REPO >>$LOG_DIR/$LOGFILE 2>&1"
logit='$t_mode echo "svnsync for $NAME ended at `date` exit status $?"
>>$LOG_DIR/$LOGFILE'
else
echo "$0: Logic error; mode => $mode; not recognized" >&2
exit 3
fi
log_init="$t_mode echo 'svnsync started at `date`' >>$LOG_DIR/$LOGFILE"
status=-1
eval "$log_init"
until [ $status -eq 0 ]; do
if [ "$mode" = "local" ]; then
eval "$cmd"
status="$?"
eval "$logit"
else
for ITEM in $REPO_LIST; do
NAME=$( echo $ITEM | sed -e 's/:.*$//' )
SUFFIX=$( echo $ITEM | sed -e 's/^.*://' )
REPO="${REPO_BASE}${SUFFIX}"
eval "$cmd"
status="$?"
eval "$logit"
done
fi
done
for dir in $WORKING_COPIES; do
cmd="$SVN update $dir >/var/tmp/"$(basename $dir)"_update 2>&1"
$t_mode eval "$cmd"
done
exit $?
pgpUcOoMdTrFD.pgp
Description: PGP signature
