* * * * * /website/config/sync.sh > /dev/null 2>&1
runs like this from crontab on each webserver.
On Thu, 12 Apr 2001, Dan Phoenix wrote:
> Date: Thu, 12 Apr 2001 13:31:24 -0700 (PDT)
> From: Dan Phoenix <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Subject: Re: rsync across nfs
>
>
>
> Here is my contribution to rsync.
> Attached is my current script that syncs off nfs.
> IT works along with cvs checking your commit log for checksum changes.
> This is a nice script for small networks that has gotten me by this far...
> i will be writing another one for bigger networks shorty pushing data
> in concurrent threads with perl.
>
> IF you would like to post this on your website to replace other script you
> have this one has alot more examples on what you can do :)
>
>
> Regards,
>
> Dan Phoenix.
>
>
> On Thu, 12 Apr 2001, Dan Phoenix wrote:
>
> > Date: Thu, 12 Apr 2001 13:12:13 -0700 (PDT)
> > From: Dan Phoenix <[EMAIL PROTECTED]>
> > To: Eric Whiting <[EMAIL PROTECTED]>
> > Cc: [EMAIL PROTECTED], [EMAIL PROTECTED]
> > Subject: Re: rsync across nfs
> >
> >
> >
> > We went over this about 5 months ago ...pushing data is way slower that
> > getting it....that is why this approach was taken.
> > But that is a good idea as server load just gets hit to hard.
> >
> >
> >
> > On Wed, 11 Apr 2001, Eric Whiting wrote:
> >
> > > Date: Wed, 11 Apr 2001 21:08:51 -0600
> > > From: Eric Whiting <[EMAIL PROTECTED]>
> > > To: [EMAIL PROTECTED]
> > > Cc: [EMAIL PROTECTED], [EMAIL PROTECTED]
> > > Subject: Re: rsync across nfs
> > >
> > > I have a similar setup and goal as Tim.
> > >
> > > As another approach to the problem I've been pushing the sync's out to
> > > the destinations rather than having the destinations pull the data. I
> > > have 2G RAM on the source box (Solaris with a netapps disk) and I push
> > > the data via rsync/ssh to destinations in parallel with a simple perl
> > > script that forks a child per destination. (and loops over a few top
> > > level dirs)
> > >
> > > That doesn't really relate to Dan's question does it?? Oh well, I
> > > guess my opinion is that if you can rsync using rsh/ssh (push or pull)
> > > you will be much happier than if you have to use NFS.
> > >
> > >
> > > eric
> > >
> > >
> > >
> > >
> > > [EMAIL PROTECTED] wrote:
> > > >
> > > > One thing you can do to decrease your load is to ad the -W option. If you're
>reading it via nfs to do the checksumming, you have to read the entire file anyway,
>so you might as well just move the entire file, instead of wasting processor power,
>and
> > > > reading the entire file twice (or more, actually).
> > > > The single-processor bsd machine would max out at one transfer at a time,
>probably. using it as rsyncd, though, gives you the advantage of letting you use the
>"max connections" option, and having the individual machines retry until they
>succeed, thus
> > > > controlling the load on the solaris machine. I am developing a similar
>solution for our system, where we have a single master copy of a set of tools, with
>identical copies all over the world that must be kept up to date.
> > > >
> > > > Tim Conway
> > > > [EMAIL PROTECTED]
> > > > 303.682.4917
> > > > Philips Semiconductor - Colorado TC
> > > > 1880 Industrial Circle
> > > > Suite D
> > > > Longmont, CO 80501
> > > >
> > > > [EMAIL PROTECTED]@[EMAIL PROTECTED] on 04/11/2001 03:08:28 PM
> > > > Sent by: [EMAIL PROTECTED]
> > > > To: [EMAIL PROTECTED]@SMTP
> > > > cc:
> > > > Subject: rsync across nfs
> > > > Classification:
> > > >
> > > > I currently use rsync across an nfs mount.
> > > > This nfs server is a sparc solaris machine
> > > > mounting to approx 30 freebsd and 10 linux machines.
> > > >
> > > > When a typical rsync occurs to replicate data across all these
> > > > machines....
> > > > they all rsync
> > > > /usr/local/bin/rsync -az --delete --force /home/cvs/website/config /website
> > > >
> > > > where /home/cvs is an nfs mount and /website is just the local drive.
> > > > Problem is they all hit the solaris box at once driving it to load average
> > > > as high as 75 for the 10-20 seconds that this occurs.
> > > >
> > > > My question is would i be better off taking a single processor freebsd
> > > > machine....running a rsync server socket type deal...all getting 40
> > > > machines or so to connect this way....or would that be worse.
> > > > I like having all my machines update at once....just looking for an
> > > > efficient way that a) i could even dedicate 1 box as just an rsync server
> > > > and b) a single processor machine could actually handle that kind of load.
> > > > My guess is i would prob have to stripe some drives together as IO may be
> > > > a problem with that many webservers connecting at once? or will cpu be
> > > > more a factor.
> > > >
> > > > I have considered changing rsync times around so that they don;t all
> > > > connect to same server at once...this is done from crontab btw, but I like
> > > > the way it currently is. Any suggestions would be much appreciated.
> > > >
> > > > --
> > > > Dan
> > > >
> > > > +------------------------------------------------------+
> > > > | BRAVENET WEB SERVICES |
> > > > | [EMAIL PROTECTED] |
> > > > | make installworld |
> > > > | ln -s /var/qmail/bin/sendmail /usr/sbin/sendmail |
> > > > | ln -s /var/qmail/bin/newaliases /usr/sbin/newaliases |
> > > > +______________________________________________________+
> > >
> >
> >
>
#!/bin/bash
#this script does a bunch of much need things to take load off nfs
#don't ever name this script rsync.sh or you will screw up my ps statement
#First it checks if rsync is running..if it is it exits
#Second it does checksums on commit logs to see if anything has changed
#..if not...it exits.
#Third we check if this is linux,solaris or freebsd and execute right rsync statement
#if it passed other 2 tests....
#note for freebsd:ln -s /usr/local/bin/bash /bin/bash
#i have added example on to do diff rsync with certain machines depending on hostnames
#hostnames of my machines have been hidden for brevity sake but you get the idea
#/home/cvs is my nfs mount on all webservers and /website is a local drive
#sync.sh runs from crontab like this...* * * * * /website/config/sync.sh > /dev/null
2>&1
#this way it checks your cvs commit log every minute for changes
#Coded by Dan Phoenix
THISCPU=`uname`
HOSTNAME=`hostname`
export THISCPU HOSTNAME
PATH=/usr/local/bin:/usr/bin:/bin
case "$THISCPU"
in
FreeBSD) # For FreeBSD
RUN=`ps awwx | grep rsync | grep -v grep | wc -l`
;;
Linux) # For Linux
RUN=`ps awwx | grep rsync | grep -v grep | wc -l`
;;
SunOS) #Solaris
RUN=`ps -ef | grep rsync | grep -v grep | wc -l`
;;
esac
if [ "$RUN" -gt "0" ]; then
echo already running $RUN
exit 1
fi
if [ ! -f /tmp/commitlog ]; then
touch /tmp/commitlog
fi
sum1=`sum < /home/cvs/commitlog`
sum2=`sum < /tmp/commitlog`
if [ "$sum1" = "$sum2" ]; then
echo nothing to do
exit 0
fi
case "$THISCPU"
in
FreeBSD) # For FreeBSD
case "$HOSTNAME"
in
some.host.com)
echo we are somehost
/usr/local/bin/rsync -Wa --delete --force
/home/cvs/website/config /website
;;
some.host.com)
echo we are someotherhost
/usr/local/bin/rsync -Wa --delete --force
/home/cvs/website/config /website
;;
some.other.host.com)
echo we are yetanotherhost
/usr/local/bin/rsync -Wa --delete --force
/home/cvs/website/config /website
;;
*)
echo we are a freebsd webserver
/usr/local/bin/rsync -Wa --delete --force /home/cvs/website /usr
;;
esac
;;
Linux) # For Linux
case "$HOSTNAME"
in
some.linux.host.com)
echo we are somelinuxbox
/usr/bin/rsync -Wa --force --exclude=website/include/semcache.*
/home/cvs/website /mnt/xtra
;;
*)
/usr/bin/rsync -Wa --delete --force /home/cvs/website/config
/website
esac
echo we are linux
;;
SunOS) #Solaris
case "$HOSTNAME"
in
some.solaris.box.com)
echo we are somesolarisbox
/usr/local/bin/rsync -Wa --delete --force /home/cvs/website /opt
;;
*)
/usr/local/bin/rsync -Wa --delete --force
/home/cvs/website/config /website
esac
echo we are solaris
;;
esac
cp -f /home/cvs/commitlog /tmp/commitlog
exit 0
