Hi,
thank you for your answer.
But what do you mean by "slow rsync"? Do you mean limiting bandwith? I think this will not be an option with 100Gb of data to synchronise...
However, I like your idea with the LAN: I think I will bring my mirror to the remote site :-)
Just for the record, I didn't made the standard mistake, having things one directory level off from where they should be.
I was hoping doing a "scp -rp" would preserve all the necessary information needed by rsync but apparently not... Maybe I should use tar?
Regards,
Cyrille
<[EMAIL PROTECTED]> a écrit sur 19/10/2005 11:00:07 :
> Standard mistake is to somehow get things one directory level off
> from where they should be.
>
> If you have the available disk space (both ends)
> Do a fast rsync from the primary to locally available fast disk
> space (same computer or LAN)
> Do a slow rsync from that disk space to another set of available
> disk space over slow/unreliable network.
> Assuming the slow rsync completed without problems,
> do a fast rsync from the second disk space to the legitimate target..
>
> Source makes a big distinction between foo/ and foo
> They both transfer the same files.
> They differ in where they transfer the file to (BY ONE LEVEL)
> Both make sense and are quite reasonable.
> If you do both, the effects are similar to untarring twice with one
> level difference between them.
>
> I know I've done it a number of times, and it is no fun over remote
> and often bad internet connections.
> With large volumes, it is not apparent when you've done it to yourself.
>
>
> WARNING -- these scripts work but are probably not the best nor the
> most aesthetic.
> ---just a wee bit too informative and I'm too lazy to sanitize so
> I'm not posting to the list.---
> ?? hosts allow/deny ignore with auth users present ??
> The major sucurity concerns are NOT permanently losing all my data
> rather than eavesdroppers.
> The internet connection(s) are ocassionally flaky to non-existent,
> and everything needs to assume
> that they will chose the most inopportune moment to do something I don't like.
> Essentially the stunt is that /home/rsync-foo is rsync'd with other
> such over slow and bad connections.
> With bad internet I've got (not shown) multiple off-site backups AND
> the ability to bring them all upt-to-date
> if there is a connection (any, not necessarily the preferred -- some
> connect, some don't connect)
>
>
>
> ---Timing depends on what is different. Typically this one is about
> 10 to 15 minutes (about 10G) total.
> If large MySql tables are rearranged, tends to take rather longer.
> real 192m41.906s
>
>
> real 12m18.512s
> sent 13750 bytes received 7577551 bytes 10279.35 bytes/sec
> total size is 14013850399 speedup is 1846.04
> Tue Oct 18 07:28:00 CDT 2005
> rsync-sjs-dwg
> [EMAIL PROTECTED] /root]# cat rsync-sjs-dwg
> #!/bin/sh
>
> rsync client
> # staged backup
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
>
> time rsync -avz --progress --partial --timeout=1750 --password-
> file=/etc/rsync.secrets/rsync-sjs-dwg \
> [EMAIL PROTECTED]::rsync-sjs-dwg/* /home/rsync-sjs-dwg/
> date; echo rsync-sjs-dwg
>
> ----rsync server (actually is an intermediate, not the primary)
> This is an extra "junk" computer with a decent hard drive.
> This serves to get a fast
> # etc/rsyncd.conf
> pid file = /etc/rsyncd.pid
> [rsync-sjs-dwg]
> comment = backup of sjs dwg
> path = /home/rsync-sjs-dwg
> use chroot = true
> read > > list = false
> uid = root
> gid = root
> auth users = rsync-sjs-dwg
> secrets file = /etc/rsyncd.secrets
> strict modes = true
> hosts allow = 10.0.0.1 pete-162 wf
> hosts deny = *.*.*.*
> ---rsync "client" (This is a PRIMARY)
> [EMAIL PROTECTED] root]# cat rsync-sjs-dwg
> #!/bin/bash
> # rsync-sjs-dwg SJS_Drawings/ title/
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /tmp/rsync/OPENED [EMAIL PROTECTED]::rsync-sjs-dwg/
> for name in title SJS_Drawings ; do
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /home/dwg/$name [EMAIL PROTECTED]::rsync-sjs-dwg/
> done
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /tmp/rsync/CLOSED [EMAIL PROTECTED]::rsync-sjs-dwg/
> echo 'rsync-sjs-dwg'
> date
> date -u
>
>
>
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:
> [EMAIL PROTECTED] Behalf Of Cyrille Bollu
> Sent: Wednesday, October 19, 2005 2:58 AM
> To: rsync@lists.samba.org
> Subject: Initialising a mirror
>
> Dears,
>
> I would like to mirror data from 2 servers connected together via
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps).
>
> I'm sure rsync is one of the best tool to keep these data in sync
> but how should I use it to initialise the mirror?
>
> I'm currently testing the solution with 10Gb of data to keep in
> sync. But on my lines it would take more than 40 hours to initially
> create the mirror! And in production the data to keep in sync will
> reach 100Gb... :-(
>
> So, I went with a laptop on the remote site and "scp -rp" the data
> on it. I then got back to my site and "scp -rp" it on the mirror to
> initialise it.
>
> But apparently I forgot something because when rsync still sends all
> the files from the remote site to my mirror when I first run it.
>
> Does anybody has an idea about what I'm doing wrong?
>
> Does anybody has a solution to this problem?
>
> Regards,
>
> Cyrille
-- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
