RE: Initialising a mirror

[Tony]

"slow rsync"    limited by bandwidth (or whatever) compared to....

FAST (LAN), SLOW (Internet), FAST (LAN)  from "pointA" to "pointB"

 

100Gb of data    -- how much/many of what is changing?

How many files?  How big? (ranges)  where changed?

How many (major) directories?

What kind of window and how big?

What are the preferable failure modes?

    You do NOT have a choice of whether it fails.

    You DO have a choice of which way it fails.

 

Looks like -p of scp preserves times and modes.  ?? owner and group ??

 

Probably worth doing a fast look to see just what it is that you actually have on both ends.

Intentions and reality quite often manage to do a disconnect.

You tend to get better answers if you state what you did (cut&paste)

as opposed to what you thought you did.

Generally when you have troubles they are not the same.

 

-----Original Message-----
From: Cyrille Bollu [mailto:[EMAIL PROTECTED]
Sent: Wednesday, October 19, 2005 4:35 AM
To: [EMAIL PROTECTED]
Cc: rsync@lists.samba.org
Subject: RE: Initialising a mirror

Hi,

thank you for your answer.

But what do you mean by "slow rsync"? Do you mean limiting bandwith? I think this will not be an option with 100Gb of data to synchronise...

However, I like your idea with the LAN: I think I will bring my mirror to the remote site :-)

Just for the record, I didn't made the standard mistake, having things one directory level off from where they should be.

I was hoping doing a "scp -rp" would preserve all the necessary information needed by rsync but apparently not... Maybe I should use tar?

Regards,

Cyrille

<[EMAIL PROTECTED]> a écrit sur 19/10/2005 11:00:07 :

> Standard mistake is to somehow get things one directory level off
> from where they should be.
>  
> If you have the available disk space (both ends)
> Do a fast rsync from the primary to locally available fast disk
> space (same computer or LAN)
> Do a slow rsync from that disk space to another set of available
> disk space over slow/unreliable network.
> Assuming the slow rsync completed without problems,
>     do a fast rsync from the second disk space to the legitimate target..
>  
> Source makes a big distinction between foo/ and foo
> They both transfer the same files.
> They differ in where they transfer the file to  (BY ONE LEVEL)
> Both make sense and are quite reasonable.
> If you do both, the effects are similar to untarring twice with one
> level difference between them.
>  
> I know I've done it a number of times, and it is no fun over remote
> and often bad internet connections.
> With large volumes, it is not apparent when you've done it to yourself.
>  
>  
> WARNING  -- these scripts work but are probably not the best nor the
> most aesthetic.
> ---just a wee bit too informative and I'm too lazy to sanitize so
> I'm not posting to the list.---
>     ?? hosts allow/deny ignore with auth users present ??
> The major sucurity concerns are NOT permanently losing all my data
> rather than eavesdroppers.
> The internet connection(s) are ocassionally flaky to non-existent,
> and everything needs to assume
> that they will chose the most inopportune moment to do something I don't like.
> Essentially the stunt is that /home/rsync-foo is rsync'd with other
> such over slow and bad connections.
> With bad internet I've got (not shown) multiple off-site backups AND
> the ability to bring them all upt-to-date
> if there is a connection (any, not necessarily the preferred -- some
> connect, some don't connect)
>  
>  
>  
> ---Timing depends on what is different.  Typically this one is about
> 10 to 15 minutes  (about 10G) total.
> If large MySql tables are rearranged, tends to take rather longer.
> real    192m41.906s
>  
>  
> real    12m18.512s
> sent 13750 bytes  received 7577551 bytes  10279.35 bytes/sec
> total size is 14013850399  speedup is 1846.04
> Tue Oct 18 07:28:00 CDT 2005
> rsync-sjs-dwg
> [EMAIL PROTECTED] /root]# cat rsync-sjs-dwg
> #!/bin/sh
>  
> rsync client
> # staged backup
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
>  
> time rsync -avz --progress --partial --timeout=1750 --password-
> file=/etc/rsync.secrets/rsync-sjs-dwg \
>         [EMAIL PROTECTED]::rsync-sjs-dwg/* /home/rsync-sjs-dwg/
> date; echo rsync-sjs-dwg
>  
> ----rsync server   (actually is an intermediate, not the primary)
> This is an extra "junk" computer with a decent hard drive.
> This serves to get a fast
> # etc/rsyncd.conf
> pid file = /etc/rsyncd.pid
> [rsync-sjs-dwg]
>         comment = backup of sjs dwg
>         path = /home/rsync-sjs-dwg
>         use chroot = true
>         read >>         list = false
>         uid = root
>         gid = root
>         auth users = rsync-sjs-dwg
>         secrets file = /etc/rsyncd.secrets
>         strict modes = true
>         hosts allow = 10.0.0.1 pete-162 wf
>         hosts deny = *.*.*.*
> ---rsync "client"  (This is a PRIMARY)
> [EMAIL PROTECTED] root]# cat rsync-sjs-dwg
> #!/bin/bash
> # rsync-sjs-dwg  SJS_Drawings/ title/
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg  --timeout=750 \
>         /tmp/rsync/OPENED       [EMAIL PROTECTED]::rsync-sjs-dwg/
> for name in title SJS_Drawings ; do
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
>         /home/dwg/$name         [EMAIL PROTECTED]::rsync-sjs-dwg/
> done
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg  --timeout=750 \
>         /tmp/rsync/CLOSED       [EMAIL PROTECTED]::rsync-sjs-dwg/
> echo 'rsync-sjs-dwg'
> date
> date -u
>  
>
>  
>  
>  
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:
> [EMAIL PROTECTED] Behalf Of Cyrille Bollu
> Sent: Wednesday, October 19, 2005 2:58 AM
> To: rsync@lists.samba.org
> Subject: Initialising a mirror

>
> Dears,
>
> I would like to mirror data from 2 servers connected together via
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps).
>
> I'm sure rsync is one of the best tool to keep these data in sync
> but how should I use it to initialise the mirror?
>
> I'm currently testing the solution with 10Gb of data to keep in
> sync. But on my lines it would take more than 40 hours to initially
> create the mirror! And in production the data to keep in sync will
> reach 100Gb... :-(
>
> So, I went with a laptop on the remote site and "scp -rp" the data
> on it. I then got back to my site and "scp -rp" it on the mirror to
> initialise it.
>
> But apparently I forgot something because when rsync still sends all
> the files from the remote site to my mirror when I first run it.
>
> Does anybody has an idea about what I'm doing wrong?
>
> Does anybody has a solution to this problem?
>
> Regards,
>
> Cyrille

-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html