On Thu, 17 Jul 2003 at 9:27am (-0400), Reuben D. Budiardja wrote:
[...]
> Now, I don't want to copy the whole thing in one sit since it's large. I want
> to do it incrementally during off-peak hours, so, say everyday between 2-3 AM
> I would copy 500 MB, and then the next day continue with another ~500MB, and
> the next day, etc, etc, until it's completed. The content of source directory
> will change, although not by much at all, so this needs to be able to keep
> trakc of that and sync with the changes.
[...]
Well.. seems most everone else has had a crack at it.. so my 2c.. ;)
Day one..
(cd /source/dir && tar -cpf - *) |
dd bs=1k count=500k |
ssh remote 'cd /target/dir && tar -xpvf'
.. day two...
(cd /source/dir && tar -cpf - *) |
dd bs=1k count=500k skip=500k |
ssh remote 'cd /target/dir && tar -xpvf'
... day three...
(cd /source/dir && tar -cpf - *) |
dd bs=1k count=500k skip=1000k |
ssh remote 'cd /target/dir && tar -xpvf'
... and so on.. each day increase the skip by 500 untill we're all done.
Once you get the first run complet keep it in sync with rsync..
rsync -a -e ssh /source/dir/ remote:/target/dir
... which will also clean up the files that got messed up by the truncated
tar. Down side... after the first day we're still tar'ing up stuff that
we're not going to send anywhere and that will take some time.
Another approch would be to just use rsync.. I can't think how to limit
rsync to only transfering X meg a night but it should be fairly easy to get
rsync to run for a set period in your off peak window each night.
So we'll set rsync to running...
rsync -a -e ssh /source/dir/ remote:/target/dir &
... find the pid of its transport...
ps auxwwww | grep 'ssh remote .* /target/dir
... get back something like...
root 30556 0.1 0.7 3200 2040 pts/5 S 08:48 0:00 ssh remote
rsync --server -logDtpr . /target/dir
... so after an hour has gone by, and it's 3am...
kill 30556
... which is cool since we've only killed that transport that leaves the
rync processes themslves to clean up any particuarlly transfered files.
Now repeat for as many days as needed untill rsync is finished - rsync will
automatically pick up where if left off. Hmm.. probably want to do another
ps before the kill to find out if rsync is still running or not.
Down side here... before it starts transfering anything at all rsync
generates a complet list of all the files in /source/dir .. depending on
just how many files there are this can take a while, something you might
want to take into account when choosing your window.
Again, once you're done, keep it in sync with periodic rsync's. Since the
daily deltas ought to be small each day's sync shouldn't take long.
M.
--
WebCentral Pty Ltd Australia's #1 Internet Web Hosting Company
Level 5, 100 Wickham St. Network Operations - Systems Engineer
PO Box 930, Fortitude Valley. phone: +61 7 3249 2552
Queensland, Australia 4006. pgp key id: 0x900E515F
--
redhat-list mailing list
unsubscribe mailto:[EMAIL PROTECTED]
https://www.redhat.com/mailman/listinfo/redhat-list
