Re: rsync freezes when copying several million files

2006-08-22 Thread Wayne Davison
On Tue, Aug 22, 2006 at 09:37:42AM +0200, [EMAIL PROTECTED] wrote: > From what I've been reading in the FAQ and the ML rsync will use 100 > bytes per files (6 GB), and I'll probably get a x3 "bonus" (18 GB) for > using the --delete and -H options. Those are old numbers for older versions. Rsync n

Re: rsync freezes when copying several million files

2006-08-22 Thread Wayne Davison
On Mon, Aug 21, 2006 at 04:36:23PM +0200, Tomasz Chmielewski wrote: > There is no "other side": The question is really, what are the other processes doing? When rsync is sending the file list, it has 2 processes, one creating the list (the sender) and one receiving the list. Then, both sides sor

Re: rsync freezes when copying several million files

2006-08-22 Thread [EMAIL PROTECTED]
Paul Slootman wrote: >> How could I get these files rsynced ? > > The way I do that is to rsync separate logical directories. > E.g. I know that I don't have any files that are hardlinked between /usr > and /var, so it's no problem to rsync those separately from each other, > etc. The great major

Re: rsync freezes when copying several million files

2006-08-22 Thread Paul Slootman
On Tue 22 Aug 2006, [EMAIL PROTECTED] wrote: > > I'm using Dirvish to backup my servers. For those who haven't heard > about it it's sort of "front end" for rsync making a copy of the files > for the first backup, and then copying modified files and hardlinking > unmodified files every day. > I ha

Re: rsync freezes when copying several million files

2006-08-22 Thread Robert Siemer
> >I'm using Dirvish to backup my servers. For those who haven't heard > >about it it's sort of "front end" for rsync making a copy of the files > >for the first backup, and then copying modified files and hardlinking > >unmodified files every day. dirvish is using the services of rsync to let the

Re: rsync freezes when copying several million files

2006-08-22 Thread Tomasz Chmielewski
[EMAIL PROTECTED] wrote: Hello I have a similar problem. I want to rsync a huge number of files and I'm running out of memory. You might give the machine more swap. As it appeared, it wasn't a freeze - it just took a lot of time to generate the list. When I came back in the morning, the files

Re: rsync freezes when copying several million files

2006-08-22 Thread [EMAIL PROTECTED]
Hello I have a similar problem. I want to rsync a huge number of files and I'm running out of memory. I'm using Dirvish to backup my servers. For those who haven't heard about it it's sort of "front end" for rsync making a copy of the files for the first backup, and then copying modified files a

Re: rsync freezes when copying several million files

2006-08-21 Thread Tomasz Chmielewski
> On Mon 21 Aug 2006, Tomasz Chmielewski wrote: (...) >> Process 23311 detached >> >> >> [EMAIL PROTECTED] kernel]# strace -p 23310 >> Process 23310 attached - interrupt to quit >> select(6, [5], [], NULL, {4, 80}) = 0 (Timeout) >> select(6, [5], [], NULL, {60, 0}) = 0 (Timeout) >> se

Re: rsync freezes when copying several million files

2006-08-21 Thread Paul Slootman
On Mon 21 Aug 2006, Tomasz Chmielewski wrote: > I just tried to copy several million files with rsync. > > Unfortunately, it seems to freeze: > > # rsync --delete-before -a -H -v --progress --stats /srv/backuppc-data/ > /mnt/iscsi_backup/backuppc/ > building file list ... > 7891370 files to con