On Mon 21 Aug 2006, Tomasz Chmielewski wrote:
> I just tried to copy several million files with rsync.
>
> Unfortunately, it seems to freeze:
>
> # rsync --delete-before -a -H -v --progress --stats /srv/backuppc-data/
> /mnt/iscsi_backup/backuppc/
> building file list ...
> 7891370 files to consider
>
>
> And it waits here. It was already copied once sussessfully, and this time, I
> just wanted to do an update.
> Destination filesystem is iSCSI, on another device in LAN.
> I can list the files there, create new files etc., so it's not a device
> problem.
>
>
> As there were many files, it used up lost of RAM:
>
> # free
> total used free shared buffers cached
> Mem: 524460 520928 3532 0 344 6604
> -/+ buffers/cache: 513980 10480
> Swap: 1534072 1100884 433188
>
> But still, there is a lot of swap left.
>
>
> I can see that rsync is in "D" state, and doesn't use much CPU time:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 23311 root 5 -17 685m 440m 488 D 0.0 86.0 2:05.94 rsync
> 23310 root 0 -17 578m 608 608 S 0.0 0.1 8:41.55 rsync
You have 512MB of RAM, and it's using 685MB of virtual memory.
> So I made a strace, and rsync wasn't doing much:
>
> [EMAIL PROTECTED] kernel]# strace -p 23311
> Process 23311 attached - interrupt to quit
>
> (ctrl + C after one minute)
It was busy swapping in, to handle the SIGSTOP from the strace....
> Process 23311 detached
>
>
> [EMAIL PROTECTED] kernel]# strace -p 23310
> Process 23310 attached - interrupt to quit
> select(6, [5], [], NULL, {4, 800000}) = 0 (Timeout)
> select(6, [5], [], NULL, {60, 0}) = 0 (Timeout)
> select(6, [5], [], NULL, {60, 0}) = 0 (Timeout)
> select(6, [5], [], NULL, {60, 0}
>
> (ctrl + C after two minutes or so)
>
>
> It's in that state for several hours now.
> Will it eventually wake up and finish, or perhaps something doesn't work
> properly?
The other side is probably busy building its list of files for the
comparison. Can you strace that process?
> Those "Timeout" messages in strace output worry me and make me think
> something didn't go as planned.
>
> I tried rsync 2.6.6, and then upgraded to 2.6.8, but both froze (at least it
> looks like that to me).
Did you upgrade to 2.6.8 on both sides? Newer versions are better at
using less memory, although I don't know how much difference is between
2.6.6 and 2.6.8 in that respect.
Paul Slootman
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
