Hi, I have a virtual machine with 2G of memory. On this VM there is a directory tree with 33.3 million files in it. When attempting to rsync (rsync -PSHav --delete /source /dest) this tree from one directory to another on the same host, rsync uses all the memory and is killed by oom-killer.
This host is Debian oldstable so has $ rsync --version rsync version 3.1.2 protocol version 31 The normal operation of this VM does not require more than 2G of memory, but I doubled it to 4G anyway. Unfortunately rsync still uses all the memory and is killed. Most advice I can find on decreasing rsync memory usage advises to split the job up into batches. By issuing one rsync for each directory within /source I was able to make this work. The interesting thing is though, the split of file numbers between sub-directories is very uneven with the majority of them (31.5 million of the 33.3 million) being in just one of the sub-directory trees. I am kind of surprised that rsync has such a problem going just that little bit further with the last 2 million. Is there any scope for improvement with the incremental recursion code? If I upgraded the version of rsync could I expect this to work any better? I could also give the host a massive swap file. It currently has just 1G of swap, which all gets used in the failure case. I could add more but I fear that the job will go so slow it will not complete in a reasonable time. I don't know if the -H option is causing extra memory usage here; unfortunately it is necessary as there are hardlinks in there. Some years old advice says to disable incremental recursion with --no-i-r. As incremental recursion was added to reduce memory usage this seems counter-intuitive to me, but this advice is all over the Internet… These are all things I will investigate before settling for the "split into multiple jobs" approach; just wondered if anyone has any shortcuts for me. Thanks, Andy -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
