Just note this patch has nothing to do with memory consumption vs performance. It just avoids allocating memory that was left unused anyway.
On Sat, Oct 2, 2021, 12:51 devzero--- via rsync <rsync@lists.samba.org> wrote: > >In the exchange I argued that proper use of ram as a buffer would have > cut down backup time to minutes instead of days. > > could you give an example where rsync is slowing things down so much due > to ram constraints or inefficient ram use? > > please mind that disk bandwith and file copy bandwith is not the same. > random i/o and seek time is the culprit. > > why should rsync use ram for buffering data it copies, if the linux kernel > / vm subsystem already does this? > > roland > > *Gesendet:* Samstag, 02. Oktober 2021 um 12:07 Uhr > *Von:* "Rupert Gallagher via rsync" <rsync@lists.samba.org> > *An:* makov...@gmail.com, rsync@lists.samba.org > *Betreff:* Re: [PATCH] Reduce memory usage > If you look at my previous exchange in the list, I raised the need for > more ram usage via a tool option. In the exchange I argued that proper use > of ram as a buffer would have cut down backup time to minutes instead of > days. At the time, my proposal was dismissed by someone saying that rsync > uses as much ram as it needs. I still feel the need to free rsync from this > mindless constraint, while also welcoming ram usage optimisations such as > yours in this patch. How hard can it be to allow rsync to use 1GB of ram > instead of 100MB? The benefit would be huge. In my case, where a supermicro > server uses a shared bus to transfer data from two disks, the overhead > caused by frequent small buffer IO is so high that backup time is still > huge. And I am using server hardware! PC and laptops are even worse. > > RG > > > > > -------- Original Message -------- > On Sep 26, 2021, 13:54, Jindřich Makovička via rsync < > rsync@lists.samba.org> wrote: > Hi, > > When using rsync to back up the file system on my laptop, containing a > pretty much default linux desktop, I was wondering how rsync uses over > 100MB of RAM it allocates. > > It turned out that most of the memory is used for the arrays of > file_struct pointers, most of which end up unused - much more than the > actual file_struct entries. In my case, the peak usage was 135MB of > pointers, and just 1.5MB of the file_struct entries themselves. > > The problem seems to be that the default file_list allocation parameters > predate the incremental recursion, which allocates a huge number of small > file lists, while AFAICS originally rsync allocated just one large list. > > Applying the attached patch, which reduces the default allocation to 32 > pointers, and preallocates 32K pointers only for the main file lists in > send_file_list and recv_file_list, reduces the peak memory usage in my case > from 142MB to 12MB. > > Regards, > -- > Jindřich Makovička > -- Please use reply-all for most replies to avoid omitting the mailing > list. To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
