On Tue, 29 Jul 2003, Josip Rodin wrote: > On Tue, Jul 29, 2003 at 10:00:16AM +0200, Attila Nagy wrote: > > >For Debian mirrors, we have practically no use in features like > > >compression, > > >RCS, deltas and whatever else. To eliminate the initial delay in rsyncing, > > >now that would be a useful thing. Did you try that, is that improved with > > >cvsup? > > No, I didn't try it with Debian mirrors, but I did with FreeBSD. It > > contains much more files than Debian, because of the CVS repositories > > checked out. > > It starts the synchronization much earlier than rsync > > debian/ mirrors don't contain CVS stuff, and most probably never will. When > you look at it, they are quite trivial, because just a handful of files are > ever updated -- only new files are added. Hence, results with updating, > diffing files, they probably don't matter to us. We need an experiment with > our stuff. What needs to be done to test this?
1. Run cvsupd on a machine. 2. Run cvsup to update a mirror from that first machine. Just look at top and so on during this time. :) The big expected difference is due to a local cache of filenames that is kept on the client side and threaded approach to start sending needed files as soon as they are discovered, before doing a full directory tree traversal and so on. > > and which is even better, the memory requirement is very low (maximum 4-6 > > MBs). I guess only the latter worth the change. We had many rsync mirrors > > and our machine ran out of memory very soon. Just imagine a mirror of > > four-five bigger projects (like NetBSD, FreeBSD, OpenBSD and Debian) and > > voila, you need minimum 1 GBs of RAM just for the rsync in memory file > > list. > > I didn't notice any noteworthy problems with a machine with 96 MB RAM, > although I mass-mirror stuff less than half the size of debian/ (linux, > gnu, php etc). The memory usage for the rsync process is about 40 megs for debian/ and increases with every file/dir/link/whatever added. /Mattias Wadenstein
