On Thu, Aug 11, 2005 at 01:04:25AM +0100, Carlos Silva wrote: [snip] > or metadata.xml. This way, users with slow connections don't download > almost 1MB of info every time they sync. Yes, your example occupies 1MB of space. However, it does NOT equate to 1MB of bandwidth with each sync.
If you go and dig up the rsync stats output, you will get some data like this: (I generated this artificaily, it's the difference between the 20050801 and 20050802 snapshots, as distributed by rsync) ==== Number of files: 119194 Number of files transferred: 923 Total file size: 94682691 bytes Total transferred file size: 1956000 bytes Literal data: 1956000 bytes Matched data: 0 bytes File list size: 2898171 File list generation time: 40.292 seconds File list transfer time: 0.000 seconds Total bytes sent: 24233 Total bytes received: 3662519 sent 24233 bytes received 3662519 bytes 74479.84 bytes/sec total size is 94682691 speedup is 25.68 ==== Now look at the recived size data. We got a total of 3662519 bytes. Of that, 2898171 bytes was the file list alone. The file list accounts for 79% of the traffic! One of the other reasons for the actual file traffic being negliable is rsync's compression, over a set of files being transferred, 'DESCRIPTION="GPL-2"' should turn up often enough that it gets compressed (probably as two symbols, as 'DESCRIPTION="' is more common). The diff between 20050801 and 20050802 is only 862668 bytes (uncompressed) (and 157728 bytes when bzip2'd), so either rsync needs some serious work done in it's file list code, or we should consider if rsync is still the best fit for portage. Another alternative would also be to reduce the number of the files in the tree. (Merging digests and manifests would shave off ~20k files, converting the metadata cache files into large single files would shave off another ~20k). -- Robin Hugh Johnson E-Mail : [EMAIL PROTECTED] Home Page : http://www.orbis-terrarum.net/?l=people.robbat2 ICQ# : 30269588 or 41961639 GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
pgpf9aDjSoKKB.pgp
Description: PGP signature
