Hi Using 's3cmd sync' eats up lots of memory. q
I'm trying to sync some 500.000 files. It gets to about 30% of RAM before it starts "Retrieving list of remote files" which consumes more memory until it starts to swap and I have to kill it. $ ./s3cmd sync --debug --delete-removed --skip-existing ... INFO: Compiling list of local files... INFO: Retrieving list of remote files for s3://... ^CSee ya! I have tried a couple of versions (0.9.9.91,q 0.9.9 and 1.0.0-rc1) and they all have this problem. Eventually it starts to swap and I have to kill the s3cmd process before it even begins to upload. Using fewer files works just fine. Comparing this to the 'put' command. It looks like put uses the same amount of RAM to collect the local files and the starts uploading them. INFO: Applying --exclude/--include INFO: Summary: 566259 local files to upload [... uploading ...] 3349 of 3349 100% in 0s 8.48 kB/s done Python 2.6.4 Looks like this guy has the same problem: http://sourceforge.net/mailarchive/forum.php?thread_name=499530.91465.qm%40web30601.mail.mud.yahoo.com&forum_name=s3tools-general <http://sourceforge.net/mailarchive/forum.php?thread_name=499530.91465.qm%40web30601.mail.mud.yahoo.com&forum_name=s3tools-general> ------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev _______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
