Re: [S3tools-general] Memory issues

2010-11-02 Thread Martin Wickman
Michal Ludvig skrev 2010-11-02 09:16: >> * The 'put' command should have a flag for checking md5/file size before >> uploading, like sync. I hacked it like this: > What's the difference between /sync/ and /put/ then? 'sync' checks for delete files as well (and requires lots of memory). >> * Ther

Re: [S3tools-general] Memory issues

2010-11-02 Thread Michal Ludvig
Hi Martin, On 11/02/2010 08:30 PM, Martin Wickman wrote: * You may want to include the parallel patch (http://blog.50projects.com/p/s3cmd-modifications.html). It speeds up s3cmd _considerably_. We're talking x10 here. I know about this patch but it's too huge for inclusion in -rc phase just be

Re: [S3tools-general] Memory issues

2010-11-02 Thread Martin Wickman
Michal Ludvig skrev 2010-10-30 20:45: >On 10/30/2010 12:52 AM, Martin Wickman wrote: >> Using 's3cmd sync' eats up lots of memory. >> >> I'm trying to sync some 500.000 files. It gets to about 30% of RAM >> before it starts "Retrieving list of remote files" which consumes more >> memory until i

Re: [S3tools-general] Memory issues

2010-10-30 Thread Michal Ludvig
On 10/30/2010 12:52 AM, Martin Wickman wrote: > Using 's3cmd sync' eats up lots of memory. q > > I'm trying to sync some 500.000 files. It gets to about 30% of RAM > before it starts "Retrieving list of remote files" which consumes more > memory until it starts to swap and I have to kill it. Hi M

[S3tools-general] Memory issues

2010-10-29 Thread Martin Wickman
Hi Using 's3cmd sync' eats up lots of memory. q I'm trying to sync some 500.000 files. It gets to about 30% of RAM before it starts "Retrieving list of remote files" which consumes more memory until it starts to swap and I have to kill it. $ ./s3cmd sync --debug --delete-removed --skip-exist