On 01/12/2011 07:01 AM, Sean McCleary wrote: > I regularly use s3cmd for deleting a bucket and recursively copying > another bucket to replace the deleted bucket. This process takes a > very long time. I would wager that processing commands like this > would benefit greatly from parallel execution as far a time is concerned. > > After a quick search I noticed that someone forked s3cmd and made a > parallel version: https://github.com/pcorliss/s3cmd-modification > > Would this forked project be good to merge into s3cmd? Is there a > parallel option that I am missing in the existing s3cmd?
Hi Sean, at the moment there is no support for parallel processing in s3cmd. With the recent introduction of multi part uploads this would indeed be a very welcome feature to have and it's one of the two top priorities on my todo list. Another way to speed up things is to use persistent http connections (I used to maintain a s3cmd-speedup branch in SVN some time ago). I vaguely remember there was some problem for users behind HTTP proxies, however I assume we should be able to fall back to the current slow mode if running from behind a proxy. Ideally we'll have these two approaches combined for a super fast operation :-) BTW I'm aware of the the parallel-fork but must admit I haven't yet reviewed the patch. Michal ------------------------------------------------------------------------------ Protect Your Site and Customers from Malware Attacks Learn about various malware tactics and how to avoid them. Understand malware threats, the impact they can have on your business, and how you can protect your company and customers by using code signing. http://p.sf.net/sfu/oracle-sfdevnl _______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
