Hi Philip, thanks for the contribution. Your patch is too large to be integrated just before s3cmd 1.0.0 comes out but I will definitely keep it in mind for future.
I've got some big plans on parallel processing in s3cmd and parallel transfer is indeed a part of that. Other possible improvements include parallel MD5 hashing of local files, early transfer of files that are known to need upload/download (at the moment any transfer starts only after all MD5 hashing is done which can take a lot of time), etc. I'm definitely open for improvements in this area, however not before 1.0.0 :) Michal On 08/29/2010 07:30 AM, Philip Corliss wrote: > I've just finished testing a patch to enable the transfer operations > to be threaded. This speeds up operations dramatically (4-8x in my > tests), especially when you have many small files. Hopefully this can > be integrated into the main release. > > You can find the patch at > http://github.com/pcorliss/s3cmd-modification/blob/master/patch.parallel.20100827_1954 > > Or you can find a forked version with the patch already applied at > http://github.com/pcorliss/s3cmd-modification > > Thanks, > Philip Corliss > http://blog.50projects.com > > ------------------------------------------------------------------------------ > Sell apps to millions through the Intel(R) Atom(Tm) Developer Program > Be part of this innovative community and reach millions of netbook users > worldwide. Take advantage of special opportunities to increase revenue and > speed time-to-market. Join now, and jumpstart your future. > http://p.sf.net/sfu/intel-atom-d2d > _______________________________________________ > S3tools-general mailing list > S3tools-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/s3tools-general > ------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev _______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
