Hi Matt, Sorry for the delay in testing and responding. I appreciate the effort you put in this to date.
When I run a s3cmd sync like so: s3cmd sync --no-preserve --no-check-md5 --verbose --progress /blah/ s3://blah/ I see this in the output indicating it seemingly still generating md5 values: INFO: Compiling list of local files... INFO: Running stat() and reading/calculating MD5 values on 21726 files, this may take some time... INFO: [1000/21726] INFO: [2000/21726] INFO: [3000/21726] INFO: [4000/21726] INFO: [5000/21726] INFO: [6000/21726] INFO: [7000/21726] INFO: [8000/21726] INFO: [9000/21726] INFO: [10000/21726] INFO: [11000/21726] INFO: [12000/21726] INFO: [13000/21726] INFO: [14000/21726] I am not a github expert, but I believe I pulled down the correct s3cmd version. I did "Download Zip" from here: https://github.com/s3tools/s3cmd/tree/master Unzipped it and did python setup.py install The s3cmd --version output is: s3cmd version 1.5.0-beta1 I grabbed the latest github s3cmd master branch commit diff and compared it to the corresponding file from the downloaded zip and it matches, so unless I think I should be using the version with this patch incorporated. Mike On Mar 13, 2014, at 6:32 PM, Matt Domsch <m...@domsch.com> wrote: > Try the upstream master branch now with --no-check-md5. This should disable > all md5 calculations, thus also disable hardlinking and remote copying. > > Thanks, > Matt > > > On Wed, Mar 12, 2014 at 9:12 PM, Matt Domsch <m...@domsch.com> wrote: > There's a bug I see (and I created) in current upstream master, where > --no-check-md5 will still do the file I/O on local files to get the md5sums > for them, exactly to decide if it can do remote copying. That's annoying. > > This bug also means --no-check-md5 won't, as you might expect, disable remote > copying. As no one has asked to be able to disable remote copying, I never > coded for it. > > I'll think about this a bit. There's probably a cleaner way to solve both > problems. > > > > On Wed, Mar 12, 2014 at 5:14 PM, WagnerOne <wag...@wagnerone.com> wrote: > While this feature is fantastic, I can't find a lot of detail on it in > general. I wonder how to disable it? > > During initial uploads at least, our DirectConnect link seems to be faster in > copying the files themselves than s3cmd is at telling S3 to "remote copy" > objects. > > Would that simply be using s3cmd switch --no-check-md5 ? > > This would seem likely to reduce the RAM required to enumerate the source > files too? > > Thanks, > Mike > > > > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/13534_NeoTech > _______________________________________________ > S3tools-general mailing list > S3tools-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/13534_NeoTech_______________________________________________ > S3tools-general mailing list > S3tools-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/s3tools-general -- wag...@wagnerone.com "Every generation laughs at the old fashions, but follows religiously the new."-Thoreau
------------------------------------------------------------------------------
_______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
