On Tue, 2012-07-31 at 13:30 -0500, Matt Domsch wrote: > On Fri, Jul 20, 2012 at 09:27:04AM -0500, Domsch, Matt wrote: > > On Fri, Jul 20, 2012 at 07:06:33AM -0500, John Sauter wrote: > > > Matt, > > > > > > When I first attempted to use your merged s3cmd, I found that specifying > > > > > > --exclude='*' --include='*.gpg' > > > > > > gave a different result. Previously, it would include all the gpg files > > > in the tree; now it includes only the gpg files at the top level. You > > > noted that you were now applying excludes/includes at local os.walk() > > > time--I wonder if this might be the cause of the changed behavior. > > > > > > I have reverted to the previous s3cmd, but I will install your new one > > > and re-test if you need more details or have a workaround. > > > > You are correct, the way excludes/includes are processed now this > > won't work because the * matches all directories during the walk. > > You'll have to somehow explicitly include the directories that lead to > > each of the *.gpg files. I need to think about this some more for an > > example. > > I have found that by using: > --exclude='*' --include='small/' --include='*.gpg' > > I can cause the files in small/somefile.gpg to be included even though > the rest of the directories alongside small/ are excluded. How deep > is your directory tree? Could you put these patterns in a file and > pull them in with --include-from and --exclude-from ? > > The problem I was solving was avoiding the need to recurse through a > ton of subdirectories (e.g. netapp .snapshot*) that were meaningless > to my sync, yet the os.walk() would be forced to decend through them. > The time for my sync was greatly reduced by implementing excludes at > os.walk() time. The cost of such capability is needing to be more > explicit about what's excluded and included knowing it applies first > during os.walk(), not only afterwards to the set discovered by > os.walk(). > > > > > There's also a fairly large failure introduced by the merge such that > > sync_local2remote looks awful, unnecessarily duplicated code and > > errors with undefined references. I'll try to fix that up too. > > Fixed these in my merge tree. Also pulled in the newer pull request > by Christopher Noyes to include the encoding type in the content-type string. > > > > By the way, it would be easier to discuss your new s3cmd if it had a > > > different version number than the s3cmd it is derived from. > > Done. > > -version = "1.1.0-beta3" > +version = "1.1.0-beta3-mdomsch-merge-20120730" > > > Still hoping Michal Ludvig will pull all these changes into his tree. > > Happy to take feedback on functionality of my 'merge' branch. > > Thanks, > Matt >
Matt,
One of the directories I use s3cmd to back up is both wide and deep: it
contains all of the sound effects and microphone notes for the stage
plays I have worked on since 1998. Adding another line to a file
whenever I add a directory to it would be a pain.
As an alternative, how about using a different keyword (or keywords) to
do the os.walk excludes? That would restore the meaning of the old
keywords and still let you gain your performance improvements.
John Sauter (john_sau...@systemeyescomputerstore.com)
signature.asc
Description: This is a digitally signed message part
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
