Ah ha! Thanks for the fix.
It'd be useful to see the counts of "new files to upload" and "updated files to
upload" called out separately in the verbose "INFO: Summary" section.
I thought I might take a crack at that, but got lost pretty fast in s3cmd src.
:|
Mike
On Apr 21, 2014, at 7:19 PM, Matt Domsch <m...@domsch.com> wrote:
> Thanks for the report.
>
> Try with this patch. The reported number of files to upload in the Summary
> line was correct, but the number y in [x of y] was incorrect, it was only
> using the "new files to upload" count, not "new files plus updated files"
> count.
>
> diff --git a/s3cmd b/s3cmd
> index 913d04a..467dc07 100755
> --- a/s3cmd
> +++ b/s3cmd
> @@ -1311,8 +1311,9 @@ def cmd_sync_local2remote(args):
> update_count = len(update_list)
> copy_count = len(copy_pairs)
> remote_count = len(remote_list)
> + upload_count = local_count + update_count
>
> - info(u"Summary: %d local files to upload, %d files to remote copy,
> %d remote files to delete" % (local_count + update_count, copy_count,
> remote_count))
> + info(u"Summary: %d local files to upload, %d files to remote copy,
> %d remote files to delete" % (upload_count, copy_count, remote_count))
>
> _set_remote_uri(local_list, destination_base, single_file_local)
> _set_remote_uri(update_list, destination_base, single_file_local)
> @@ -1349,8 +1350,8 @@ def cmd_sync_local2remote(args):
> total_size = 0
> total_elapsed = 0.0
> timestamp_start = time.time()
> - n, total_size = _upload(local_list, 0, local_count, total_size)
> - n, total_size = _upload(update_list, n, local_count, total_size)
> + n, total_size = _upload(local_list, 0, upload_count, total_size)
> + n, total_size = _upload(update_list, n, upload_count, total_size)
> n_copies, saved_bytes, failed_copy_files = remote_copy(s3,
> copy_pairs, destination_base)
>
> #upload file that could not be copied
>
>
>
> On Mon, Apr 21, 2014 at 3:45 PM, WagnerOne <wag...@wagnerone.com> wrote:
> example s3cmd for the below
>
> /usr/bin/s3cmd -c /home/ec2-user/.s3cfg sync --delete-removed --no-preserve
> --verbose --progress /localdirectory/ s3://mybucket/directory/
>
>
> syncs were running build commit d52d5edcc916512e979917f04abcea19d3a25af7
> Date: Sat Apr 12 20:40:16 2014 -0500
>
>
> s3cmd, when syncing a particular directory to S3 is telling me this:
>
> INFO: Found 1567064 local files, 1566681 remote files
> INFO: Verifying attributes...
> [list of a few "disabled md5 check" objects]
> INFO: Summary: 14263 local files to upload, 417 files to remote copy, 0
> remote files to delete
> WARNING: Module python-magic is not available. Guessing MIME types based on
> file extensions.
> /file.jpg -> s3://mybucket/file.jpg [1 of 111]
> ^M 4096 of 83840 4% in 0s 403.67 kB/s^M 12288 of 83840 14% in
> 0s 1188.59 kB/s failed
> WARNING: Upload failed: /file.jpg (timed out)
> WARNING: Retrying on lower speed (throttle=0.00)
> WARNING: Waiting 3 sec...
> /file.jpg -> s3://mybucket/file.jpg [1 of 111]
>
> and continues to the sync finish...
>
> ^M 4096 of 15216 26% in 0s 433.79 kB/s^M 15216 of 15216 100% in
> 0s 201.74 kB/s done
> /file.txt -> s3://mybucket/file.txt [14263 of 111]
>
> then it begins to do remote copies as applicable for the sync.
>
>
> I've not seen this before where the items to be uploaded in the progress
> output doesn't match the INFO line statement of objects to upload.
>
> I also think I have not encountered the "(timed out)" error before and wonder
> if maybe something related to that function is somehow mucking up the
> "objects to be uploaded" count display. Just an idea.
>
>
> Another one from the same set of batched sync commands:
>
> INFO: Found 615171 local files, 587979 remote files
> INFO: Verifying attributes...
> [list of a few "disabled md5 check" objects]
> INFO: Summary: 6446 local files to upload, 21557 files to remote copy, 0
> remote files to delete
> WARNING: Module python-magic is not available. Guessing MIME types based on
> file extensions.
> /file.png -> s3://mybucket.png [1 of 5741]
> ...
> /file.mp3 -> s3://mybucket/file.mp3 [6446 of 5741]
> ...
> Done. Uploaded 233380819 bytes in 2578.0 seconds, 88.41 kB/s. Copied 21557
> files saving 373663253 bytes transfer.
>
> That one also had an initial file timeout...
>
>
> Another one... almost matches! :)
>
> INFO: Summary: 68371 local files to upload, 29128 files to remote copy, 0
> remote files to delete
> WARNING: Module python-magic is not available. Guessing MIME types based on
> file extensions.
> /file.pdf -> s3://mybucket/file.pdf [1 of 68321]
> ...
> completed with "[68371 of 68321]"
>
>
> FWIW, as I inspect the logs, I did see this on subsequent syncs that did not
> time out on the initial object upload.
>
>
> Here is one that is starting to maybe indicate what's happening maybe...
>
> INFO: Found 206744 local files, 206740 remote files
> INFO: Verifying attributes...
> INFO: Summary: 6164 local files to upload, 15 files to remote copy, 0 remote
> files to delete
> WARNING: Module python-magic is not available. Guessing MIME types based on
> file extensions.
> /file.jpg -> s3://mybucket/file.jpg [1 of 4]
>
> It's wanting to upload 6164 local files (not sure why though), yet that "[1
> of 4]" seems to make more sense with the object count difference (4).
>
> Also, some of the syncs ran normal - as expected with equal counts on either
> side, etc.
>
> Thanks,
> Mike
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> S3tools-general mailing list
> S3tools-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform_______________________________________________
> S3tools-general mailing list
> S3tools-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
--
Michael Wagner
www.wagnerone.com
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
