It has gotten more noticeable lately, but as part of a new
deployment in the midwest against the us-west-1 S3 domain (our
primary servers are in the UK and run against S3 in Amsterdam). We
have gotten random (but several times a month) issues against S3
Amsterdam from London; now it's become dysfunctional and a severe
issue. The workaround in the original post seems to be working (day
2 coming up).
So is it Amazon? Who knows. I suspect rather s3cmd failure to
handle redirects from Amazon - a few posts on that:
https://github.com/boto/boto/issues/2207
http://serverfault.com/questions/330453/s3cmd-fails-too-many-times
which was what caused us to try to specify the absolute s3 domain
URI instead of a generic one.
I'll repeat what was noted before - there is clearly no size
issue related to our failures.
We're hoping someone from the s3cmd team will note this issue is
becoming prevalent in the community (there are a handful of google
hits on "s3cmd errono 104", and that's usu the tip of the actual
prevalence iceberg ... ) and address it as suggested in the original
post.
Cheers -
Stephen
On 3/9/2016 7:31 PM, Jeremy Wadsack
wrote:
This is interesting. We've seen the same error arise from an S3
library. We used to see it irregularly but lately the number of
tines we get it has increased.
My research suggested it was related to file size and
specifically to not using multipart chunks. We upload about 60
files a day using s3cmd, usually in the 10MiB to 100MiB range
and I haven't seen that error out in a while. We upload
thousands of small files (~10KB) with our library and have not
seen issues with that. Where we so see the failure is in very
large files in the 100MiB range.
We're addressing this by setting the chunk size to 100 MiB. We
just made this change Monday so I'm still waiting to see if it
really helps.
"Socket reset by peer" is pretty generic so these could be
unrelated issues (especially as I'm seeing it in a different
library). I'll keep your bucket changes in mind.
Have you seen an increase in these errors lately? Could this be
a change at AWS?
Jeremy Wadsack
On Mar 9, 2016 6:38 PM, "Stephen Sykes"
< stephen.sy...@4js.com>
wrote:
We are seeing (and have
seen online references to) random connection reset errors
during put operations. Following online hints,
we have seen these errors fixed by using
- the --region=<region> command
line argument (odd, since it is not noted to effect the
put command)
- setting the bucket region in the config file via both
configuration setting (its in unclear which is
having the effect)
Neither of these works 100% of the time; so far we hope the
combination will prove to be robust.
File size seems unrelated to the error - we have successful
500MiB uploads followed by 3 MiB file upload failures.
When we get the errors, they last for some indeterminate
period (portions of an hour up to several hours), long
enough to be certain the above settings resolve the issue
(testing with and without the settings show success and
failure during an extended, hour or so long, testing
period.) Again, none of the settings works 100% of the time
- oddly sometimes one setting works and the other does not,
and vice versa.
This seems like something that
- Should be addressed internally by the s3cmd
- you could perhaps determine the bucket region and
explicitly use that region
- or use any other method the team determines will
make the command robust
- Should be posted in a FAQ (with clear instructions for
what work around(s) help) in the interim
Thanks -
Stephen
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
|
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
