Hello:
I am running s3cmd version 1.5.0-alpha3, and I have been
attempting to upload a file that is approximately 155GB in my
test. Unfortunately, this test with a large file has failed
in all versions of s3cmd that I have tried (Internal 500 error).
Here is a snippet of the debug log of the problem:
INFO: Sending file
'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg', please wait...
DEBUG: get_hostname(rsync1-rsnapshots.booyah.com):
rsync1-rsnapshots.booyah.com.s3.amazonaws.com
DEBUG: ConnMan.get(): re-using connection:
https://rsync1-rsnapshots.booyah.com.s3.amazonaws.com#750
DEBUG: format_uri():
/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1658&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
DEBUG: ConnMan.put(): connection put back to pool (
https://rsync1-rsnapshots.booyah.com.s3.amazonaws.com#751)
DEBUG: Response: {'status': 200, 'headers': {'content-length': '0',
'x-amz-id-2':
'/uetFULiHUfpEucJr72ZSdnOFC9X9yI6E/m5c8xnOIYKJt0G1TDaC62p6FqXFq6K',
'server': 'AmazonS3', 'x-amz-request-id': '671AE98C3CE1DB02', 'etag':
'"faef045d6d44d4bfd2b5c659d4a56751"', 'date': 'Mon, 24 Jun 2013 22:39:29
GMT'}, 'reason': 'OK', 'data': '', 'size': 15728640}
DEBUG: MD5 sums: computed=faef045d6d44d4bfd2b5c659d4a56751,
received="faef045d6d44d4bfd2b5c659d4a56751"
DEBUG: Unicodising
'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg' using UTF-8
DEBUG: Uploading part 1659 of
'orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--'
(15728640 bytes)
DEBUG: String 'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg'
encoded to 'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg'
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 24 Jun 2013 22:39:39
+0000\n/
rsync1-rsnapshots.booyah.com/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1659&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
'
DEBUG: CreateRequest:
resource[uri]=/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1659&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 24 Jun 2013 22:39:39
+0000\n/
rsync1-rsnapshots.booyah.com/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1659&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
'
INFO: Sending file
'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg', please wait...
DEBUG: get_hostname(rsync1-rsnapshots.booyah.com):
rsync1-rsnapshots.booyah.com.s3.amazonaws.com
DEBUG: ConnMan.get(): re-using connection:
https://rsync1-rsnapshots.booyah.com.s3.amazonaws.com#751
DEBUG: format_uri():
/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1659&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
DEBUG: ConnMan.put(): connection put back to pool (
https://rsync1-rsnapshots.booyah.com.s3.amazonaws.com#752)
DEBUG: Response: {'status': 200, 'headers': {'content-length': '0',
'x-amz-id-2':
'sawcMYGl3oYDL0ci8hx7tVoMRucHMMTfbeEKTzdO4n/OpzNOLneT9ysd6IS+mgWi',
'server': 'AmazonS3', 'x-amz-request-id': 'FB6855554994D714', 'etag':
'"5cf7b636c8d60a657a8b1f0968f08a7e"', 'date': 'Mon, 24 Jun 2013 22:39:41
GMT'}, 'reason': 'OK', 'data': '', 'size': 15728640}
DEBUG: MD5 sums: computed=5cf7b636c8d60a657a8b1f0968f08a7e,
received="5cf7b636c8d60a657a8b1f0968f08a7e"
DEBUG: Unicodising
'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg' using UTF-8
DEBUG: Uploading part 1660 of
'orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--'
(15728640 bytes)
DEBUG: String 'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg'
encoded to 'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg'
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 24 Jun 2013 22:39:49
+0000\n/
rsync1-rsnapshots.booyah.com/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1660&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
'
DEBUG: CreateRequest:
resource[uri]=/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1660&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 24 Jun 2013 22:39:49
+0000\n/
rsync1-rsnapshots.booyah.com/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1660&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
'
INFO: Sending file
'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg', please wait...
DEBUG: get_hostname(rsync1-rsnapshots.booyah.com):
rsync1-rsnapshots.booyah.com.s3.amazonaws.com
DEBUG: ConnMan.get(): re-using connection:
https://rsync1-rsnapshots.booyah.com.s3.amazonaws.com#752
DEBUG: format_uri():
/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1660&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
DEBUG: ConnMan.put(): connection put back to pool (
https://rsync1-rsnapshots.booyah.com.s3.amazonaws.com#753)
DEBUG: Response: {'status': 500, 'headers': {'x-amz-id-2':
'wK/MCtWIIoXgz8F4LNEooaGwihorCMnKdzZk1HrcbVTEt4fsodX/Lx6aG7vJHeSf',
'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection':
'close', 'x-amz-request-id': 'DBC7CC511CBFD997', 'date': 'Mon, 24 Jun 2013
22:41:20 GMT', 'content-type': 'application/xml'}, 'reason': 'Internal
Server Error', 'data': '<?xml version="1.0"
encoding="UTF-8"?>\n<Error><Code>InternalError</Code><Message>We
encountered an internal error. Please try
again.</Message><RequestId>DBC7CC511CBFD997</RequestId><HostId>wK/MCtWIIoXgz8F4LNEooaGwihorCMnKdzZk1HrcbVTEt4fsodX/Lx6aG7vJHeSf</HostId></Error>',
'size': 15728640}
DEBUG: S3Error: 500 (Internal Server Error)
DEBUG: HttpHeader: x-amz-id-2:
wK/MCtWIIoXgz8F4LNEooaGwihorCMnKdzZk1HrcbVTEt4fsodX/Lx6aG7vJHeSf
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: connection: close
DEBUG: HttpHeader: x-amz-request-id: DBC7CC511CBFD997
DEBUG: HttpHeader: etag:
DEBUG: HttpHeader: date: Mon, 24 Jun 2013 22:41:20 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'InternalError'
DEBUG: ErrorXML: Message: 'We encountered an internal error. Please try
again.'
DEBUG: ErrorXML: RequestId: 'DBC7CC511CBFD997'
DEBUG: ErrorXML: HostId:
'wK/MCtWIIoXgz8F4LNEooaGwihorCMnKdzZk1HrcbVTEt4fsodX/Lx6aG7vJHeSf'
DEBUG: DeUnicodising u'500 (InternalError): We encountered an internal
error. Please try again.' using UTF-8
WARNING: Upload failed:
/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1660&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
(500 (InternalError): We encountered an internal error. Please try again.)
WARNING: Waiting 3 sec...
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 24 Jun 2013 22:41:24
+0000\n/
rsync1-rsnapshots.booyah.com/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1660&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
'
INFO: Sending file
'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg', please wait...
DEBUG: get_hostname(rsync1-rsnapshots.booyah.com):
rsync1-rsnapshots.booyah.com.s3.amazonaws.com
DEBUG: ConnMan.get(): re-using connection:
https://rsync1-rsnapshots.booyah.com.s3.amazonaws.com#753
DEBUG: format_uri():
/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1660&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
DEBUG: ConnMan.put(): connection put back to pool (
https://rsync1-rsnapshots.booyah.com.s3.amazonaws.com#754)
DEBUG: Response: {'status': 200, 'headers': {'content-length': '0',
'x-amz-id-2':
'eYylaZmGaDVNpYBWYHXaEdvZu7IvRr9l2JusPkromdpr5yF5HJEUzWB8kaFUr8L8',
'server': 'AmazonS3', 'x-amz-request-id': 'DBDFD85D1B5D2405', 'etag':
'"66c4e56b0eab08c8a87d6f061acbcc1f"', 'date': 'Mon, 24 Jun 2013 22:41:29
GMT'}, 'reason': 'OK', 'data': '', 'size': 15728640}
DEBUG: MD5 sums: computed=66c4e56b0eab08c8a87d6f061acbcc1f,
received="66c4e56b0eab08c8a87d6f061acbcc1f"
DEBUG: Unicodising
'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg' using UTF-8
DEBUG: Uploading part 1661 of
'orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--'
(15728640 bytes)
DEBUG: String 'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg'
encoded to 'jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg'
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 24 Jun 2013 22:41:33
+0000\n/
rsync1-rsnapshots.booyah.com/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1661&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
'
DEBUG: CreateRequest:
resource[uri]=/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1661&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 24 Jun 2013 22:41:33
+0000\n/
rsync1-rsnapshots.booyah.com/jira1.booyahcorp.com-weekly-20130511-001357.tar.gz.gpg?partNumber=1661&uploadId=orrprJnrZVXso1zJZrVSdCXmIdA4y2wnCeYCF2x9fGnczhYiTYzjDSsyOzFjf8fQCnV46Q8GhgBBaLy8UYwhEQ--
'
Unfortunately, it will never complete a sucessful and clean upload.
Should I do my own manual multipart file upload instead? What
additional information do you need to help diagnosis this
problem?
Thanks.
------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:
Build for Windows Store.
http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
