[
https://issues.apache.org/jira/browse/HADOOP-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886191#comment-17886191
]
Steve Loughran commented on HADOOP-19295:
-----------------------------------------
Another iteration this morning. Here there were a lot of errors reported and
retried, and upload time went up to 10 minutes because of them
Reading those stats, the first MPU was aborted, but a second attempt worked. i
wan't expecting a second attempt
{code}
time bin/hadoop fs -put -f
/Users/stevel/Projects/Misc/client-validator/downloads/hadoop-3.4.1-RC2/hadoop-3.4.1.tar.gz
s3a
2024-10-01 11:50:18,821 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 0:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:50:18,897 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 0:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:50:18,978 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 0:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:50:19,144 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 0:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:50:19,852 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0004-7915333819452152114.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=2,
currentStream=null}
2024-10-01 11:50:19,868 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0003-2194027550968106888.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=2,
currentStream=null}
2024-10-01 11:50:20,120 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0002-6149155212262724579.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=2,
currentStream=null}
2024-10-01 11:50:20,220 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0001-3207449200810636473.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=2,
currentStream=null}
2024-10-01 11:51:19,871 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 1:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:51:19,884 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 1:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:51:20,144 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 1:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:51:20,241 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 1:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:51:21,623 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0003-2194027550968106888.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=3,
currentStream=null}
2024-10-01 11:51:22,197 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0004-7915333819452152114.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=3,
currentStream=null}
2024-10-01 11:51:22,532 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0001-3207449200810636473.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=3,
currentStream=null}
2024-10-01 11:51:22,839 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0002-6149155212262724579.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=3,
currentStream=null}
2024-10-01 11:52:21,647 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 2:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:52:22,211 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 2:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:52:22,535 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 2:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:52:22,838 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 2:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:52:23,933 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0003-2194027550968106888.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=4,
currentStream=null}
2024-10-01 11:52:26,534 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0004-7915333819452152114.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=4,
currentStream=null}
2024-10-01 11:52:27,342 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0001-3207449200810636473.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=4,
currentStream=null}
2024-10-01 11:52:28,741 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0002-6149155212262724579.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=4,
currentStream=null}
2024-10-01 11:53:23,950 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 3:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:53:26,542 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 3:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:53:27,423 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 3:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:53:28,745 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 3:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:53:31,427 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0003-2194027550968106888.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=5,
currentStream=null}
2024-10-01 11:53:34,789 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0004-7915333819452152114.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=5,
currentStream=null}
2024-10-01 11:53:35,771 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0002-6149155212262724579.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=5,
currentStream=null}
2024-10-01 11:53:39,082 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0001-3207449200810636473.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=5,
currentStream=null}
2024-10-01 11:54:31,450 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 4:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:54:34,812 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 4:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:54:35,786 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 4:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:54:39,088 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 4:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:54:42,947 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0004-7915333819452152114.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=6,
currentStream=null}
2024-10-01 11:54:47,979 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0001-3207449200810636473.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=6,
currentStream=null}
2024-10-01 11:54:52,977 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0002-6149155212262724579.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=6,
currentStream=null}
2024-10-01 11:54:54,015 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0003-2194027550968106888.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=6,
currentStream=null}
2024-10-01 11:55:42,982 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 5:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #4 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:55:48,020 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 5:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #1 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:55:53,023 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 5:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:55:54,026 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 5:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:56:18,742 [s3a-transfer-stevel-london-bounded-pool1-t1] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0001-3207449200810636473.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=7,
currentStream=null}
2024-10-01 11:56:19,270 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0003-2194027550968106888.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=7,
currentStream=null}
2024-10-01 11:56:20,758 [s3a-transfer-stevel-london-bounded-pool1-t4] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0004-7915333819452152114.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=7,
currentStream=null}
2024-10-01 11:56:21,214 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0002-6149155212262724579.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=7,
currentStream=null}
2024-10-01 11:57:19,279 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 6:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #3 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:57:21,224 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 6:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #2 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:58:17,812 [s3a-transfer-stevel-london-bounded-pool1-t3] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0003-2194027550968106888.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=8,
currentStream=null}
2024-10-01 11:58:30,488 [s3a-transfer-stevel-london-bounded-pool1-t2] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0002-6149155212262724579.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=8,
currentStream=null}
2024-10-01 11:59:00,205 [s3a-transfer-stevel-london-bounded-pool1-t7] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #7 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 0:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #7 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:59:00,352 [s3a-transfer-stevel-london-bounded-pool1-t8] INFO
s3a.WriteOperationHelper (WriteOperationHelper.java:operationRetried(184)) -
upload part #8 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_: Retried 0:
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: upload part #8 upload ID
RN5oIKsspzQQjIxiD5DbBZIqA9pTmlmXRjPXTJf4X2MPCwDZ8cWGb1whnRW4CkuRTYgDYvRv9IlBHGdZn8z6JmbQ0sDx4hiWqf06eH9PpUy6p5nFC2K8ytlMDIioH1gC
on hadoop-3.4.1.tar.gz._COPYING_:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 60000 millis
2024-10-01 11:59:00,858 [s3a-transfer-stevel-london-bounded-pool1-t7] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0007-8015574127550198023.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=2,
currentStream=null}
2024-10-01 11:59:01,413 [s3a-transfer-stevel-london-bounded-pool1-t8] INFO
impl.UploadContentProviders (UploadContentProviders.java:newStream(278)) -
Stream created more than once:
FileWithOffsetContentProvider{file=/tmp/hadoop-stevel/s3a/s3ablock-0008-2688580947765969860.tmp,
offset=0} BaseContentProvider{size=67108864, streamCreationCount=2,
currentStream=null}
2024-10-01 11:59:17,822 [s3a-transfer-stevel-london-bounded-pool1-t3] WARN
s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:progressChanged(1295)) -
Transfer failure of block FileBlock{index=3,
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0003-2194027550968106888.tmp,
state=Upload, dataSize=67108864, limit=67108864}
2024-10-01 11:59:30,499 [s3a-transfer-stevel-london-bounded-pool1-t2] WARN
s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:progressChanged(1295)) -
Transfer failure of block FileBlock{index=2,
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0002-6149155212262724579.tmp,
state=Upload, dataSize=67108864, limit=67108864}
2024-10-01 11:59:30,506 [s3a-transfer-stevel-london-bounded-pool1-t9] WARN
s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:progressChanged(1295)) -
Transfer failure of block FileBlock{index=9,
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0009-2464982840061319105.tmp,
state=Closed, dataSize=67108864, limit=67108864}
2024-10-01 11:59:30,507 [s3a-transfer-stevel-london-bounded-pool1-t7] WARN
s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:progressChanged(1295)) -
Transfer failure of block FileBlock{index=7,
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0007-8015574127550198023.tmp,
state=Closed, dataSize=67108864, limit=67108864}
2024-10-01 11:59:30,662 [s3a-transfer-stevel-london-bounded-pool1-t8] WARN
s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:progressChanged(1295)) -
Transfer failure of block FileBlock{index=8,
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0008-2688580947765969860.tmp,
state=Closed, dataSize=67108864, limit=67108864}
put: Interrupted
2024-10-01 11:59:30,785 [shutdown-hook-0] INFO statistics.IOStatisticsLogging
(IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics:
counters=((action_executor_acquired=9)
(action_http_head_request=3)
(audit_request_execution=44)
(audit_span_creation=6)
(files_created=1)
(filesystem_close=1)
(filesystem_initialization=1)
(ignored_errors=28)
(multipart_upload_part_put=37)
(object_list_request=2)
(object_metadata_request=3)
(object_multipart_aborted=2)
(object_multipart_initiated=2)
(object_put_bytes=268435456)
(object_put_request_completed=4)
(op_create=1)
(op_get_file_status=3)
(op_get_file_status.failures=2)
(op_glob_status=1)
(store_client_creation=1)
(store_io_request=14)
(store_io_retry=28)
(stream_write_block_uploads=18)
(stream_write_bytes=671088640)
(stream_write_exceptions=10)
(stream_write_queue_duration=598630)
(stream_write_total_data=1207959552)
(stream_write_total_time=3045456));
gauges=();
minimums=((action_executor_acquired.min=0)
(action_http_head_request.min=38)
(filesystem_close.min=3)
(filesystem_initialization.min=515)
(object_list_request.min=45)
(object_multipart_aborted.min=185)
(object_multipart_initiated.min=125)
(op_create.min=12)
(op_get_file_status.failures.min=83)
(op_get_file_status.min=456)
(op_glob_status.min=479)
(store_client_creation.min=458));
maximums=((action_executor_acquired.max=478653)
(action_http_head_request.max=448)
(filesystem_close.max=3)
(filesystem_initialization.max=515)
(object_list_request.max=53)
(object_multipart_aborted.max=185)
(object_multipart_initiated.max=125)
(op_create.max=12)
(op_get_file_status.failures.max=109)
(op_get_file_status.max=456)
(op_glob_status.max=479)
(store_client_creation.max=458));
means=((action_executor_acquired.mean=(samples=18, sum=1197250,
mean=66513.8889))
(action_http_head_request.mean=(samples=3, sum=537, mean=179.0000))
(filesystem_close.mean=(samples=1, sum=3, mean=3.0000))
(filesystem_initialization.mean=(samples=1, sum=515, mean=515.0000))
(object_list_request.mean=(samples=2, sum=98, mean=49.0000))
(object_multipart_aborted.mean=(samples=2, sum=363, mean=181.5000))
(object_multipart_initiated.mean=(samples=2, sum=222, mean=111.0000))
(op_create.mean=(samples=1, sum=12, mean=12.0000))
(op_get_file_status.failures.mean=(samples=2, sum=192, mean=96.0000))
(op_get_file_status.mean=(samples=1, sum=456, mean=456.0000))
(op_glob_status.mean=(samples=1, sum=479, mean=479.0000))
(store_client_creation.mean=(samples=1, sum=458, mean=458.0000)));
________________________________________________________
Executed in 614.45 secs fish external
usr time 39.99 secs 0.09 millis 39.99 secs
sys time 6.77 secs 1.56 millis 6.77 secs
{code}
> S3A: fs.s3a.connection.request.timeout too low for large uploads over slow
> links
> --------------------------------------------------------------------------------
>
> Key: HADOOP-19295
> URL: https://issues.apache.org/jira/browse/HADOOP-19295
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0, 3.4.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> The value of {{fs.s3a.connection.request.timeout}} (default = 60s} is too low
> for large uploads over slow connections.
> I suspect something changed between the v1 and v2 SDK versions so that put
> was exempt from the normal timeouts, It is not and now surfaces in failures
> to upload 1+ GB files over slower network connections. Smailer (for example
> 128 MB) files work.
> The parallel queuing of writes in the S3ABlockOutputStream is helping create
> this problem as it queues multiple blocks at the same time, so per-block
> bandwidth becomes available/blocks ; four blocks cuts the capacity down by a
> quarter.
> The fix is straightforward: use a much bigger timeout. I'm going to propose
> 15 minutes. We need to strike a balance between upload time allocation and
> other requests timing out.
> I do worry about other consequences; we've found that timeout exception happy
> to hide the underlying causes of retry failures -so in fact this may be
> better for all but a server hanging after the HTTP request is initiated.
> too bad we can't alter the timeout for different requests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
