[
https://issues.apache.org/jira/browse/HADOOP-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukund Thakur resolved HADOOP-18353.
------------------------------------
Resolution: Information Provided
> HEAD OBJECT returns only 400 BAD REQUEST when token is expired
> --------------------------------------------------------------
>
> Key: HADOOP-18353
> URL: https://issues.apache.org/jira/browse/HADOOP-18353
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.3
> Reporter: Mukund Thakur
> Priority: Major
>
> I tried reproducing this today by changing this test
> [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java#L116]
> . Getting a session token for 15 mins and trying every one mins and finally
> it fails after 15 mins.
>
> Looks like the AWS SDK is not having the Expired Token error message as I
> could see the same in access logs but I see BadRequest on the SDK logs.
>
> *S3A Connector logs with SDK debug enabled.*
> 2022-07-13 15:44:15,318 [JUnit-testSTS] DEBUG s3a.AWSCredentialProviderList
> (AWSCredentialProviderList.java:getCredentials(184)) - Using credentials from
> TemporaryAWSCredentialsProvider
> 2022-07-13 15:44:15,319 [JUnit-testSTS] DEBUG amazonaws.request
> (AmazonHttpClient.java:executeOneRequest(1285)) - Sending Request: HEAD
> [https://mthakur-us-west-1.s3.us-west-1.amazonaws.com|https://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
> /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e Headers:
> (amz-sdk-invocation-id: 41e6e504-1c2b-2701-09bb-ae692dff2515, Content-Type:
> application/octet-stream, Referer:
> [https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318],
> User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
> Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
> vendor/Oracle_Corporation cfg/retry-mode/legacy, )
> 2022-07-13 15:44:15,623 [JUnit-testSTS] DEBUG amazonaws.request
> (AmazonHttpClient.java:handleErrorResponse(1846)) - \{*}Received error
> response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
> (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request
> ID{*}: WMGQ0KC4MHEMZTQC; S3 Extended Request ID:
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=;
> Proxy: null), S3 Extended Request ID:
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
> 2022-07-13 15:44:15,624 [JUnit-testSTS] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:close(3814)) - Filesystem s3a://mthakur-us-west-1 is
> closed
> *AWS access logs*
> 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
> mthakur-us-west-1 [13/Jul/2022:20:44:15 +0000] 67.79.115.98 -
> WMGQ0KC4MHEMZTQC REST.HEAD.OBJECT
> test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e "HEAD
> /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e HTTP/1.1" *400
> ExpiredToken* 556 - 5 -
> "[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318]";
> "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
> Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
> vendor/Oracle_Corporation cfg/retry-mode/legacy" -
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
> SigV4 ECDHE-RSA-AES128-SHA AuthHeader
> [mthakur-us-west-1.s3.us-west-1.amazonaws.com|http://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
> TLSv1.2 -
>
> I tested by running repeatedly ITestCustomSigner in S3A, and also just
> ListObjectsV2 on loop… I did just notice your test is failing with HEAD, and
> *I can reproduce* by running this after credential expiry.
>
> aws s3api head-object --bucket djonesoa-us-west-2 --region us-west-2 --key
> test-object –debug
>
> To summarise:
> * If I run ListObjectsV2, I get “400 ExpiredToken”{+}{+}{+}{+}
> * If I run HeadObject, I get “400 Bad Request”{+}{+}{+}{+}
> * If I run GetObject, I get “400 ExpiredToken”
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
