virajjasani commented on PR #6479:
URL: https://github.com/apache/hadoop/pull/6479#issuecomment-1910667743
> > On the other hand, on any existing bucket from the other region (e.g.
us-west-2)
>
> do you mean for here you set the region to us-west-2 in
fs.s3a.endpoint.region for this?
Not really, i meant that with the above combination of endpoint
`s3.amazonaws.com` (and no region specified for `fs.s3a.endpoint.region`) and
this patch setting `us-east-2` with cross-region access enabled client
internally, is able to perform all operations on existing bucket from other
region. Only when bucket is not present, it gives 400 instead of 404. If it is
present, headBucket goes well. Similar case for object operations: if object is
not present and we do `fs#exists`, it fails with 400 instead of 404. If it
exists, headObject goes well.
Hence, tests that perform file system CRUD operations on real existing
bucket from other region, are passing without any issues **with this patch**
and settings:
1. `fs.s3a.endpoint` = `s3.amazonaws.com`
2. Nothing set for `fs.s3a.endpoint.region`, which would internally result
into `us-east-2` with cross region access (as per this patch).
**Without this patch**, file system CRUD operations fail on real existing
bucket from other region, which is expected.
e.g.
```
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on
s3a://${bucket}/user/${user}/${dir-path}:
software.amazon.awssdk.services.s3.model.S3Exception: The authorization header
is malformed; the region 'us-east-2' is wrong; expecting 'us-west-2' (Service:
S3, Status Code: 400, Request ID: G85CNFC579T4MJ76, Extended Request ID:
xrYGGqXdYtr72cYyFN3v4yemDxBCYkdt8mYd8cGItNhdx1EmZMLxMhwJTwzmWZT6ershid/WT4w=):AuthorizationHeaderMalformed:
The authorization header is malformed; the region 'us-east-2' is wrong;
expecting 'us-west-2' (Service: S3, Status Code: 400, Request ID:
G85CNFC579T4MJ76, Extended Request ID:
xrYGGqXdYtr72cYyFN3v4yemDxBCYkdt8mYd8cGItNhdx1EmZMLxMhwJTwzmWZT6ershid/WT4w=)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:259)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:154)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4075)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3934)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$MkdirOperationCallbacksImpl.probePathStatus(S3AFileSystem.java:3806)
at
org.apache.hadoop.fs.s3a.impl.MkdirOperation.probePathStatusOrNull(MkdirOperation.java:173)
at
org.apache.hadoop.fs.s3a.impl.MkdirOperation.getPathStatusExpectingDir(MkdirOperation.java:194)
at
org.apache.hadoop.fs.s3a.impl.MkdirOperation.execute(MkdirOperation.java:108)
at
org.apache.hadoop.fs.s3a.impl.MkdirOperation.execute(MkdirOperation.java:57)
at
org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation.apply(ExecutingStoreOperation.java:76)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2719)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2738)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:3778)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2494)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
