[
https://issues.apache.org/jira/browse/FLINK-31095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689931#comment-17689931
]
Sylvia Lin edited comment on FLINK-31095 at 2/16/23 7:19 PM:
-------------------------------------------------------------
Thank you both for the reply!
I've enabled the *flink-s3-fs-hadoop* plugin, after working with [~sap1ens]
offline, we changed the S3 path prefix from `s3` to `s3a` and above error got
fixed.
But with above *UnsupportedOperationException* fixed, we have a new permission
error for S3 access denied:
{code:java}
Caused by: java.nio.file.AccessDeniedException:
java.nio.file.AccessDeniedException: translated/***: initiate MultiPartUpload
on translated/***: com.amazonaws.services.s3.model.AmazonS3Exception: Access
Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request
ID: ***; S3 Extended Request ID: ***; Proxy: null), S3 Extended Request ID:
***=:AccessDenied
{code}
And we pull the S3 request log it shows as below, the request IAM role is the
node group IAM role, but not the IAM role we bind to the service account.
{code:java}
User Agent: Hadoop 3.3.2, aws-sdk-java/1.11.951
Linux/5.4.209-116.367.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/25.362-b09
java/1.8.0_362 vendor/Temurin
Requester: arn:aws:iam::***:role/second_group-eks-node-group-***
Response Code: 403
Error Code: AccessDenied {code}
And confirmed with [~sap1ens] on slack, he needs to rely on the node group iam
role, which is not the way how IRSA suppose to work.
was (Author: JIRAUSER292782):
Thank you both for the reply!
I've enabled the *flink-s3-fs-hadoop* plugin, after working with [~sap1ens]
offline, we changed the S3 path prefix from `s3` to `s3a` and above error got
fixed.
But with above *UnsupportedOperationException* fixed, we have a new permission
error for S3 access denied:
{code:java}
Caused by: java.nio.file.AccessDeniedException:
java.nio.file.AccessDeniedException: translated/***: initiate MultiPartUpload
on translated/***: com.amazonaws.services.s3.model.AmazonS3Exception: Access
Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request
ID: ***; S3 Extended Request ID: ***; Proxy: null), S3 Extended Request ID:
***=:AccessDenied
{code}
And we pull the S3 request log it shows as below, the request IAM role is the
node group IAM role, but not the IAM role we bind to the service account.
{code:java}
User Agent: Hadoop 3.3.2, aws-sdk-java/1.11.951
Linux/5.4.209-116.367.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/25.362-b09
java/1.8.0_362 vendor/Temurin
Requester: arn:aws:iam::***:role/second_group-eks-node-group-***
Response Code: 403
Error Code: AccessDenied {code}
> FileSink doesn't work with s3a on EKS
> -------------------------------------
>
> Key: FLINK-31095
> URL: https://issues.apache.org/jira/browse/FLINK-31095
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.16.1
> Reporter: Sylvia Lin
> Priority: Major
>
> FileSink gives below exception on AWS EKS cluster:
> {code:java}
> Caused by: java.lang.UnsupportedOperationException: This s3 file system
> implementation does not support recoverable writers.
> at
> org.apache.flink.fs.s3.common.FlinkS3FileSystem.createRecoverableWriter(FlinkS3FileSystem.java:136)
> ~[?:?]
> at
> org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.createRecoverableWriter(PluginFileSystemFactory.java:134)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.connector.file.sink.FileSink$RowFormatBuilder.createBucketWriter(FileSink.java:475)
> ~[flink-connector-files-1.16.1.jar:1.16.1]
> at
> org.apache.flink.connector.file.sink.FileSink$RowFormatBuilder.getCommittableSerializer(FileSink.java:466)
> ~[flink-connector-files-1.16.1.jar:1.16.1]
> at
> org.apache.flink.connector.file.sink.FileSink.getCommittableSerializer(FileSink.java:175)
> ~[flink-connector-files-1.16.1.jar:1.16.1]{code}
> [https://github.com/apache/flink/blob/278dc7b793303d228f7816585054629708983af6/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#LL136C16-L136C16]
> And this may be related to
> https://issues.apache.org/jira/browse/FLINK-23487?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
