Ingo, It looks like I'm now seeing "Caused by: java.lang.NullPointerException: You must specify a value for roleArn and roleSessionName". I assume I would also need to specify that through the configuration file. Could you suggest the key for this configuration? Thanks.
Thomas On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang <w...@datability.io> wrote: > Thanks Ingo. Adding the following setting worked. > > fs.s3a.aws.credentials.provider: > com.amazonaws.auth.WebIdentityTokenCredentialsProvider > > Thomas > > On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk <i...@ververica.com> wrote: > >> Hi Thomas, >> >> I think you might be looking for this: >> https://github.com/apache/flink/pull/16717 >> >> >> Best >> Ingo >> >> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote: >> >>> Hi, >>> >>> I'm using the official docker image: >>> apache/flink:1.12.1-scala_2.11-java11 >>> >>> I'm trying to run a Flink job on an EKS cluster. The job is running >>> under a k8s service account that is tied to an IAM role. If I'm not using >>> s3 as RocksDB checkpoint backend, everything works just fine. However, when >>> I enabled s3 as RocksDB checkpoint backend, I got permission denied. >>> >>> The IAM role tied to the service account has the appropriate permissions >>> to s3. However the underlying role tied to the EKS node doesn't. After >>> debugging with AWS support, it looks like the request to s3 was made under >>> the EKS node role, not the role tied to the service account. Thus the >>> permission denial. >>> >>> With the same Flink application, I'm also making requests to AWS Secrets >>> Manager to get some sensitive information and those requests were made >>> explicitly with AWS Java SDK 2.x bundled in the same application Jar file. >>> Those requests were made correctly with the IAM role tied to the service >>> account. >>> >>> Based on the info above, I suspect Flink may be using an older version >>> of the AWS SDK that doesn't support assuming an IAM role via an IODC web >>> identity token file. Please see AWS doc here: >>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html >>> >>> Could someone help me confirm this bug and maybe have it fixed some >>> time? Thanks. >>> >>> Thomas >>> >>
