Hi Matthias, Looks like the service account token volume projection was not working fine with the EKS version I was running. Upgraded the version and with the same configs now the s3 checkpointing is working fine. So, in short, on AWS use EKS v1.20+ for IAM Pod Identity Webhook.
Thanks, Hemant On Mon, Nov 22, 2021 at 7:26 PM Matthias Pohl <matth...@ververica.com> wrote: > Hi bat man, > this feature seems to be tied to a certain AWS SDK version [1] which you > already considered. But I checked the version used in Flink 1.13.1 for the > s3 filesystem. It seems like the version that's used (1.11.788) is good > enough to provide this feature (which was added in 1.11.704): > ``` > $ git checkout release-1.13.1 > $ cd flink-filesystems/flink-s3-fs-base; mvn dependency:tree | grep > com.amazonaws:aws-java-sdk-s3 > [INFO] +- com.amazonaws:aws-java-sdk-s3:jar:1.11.788:compile > ``` > > Matthias > > [1] > https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html > > On Mon, Nov 22, 2021 at 8:04 AM bat man <tintin0...@gmail.com> wrote: > >> Hi, >> >> I am using flink 1.13.1 to use checkpointing(RocksDB) on s3 with native >> kubernetes. >> Passing in this parameter to job - >> >> >> *-Dfs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider* >> I am getting this error in job-manager logs - >> >> *Caused by: com.amazonaws.AmazonClientException: No AWS Credentials >> provided by WebIdentityTokenCredentialsProvider : >> com.amazonaws.SdkClientException: Unable to locate specified web identity >> token file: /var/run/secrets/eks.amazonaws.com/serviceaccount/token >> <http://eks.amazonaws.com/serviceaccount/token> at >> org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:139) >> ~[?:?]* >> >> Describing the pod shows that that volume is mounted to the jobmanager >> pod. >> Is there anything specific that needs to be done as on the same EKS >> cluster for testing I ran a sample pod with aws cli image and it's able to >> do *ls* on the s3 buckets. >> Is this related to aws sdk used in Flink 1.13.1, shall I try with recent >> flink versions. >> >> Any help would be appreciated. >> >> Thanks. >> >
