Hi team, I'm trying to read and write from and to S3 using a custom AWS Credential Provider using Flink v1.9.2 on YARN.
I followed the instructions to create a plugins directory in our Flink distribution location and copy the FS implementation (I'm using s3-fs-hadoop) package into it. I have also placed the package that contains our custom CredentialsProvider implementation in that same directory as well. $ ls /flink-1.9.2/plugins/s3-fs-hadoop/ total 20664 14469 Jun 17 10:57 aws-hadoop-utils-0.0.9.jar <-- contains our custom CredentialsProvider class 21141329 Jul 28 15:43 flink-s3-fs-hadoop-1.9.2.jar I've placed this directory in the java classpath when running the Flink application. I have added the 'fs.s3a.assumed.role.credentials.provider' and 'fs.s3a.assumed.role.arn' to our flink-conf.yaml as well. When trying to run a basic app that reads a file, I get the following exception: Caused by: java.io.IOException: Class class com.gs.ep.da.lake.aws.CustomAwsCredentialProvider does not implement AWSCredentialsProvider at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:400) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:367) at org.apache.hadoop.fs.s3a.S3ClientFactory$DefaultS3ClientFactory.createS3Client(S3ClientFactory.java:73) Have I missed a step here? Do I need to make the packages also available in my YARN classpath as well? I saw some discussion that suggest that there were some related problems around this that were resolved in v1.10 [1][2][3]. [1] https://issues.apache.org/jira/browse/FLINK-14574 [2] https://issues.apache.org/jira/browse/FLINK-13044 [3] https://issues.apache.org/jira/browse/FLINK-11956 Best, Andreas ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>