Hi Arvid, no. We are leveraging it as part of our application code, but not Kinesis – after finding and excluding duplicates of this package in our classpath, we are able to submit a job. Thanks.
// ah From: Arvid Heise <ar...@apache.org> Sent: Friday, July 30, 2021 1:34 PM To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com> Cc: Ingo Bürk <i...@ververica.com>; user <user@flink.apache.org> Subject: Re: Unable to use custom AWS credentials provider - 1.9.2 Well usually the plugins should be properly isolated but Flink 1.9 is quite old so there is a chance the plugin classloader was not fully isolated. But I also have a hard time concluding anything with the small stacktrace. Do you need aws-java-sdk-core because of Kinesis? On Fri, Jul 30, 2021 at 6:46 PM Hailu, Andreas [Engineering] <andreas.ha...@gs.com<mailto:andreas.ha...@gs.com>> wrote: Hi Arvid, Yes, we do have AWSCredentialsProvider in our user JAR. It’s coming from aws-java-sdk-core. Must we exclude that, then? // ah From: Arvid Heise <ar...@apache.org<mailto:ar...@apache.org>> Sent: Friday, July 30, 2021 11:26 AM To: Ingo Bürk <i...@ververica.com<mailto:i...@ververica.com>> Cc: user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Unable to use custom AWS credentials provider - 1.9.2 Can you double-check if you have a AWSCredentialsProvider in your user jar or in your flink/lib/ ? Same for S3AUtils? On Fri, Jul 30, 2021 at 9:50 AM Ingo Bürk <i...@ververica.com<mailto:i...@ververica.com>> wrote: Hi Andreas, Such an exception can occur if the class in question (your provider) and the one being checked (AWSCredentialsProvider) were loaded from different class loaders. Any chance you can try once with 1.10+ to see if it would work? It does look like a Flink issue to me, but I'm not sure this can be worked around in 1.9. [Initially sent to Andreas directly by accident] Best Ingo On 29.07.21 17:37, Hailu, Andreas [Engineering] wrote: > Hi team, I’m trying to read and write from and to S3 using a custom AWS > Credential Provider using Flink v1.9.2 on YARN. > > > > I followed the instructions to create a plugins directory in our Flink > distribution location and copy the FS implementation (I’m using > s3-fs-hadoop) package into it. I have also placed the package that > contains our custom CredentialsProvider implementation in that same > directory as well. > > > > $ ls /flink-1.9.2/plugins/s3-fs-hadoop/ > > total 20664 > > 14469 Jun 17 10:57 aws-hadoop-utils-0.0.9.jar ßcontains our custom > CredentialsProvider class > > 21141329 Jul 28 15:43 flink-s3-fs-hadoop-1.9.2.jar > > > > I’ve placed this directory in the java classpath when running the Flink > application. I have added the ‘fs.s3a.assumed.role.credentials.provider’ > and ‘fs.s3a.assumed.role.arn’ to our flink-conf.yaml as well. When > trying to run a basic app that reads a file, I get the following exception: > > > > Caused by: java.io.IOException: Class class > com.gs.ep.da.lake.aws.CustomAwsCredentialProvider does not implement > AWSCredentialsProvider > > at > org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:400) > > at > org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:367) > > at > org.apache.hadoop.fs.s3a.S3ClientFactory$DefaultS3ClientFactory.createS3Client(S3ClientFactory.java:73) > > > > Have I missed a step here? Do I need to make the packages also available > in my YARN classpath as well? I saw some discussion that suggest that > there were some related problems around this that were resolved in v1.10 > [1][2][3]. > > > > [1] > https://issues.apache.org/jira/browse/FLINK-14574<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D14574&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=gpQJJW86RnCu0WMZ7mBrjMeu3Ou6dfc0-wwZHD1yL98&s=llyqmixMoNm0FF71rXQP1wWDK0_QIx-wv3SwxSJ4A6g&e=> > <https://issues.apache.org/jira/browse/FLINK-14574<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D14574&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=gpQJJW86RnCu0WMZ7mBrjMeu3Ou6dfc0-wwZHD1yL98&s=llyqmixMoNm0FF71rXQP1wWDK0_QIx-wv3SwxSJ4A6g&e=>> > > [2] > https://issues.apache.org/jira/browse/FLINK-13044<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D13044&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=gpQJJW86RnCu0WMZ7mBrjMeu3Ou6dfc0-wwZHD1yL98&s=_J5mLFPB16qgGzCe0tgUbXb05GxmOyYym1deVQKY1bI&e=> > <https://issues.apache.org/jira/browse/FLINK-13044<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D13044&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=gpQJJW86RnCu0WMZ7mBrjMeu3Ou6dfc0-wwZHD1yL98&s=_J5mLFPB16qgGzCe0tgUbXb05GxmOyYym1deVQKY1bI&e=>> > > [3] > https://issues.apache.org/jira/browse/FLINK-11956<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D11956&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=gpQJJW86RnCu0WMZ7mBrjMeu3Ou6dfc0-wwZHD1yL98&s=z31ZYJqnf00rLBW3HSiVrkQ8R6FYdw5F9QKqmWJ2CH4&e=> > <https://issues.apache.org/jira/browse/FLINK-11956<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D11956&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=gpQJJW86RnCu0WMZ7mBrjMeu3Ou6dfc0-wwZHD1yL98&s=z31ZYJqnf00rLBW3HSiVrkQ8R6FYdw5F9QKqmWJ2CH4&e=>> > > > > Best, > > Andreas > > > > > ------------------------------------------------------------------------ > > Your Personal Data: We may collect and process information about you > that may be subject to data protection laws. For more information about > how we use and disclose your personal data, how we protect your > information, our legal basis to use your information, your rights and > who you can contact, please refer to: > www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> > <http://www.gs.com/privacy-notices> ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
