Hey, sorry for the late response. Can you provide the full exceptions(s) including stacktrace you are seeing?
On Mon, Apr 20, 2020 at 3:39 PM orionemail <orionem...@protonmail.com> wrote: > Hi, > > New to both AWS and Flink but currently have a need to write incoming data > into a S3 bucket managed via AWS Tempory credentials. > > I am unable to get this to work, but I am not entirely sure on the steps > needed. I can write to S3 buckets that are not 'remote' and managed by STS > tempory credentials fine. > > I am using flink 1.9.1, as this will when deployed live in EMR. > > My flink-conf.yml contains the following entries: > > fs.s3a.bucket.sky-rdk-telemetry.aws.credentials.provider: > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider > fs.s3a.bucket.sky-rdk-telemetry.assumed.role.credentials.provider: > org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider > fs.s3a.bucket.sky-rdk-telemetry.access-key: xxxxx > fs.s3a.bucket.sky-rdk-telemetry.secret-key: xxxx > fs.s3a.bucket.sky-rdk-telemetry.assumed.role.arn: xxxx > fs.s3a.bucket.sky-rdk-telemetry.assumed.role.session.name: xxxx > > And my POM contains > > <dependencyManagement> > <dependencies> > <dependency> > <groupId>com.amazonaws</groupId> > <artifactId>aws-java-sdk-bom</artifactId> > <version>1.11.700</version> > <type>pom</type> > <scope>import</scope> > </dependency> > </dependencies> > </dependencyManagement> > > > <dependency> > <groupId>com.amazonaws</groupId> > <artifactId>aws-java-sdk-sts</artifactId> > <version>1.11.700</version> > </dependency> > > > I have put the jar flink-s3-fs-hadoop-1.9.1.jar into the plugins directory. > > Running my test Jar I am getting exceptions related to Class not found > for > org/apache/flink/fs/s3base/shaded/com/amazonaws/services/securitytoken/model/AWSSecurityTokenServiceException > > and poking around I see this is shaded into a package in Kinesis. I have > added some rules to maven shade to rewrite the package as needed but this > still doesn't help. > > Am I heading in the correct direction? Searching has not turned up much > information that I have been able to make use of. > > Thanks for your time, > > J > > > > > > Sent with ProtonMail <https://protonmail.com> Secure Email. > >
