Hiya, This sounds like it may be similar to the issue I had when running on ECS. Take a look at my ticket for how I got around this, and see if it's any help: https://issues.apache.org/jira/browse/FLINK-8439
Dyana On 2018/03/28 02:15:06, "Bajaj, Abhinav" <abhinav.ba...@here.com> wrote: > Hi, > > I am trying to use Flink 1.2.1 with RockDB as statebackend and S3 for > checkpoints. > I am using Flink 1.2.1 docker images and running them in Kubernetes cluster. > > I have followed the steps documented in the Flink documentation - > https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#s3-simple-storage-service > > I am using AWS IAM roles to setup access for S3. > The role has actions "s3:GetObject","s3:ListBucket", "s3:PutObject", > "s3:DeleteObject" on the bucket. > > When I run a job, the jobmanager logs below exception â > > java.io.IOException: The given file URI (s3://$MY_TEST_BUCKET/checkpoints) > points to the HDFS NameNode at $MY_TEST_BUCKET, but the File System could not > be initialized with that address: Unable to load AWS credentials from any > provider in the chain > at > org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:334) > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:265) > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:304) > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:105) > at > org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:172) > at > org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:219) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:803) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:220) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:655) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:643) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:246) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.amazonaws.AmazonClientException: Unable to load AWS > credentials from any provider in the chain > at > com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117) > at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521) > at > com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) > at > com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) > at > org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:320) > ... 13 more > > I checked if the jobmanager pod in the K8s cluster has the correct IAM role > applied. > âcurl http://169.254.169.254/latest/meta-data/iam/security-credentials/â > returned the correct role. > > After this, I installed aws cli on the jobmanager pod and could > download/upload to $MY_TEST_BUCKET. > This confirmed that the jobmanager pod has the correct IAM role associated > with it. > > So, I am not sure why the AWS library in Flink is not able to load the > credentials. > Any thoughts or suggestions to fix or troubleshoot? > > Appreciate the help. > > Regards, > Abhinav Bajaj > > > [cid:image001.png@01D3C5FF.E9E41E50] > > Abhinav Bajaj > Lead Engineer > Open Location Platform > Mobile: +1 708 329 9516 > > HERE Seattle > 701 Pike Street, suite 2000 > Seattle, WA 98101 USA > 47° 36' 41" N 122° 19' 57" W > > [cid:image002.png@01D3C5FF.E9E41E50]<http://360.here.com/> > [cid:image003.png@01D3C5FF.E9E41E50] <https://www.twitter.com/here> > [cid:image004.png@01D3C5FF.E9E41E50] <https://www.facebook.com/here> > [cid:image005.png@01D3C5FF.E9E41E50] > <https://www.linkedin.com/company/heremaps> > [cid:image006.png@01D3C5FF.E9E41E50] <https://www.instagram.com/here/> > > >
