Hi, Thanks for the suggestions.
We are using Kube2iam for our Kubernetes cluster and it seems to be setup correctly to support IAM Roles. I also checked AWS documentation<https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html#troubleshoot_iam-ec2_no-keys> to troubleshoot the pods for access to the temporary security credentials. AWS CLI works as expected on the cluster pods. I also tested with a script that use ‘aws-java-sdk-1.7.4.jar’ to get credentials using the InstanceProfileCredentialsProvider. At this point, I think Flink is not using the InstanceProfileCredentialsProvider in my setup, probably some dependencies mismatch. I am using the same dependencies as documented by Flink<https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#flink-for-hadoop-27> and the core-site.xml. * S3AFileSystem: * hadoop-aws-2.7.2.jar * aws-java-sdk-1.7.4.jar * httpcore-4.2.5.jar * httpclient-4.2.5.jar Am I missing some dependencies here? Any suggestions on troubleshooting the issue? @Stephan Ewen<mailto:se...@apache.org> We need to support Flink 1.2.1 for now. Thanks for your response. ~ Abhinav From: Stephan Ewen <se...@apache.org> Date: Thursday, March 29, 2018 at 2:30 AM To: "dyana.rose" <dyana.r...@salecycle.com> Cc: user <user@flink.apache.org> Subject: Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes Using AWS credentials with Kubernetes are not trivial. Have you looked at AWS / Kubernetes docs and projects like https://github.com/jtblin/kube2iam<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjtblin%2Fkube2iam&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=7grOgK4xk0StaFwlGtAn%2B%2FlXRE%2BFA8ZUn6hcQeTAdJM%3D&reserved=0> which bridge between containers and AWS credentials? Also, Flink 1.2.1 is quite old, you may want to try a newer version. 1.4.x has a bit of an overhaul of the filesystems. On Wed, Mar 28, 2018 at 9:41 AM, dyana.rose <dyana.r...@salecycle.com<mailto:dyana.r...@salecycle.com>> wrote: Hiya, This sounds like it may be similar to the issue I had when running on ECS. Take a look at my ticket for how I got around this, and see if it's any help: https://issues.apache.org/jira/browse/FLINK-8439<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8439&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=tWqdbwBFGMoqo8e6axIT%2FT%2FAlIkYd0ohG57ds%2Bx95XA%3D&reserved=0> Dyana On 2018/03/28 02:15:06, "Bajaj, Abhinav" <abhinav.ba...@here.com<mailto:abhinav.ba...@here.com>> wrote: > Hi, > > I am trying to use Flink 1.2.1 with RockDB as statebackend and S3 for > checkpoints. > I am using Flink 1.2.1 docker images and running them in Kubernetes cluster. > > I have followed the steps documented in the Flink documentation - > https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#s3-simple-storage-service<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.2%2Fsetup%2Faws.html%23s3-simple-storage-service&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=12jEAdEQLvTwSRajfqRWF9gP%2BGNAdrkosxlBYnq%2FOys%3D&reserved=0> > > I am using AWS IAM roles to setup access for S3. > The role has actions "s3:GetObject","s3:ListBucket", "s3:PutObject", > "s3:DeleteObject" on the bucket. > > When I run a job, the jobmanager logs below exception – > > java.io.IOException: The given file URI (s3://$MY_TEST_BUCKET/checkpoints) > points to the HDFS NameNode at $MY_TEST_BUCKET, but the File System could not > be initialized with that address: Unable to load AWS credentials from any > provider in the chain > at > org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:334) > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:265) > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:304) > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:105) > at > org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:172) > at > org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:219) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:803) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:220) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:655) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:643) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:246) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.amazonaws.AmazonClientException: Unable to load AWS > credentials from any provider in the chain > at > com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117) > at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521) > at > com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) > at > com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) > at > org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:320) > ... 13 more > > I checked if the jobmanager pod in the K8s cluster has the correct IAM role > applied. > “curl > http://169.254.169.254/latest/meta-data/iam/security-credentials/<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F169.254.169.254%2Flatest%2Fmeta-data%2Fiam%2Fsecurity-credentials%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=pkGJ68oSVrQkV6nFmaF0ltyby89UIH6Rn%2FSMQYTsDE8%3D&reserved=0>” > returned the correct role. > > After this, I installed aws cli on the jobmanager pod and could > download/upload to $MY_TEST_BUCKET. > This confirmed that the jobmanager pod has the correct IAM role associated > with it. > > So, I am not sure why the AWS library in Flink is not able to load the > credentials. > Any thoughts or suggestions to fix or troubleshoot? > > Appreciate the help. > > Regards, > Abhinav Bajaj > > > [cid:image001.png@01D3C5FF.E9E41E50] > > Abhinav Bajaj > Lead Engineer > Open Location Platform > Mobile: +1 708 329 9516<tel:%2B1%20708%20329%209516> > > HERE Seattle > 701 Pike Street, suite 2000 > Seattle, WA 98101 USA > 47° 36' 41" N 122° 19' 57" W > > [cid:image002.png@01D3C5FF.E9E41E50]<http://360.here.com/<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F360.here.com%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=X8Ux%2Be05cX9Q%2BTvAuC0HlYcuJp%2BfwQ3aIO%2Bag7bs47w%3D&reserved=0>> > [cid:image003.png@01D3C5FF.E9E41E50] > <https://www.twitter.com/here<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.twitter.com%2Fhere&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=TJZSgQ0k8O%2FYMa9BbB%2BTXD26x598uaKBQgcP%2B8iPmCI%3D&reserved=0>> > [cid:image004.png@01D3C5FF.E9E41E50] > <https://www.facebook.com/here<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fhere&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=QbqjxO3PdJTSpgzXn9yMqIceu%2BEYeuDGkcPam9QDK7U%3D&reserved=0>> > [cid:image005.png@01D3C5FF.E9E41E50] > <https://www.linkedin.com/company/heremaps<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fheremaps&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=wIlLrDkW8bf5BUmiyNUWBq7LhNvLbUC%2FVwN1dqtr3j8%3D&reserved=0>> > [cid:image006.png@01D3C5FF.E9E41E50] > <https://www.instagram.com/here/<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fhere%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=Evj0O23%2BkW9rbV6JyxemXNbCqpugfYYmB6CUG5RL1CE%3D&reserved=0>> > > >
