Re: Flink - Pod Identity

Great, glad to hear it Swagat! Did you end up using Flink 1.6 or were you able to upgrade to Flink 1.12? Could you also link the ticket back here if you've already made it/ make sure it is not a duplicate of FLINK-18676 ? Best, Austin On Tue, Ap

Re: Flink - Pod Identity

I was able to solve the issue by providing a custom version of the presto jar. I will create a ticket and raise a pull request so that others can benefit from it. I will share the details here shortly. Thanks everyone for your help and support. Especially Austin, he stands out due to his interest

Re: Flink - Pod Identity

And actually, I've found that the correct version of the AWS SDK *is* included in Flink 1.12, which was reported and fixed in FLINK-18676 (see[1]). Since you said you saw this also occur in 1.12, can you share more details about what you saw there? Best, Austin [1]: https://issues.apache.org/jira

Re: Flink - Pod Identity

That looks interesting! I've also found the full list of S3 properties[1] for the version of presto-hive bundled with Flink 1.12 (see [2]), which includes an option for a KMS key (hive.s3.kms-key-id). (also, adding back the user list) [1]: https://prestodb.io/docs/0.187/connector/hive.html#amazon

Re: Flink - Pod Identity

Hi Austin, Thanks for your reply. Atm, I have upgraded to 1.12 version of Flink, but I still see the same issue. I have taken a look at presto as well. I am looking to experiment with the settings like S3_KMS_KEY_ID (provided in the link below). If this doesn't work, I Will look to modify the Pre

Re: Flink - Pod Identity

Hi Swagat, It looks like Flink 1.6 bundles the 1.11.165 version of the aws-java-sdk-core with the Presto implementation (transitively from Presto 0.185[1]). The minimum support version for the ServiceAccount authentication approach is 1.11.704 (see [2]) which was released on Jan 9th, 2020[3], long

Re: Flink - Pod Identity

Austin - In my case the set up is such that services are deployed on Kubernetes with Docker, running on EKS. There is also an istio service mesh. So all the services communicate and access AWS resources like S3 using the service account. Service account is associated with IAM roles. I have verifie

Re: Flink - Pod Identity

Kube2Iam needs to modify IPtables to proxy calls to ec2 metadata to a daemonset which runs privileged pods which maps a IP Address of the pods and its associated service account to make STS calls and return temporary AWS credentials. Your pod “thinks” the ec2 metadata url works locally like in a

Re: Flink - Pod Identity

If you’re just looking to attach a service account to a pod using the native AWS EKS IAM mapping[1], you should be able to attach the service account to the pod via the `kubernetes.service-account` configuration option[2]. Let me know if that works for you! Best, Austin [1]: https://docs.aws.ama

Re: Flink - Pod Identity

Can you describe your setup a little bit more? And perhaps how you use this setup to grant access to other non-Flink pods? On Sat, Apr 3, 2021 at 2:29 PM Swagat Mishra wrote: > Yes I looked at kube2iam, I haven't experimented with it. > > Given that the service account has access to S3, shouldn'

Re: Flink - Pod Identity

Yes I looked at kube2iam, I haven't experimented with it. Given that the service account has access to S3, shouldn't we have a simpler mechanism to connect to underlying resources based on the service account authorization? On Sat, Apr 3, 2021, 10:10 PM Austin Cawley-Edwards wrote: > Hi Swagat,

Re: Flink - Pod Identity

Hi Swagat, I’ve used kube2iam[1] for granting AWS access to Flink pods in the past with good results. It’s all based on mapping pod annotations to AWS IAM roles. Is this something that might work for you? Best, Austin [1]: https://github.com/jtblin/kube2iam On Sat, Apr 3, 2021 at 10:40 AM Swaga

Re: Flink - Pod Identity

No we are running on aws. The mechanisms supported by flink to connect to resources like S3, need us to make changes that will impact all services, something that we don't want to do. So providing the aws secret key ID and passcode upfront or iam rules where it connects by executing curl/ http call

Re: Flink - Pod Identity

Are you running on Azure Kubernetes Service. You should be able to do it because the identity can be mapped to the labels of the pods not necessary Flink. On Sat, Apr 3, 2021 at 6:31 AM Swagat Mishra wrote: > Hi, > > I think flink doesn't support pod identity, any plans tk achieve it in any > s