Re: Flink - Pod Identity

2021-04-03 Thread Sameer Wadkar
Kube2Iam needs to modify IPtables to proxy calls to ec2 metadata to a daemonset which runs privileged pods which maps a IP Address of the pods and its associated service account to make STS calls and return temporary AWS credentials. Your pod “thinks” the ec2 metadata url works locally like in a

Re: Flink - Pod Identity

2021-04-03 Thread Austin Cawley-Edwards
If you’re just looking to attach a service account to a pod using the native AWS EKS IAM mapping[1], you should be able to attach the service account to the pod via the `kubernetes.service-account` configuration option[2]. Let me know if that works for you! Best, Austin [1]: https://docs.aws.ama

Re: Flink - Pod Identity

2021-04-03 Thread Austin Cawley-Edwards
Can you describe your setup a little bit more? And perhaps how you use this setup to grant access to other non-Flink pods? On Sat, Apr 3, 2021 at 2:29 PM Swagat Mishra wrote: > Yes I looked at kube2iam, I haven't experimented with it. > > Given that the service account has access to S3, shouldn'

Re: Flink - Pod Identity

2021-04-03 Thread Swagat Mishra
Yes I looked at kube2iam, I haven't experimented with it. Given that the service account has access to S3, shouldn't we have a simpler mechanism to connect to underlying resources based on the service account authorization? On Sat, Apr 3, 2021, 10:10 PM Austin Cawley-Edwards wrote: > Hi Swagat,

Re: Flink - Pod Identity

2021-04-03 Thread Austin Cawley-Edwards
Hi Swagat, I’ve used kube2iam[1] for granting AWS access to Flink pods in the past with good results. It’s all based on mapping pod annotations to AWS IAM roles. Is this something that might work for you? Best, Austin [1]: https://github.com/jtblin/kube2iam On Sat, Apr 3, 2021 at 10:40 AM Swaga

Re: Flink - Pod Identity

2021-04-03 Thread Swagat Mishra
No we are running on aws. The mechanisms supported by flink to connect to resources like S3, need us to make changes that will impact all services, something that we don't want to do. So providing the aws secret key ID and passcode upfront or iam rules where it connects by executing curl/ http call

Re: Flink - Pod Identity

2021-04-03 Thread Israel Ekpo
Are you running on Azure Kubernetes Service. You should be able to do it because the identity can be mapped to the labels of the pods not necessary Flink. On Sat, Apr 3, 2021 at 6:31 AM Swagat Mishra wrote: > Hi, > > I think flink doesn't support pod identity, any plans tk achieve it in any > s

UniqueKey constraint is lost with multiple sources join in SQL

2021-04-03 Thread Kai Fu
Hi team, We have a use case to join multiple data sources to generate a continuous updated view. We defined primary key constraint on all the input sources and all the keys are the subsets in the join condition. All joins are left join. In our case, the first two inputs can produce *JoinKeyContai

Flink - Pod Identity

2021-04-03 Thread Swagat Mishra
Hi, I think flink doesn't support pod identity, any plans tk achieve it in any subsequent release. Regards, Swagat