On 9 Feb 2021, at 19:46, Rishabh Jain <[email protected]> wrote:
Hi,
We are trying to access S3 from spark job running on EKS cluster pod. I
have a service account that has an IAM role attached with full S3
permission. We are using DefaultCredentialsProviderChain. But still we are
getting 403 Forbidden from S3.
It’s hard to say without any information, but some things you might want to
double-check
- Make sure the Spark job is using sufficiently new AWS SDK, so that IAM
for service account is supported
- Modify your job to print the effective role, e.g.
val stsClient = AWSSecurityTokenServiceClientBuilder.standard().build();
val request = new GetCallerIdentityRequest()
val identity = stsClient.getCallerIdentity(request)
println(identity.getArn())
- If the above does not print the expected role, verify that the pods
actually have the right service account, and
that AWS_ROLE_ARN/AWS_WEB_IDENTITY_TOKEN_FILE variables are set on the
pod, and that
the assume policy for the role does allow EKS to assume that role.
- If the above prints the expected role, then 403 error means you did not
setup IAM policies on your role/bucket.
Is there anything wrong with our approach?
Generally speaking, IAM for service accounts in EKS + Spark works, it's
just there's a lot of things that can go wrong the first time you do it.
HTH,
