Hi Mithalee,
lets start with why, Why are you using Kubernetes and not just EMR in EC2?
Do you have extremely bespoke library dependencies and requirements? Or
does you workloads fail in case the clusters do not scale up or down in a
few minutes?
Regards,
Gourav Sengupta
On Thu, May 20, 2021 at
Hi,
I am currently trying to run spark submit in Kubernetes. I have set up the
IAM roles for serviceaccount and generated the ARN. I am trying to use the
"spark.hadoop.fs.s3a.fast.upload=true --conf
spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
