Hi Mithalee,
lets start with why, Why are you using Kubernetes and not just EMR in EC2?
Do you have extremely bespoke library dependencies and requirements? Or
does you workloads fail in case the clusters do not scale up or down in a
few minutes?
Regards,
Gourav Sengupta
On Thu, May 20, 2021 at
Hi,
I am currently trying to run spark submit in Kubernetes. I have set up the
IAM roles for serviceaccount and generated the ARN. I am trying to use the
"spark.hadoop.fs.s3a.fast.upload=true --conf
spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
we have spark 2.4.x clusters running fine; however when running spark 3.1.1
the spark history server fails to boot
note: we build spark 3.1.1 from source for hadoop 3.2.1
we use supervisord to start services so our start command is:
$SPARK_HOME/bin/spark-class org.apache.spark.deploy.history.Hist
Still get the same error with “pyspark --conf queue=default --conf
executor-memory=24G”
From: ayan guha
Sent: Thursday, May 20, 2021 12:23 AM
To: Clay McDonald
Cc: Mich Talebzadeh ; user@spark.apache.org
Subject: Re: PySpark Write File Container exited with a non-zero exit code 143
*** EXTER
