when driver pod throws exception , driver pod still running ？
kubectl logs wordcount-e3141c7834d3dd68-driver
21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to 'value1
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot
resolve '`value1`' given input columns: [key, value];
'Aggregate [key#6], [key#6, count('value1) AS cnt#14]
+- Project [(id#4 % 5) AS key#6, (id#4 % 10) AS value#7]
+- Project [value#1 AS id#4]
+- LocalRelation [value#1]
at
org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
kubectl get pods wordcount-e3141c7834d3dd68-driver
NAME READY STATUS RESTARTS AGE
wordcount-e3141c7834d3dd68-driver 1/1 Running 0 2m58s
On 03/12/2021 05:42，Attila Zsolt Piros<[email protected]> wrote：
> but the spark-submit log still running
Set the "spark.kubernetes.submission.waitAppCompletion" config to false to
change that. As the doc says:
"spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether to
wait for the application to finish before exiting the launcher process. When
changed to false, the launcher has a "fire-and-forget" behavior when launching
the Spark job.
On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros
<[email protected]> wrote:
For getting the logs please read Accessing Logs part of the Running Spark on
Kubernetes page.
For stopping and generic management of the spark application please read the
Spark Application Management, where you find the example:
$ spark-submit --kill spark:spark-pi*--master k8s://https://192.168.2.8:8443
On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <[email protected]> wrote:
when run the code in k8s , driver pod throw AnalysisException , but the
spark-submit log still running , then how to get the exception and stop pods ?
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as
value")
.groupBy("key").agg(count("value1").as("cnt"))
df.show()
spark.stop()
bin/spark-submit \
--master k8s://https://localhost:9443 \
--deploy-mode cluster \
--name wordcount \
--class k8s.WordCount \
--conf spark.kubernetes.container.image=rspark:v3.1.1 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
/data/spark-example-1.0.0.jar
