Hello,
I've been trying to work out how to replicate execution of a python app using
spark-submit via the CLI using the native spark REST api
(http://localhost:6066/v1/submissions/create) for a couple of weeks without
success. The environment is docker using the latest docker for spark 3.2 image
from bitnami: one master and 2 workers in a standalone cluster.
These are two spark-submit commands via the CLI on the master for the same
application that successfully execute the app and process the file.
bin/spark-submit file:///opt/bitnami/spark/hostfiles/bronze.py --filename
"filename.json"
bin/spark-submit --deploy-mode "client"
file:///opt/bitnami/spark/hostfiles/bronze.py --filename "filename.json"
Here's the body for the api request trying to accomplish the same thing as the
CLI spark-submit.
{ "action": "CreateSubmissionRequest",
"appArgs": [
"/opt/bitnami/spark/hostfiles/bronze.py", "-filename", "filename.json"
],
"appResource": "file:/opt/bitnami/spark/hostfiles/bronze.py",
"clientSparkVersion": "3.2.0",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"
},
"mainClass": "org.apache.spark.deploy.SparkSubmit",
"sparkProperties": {
"spark.driver.supervise": "false",
"spark.app.name": "Spark REST API - Bronze Load",
"spark.submit.deployMode": "client",
"spark.master": "spark://spark:7077",
"spark.eventLog.enabled":"true"
}
}
The CreateSubmissionResponse is successful indicating that a driver is
successfully submitted, but fails with the following log info, which I am
failing to see where it indicates the actual error occurring.
> INFO Worker: Asked to launch driver driver-20220118233748-0006
> INFO DriverRunner: Copying user jar
> file:/opt/bitnami/spark/hostfiles/bronze.py to
> /opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py
> INFO Utils: Copying /opt/bitnami/spark/hostfiles/bronze.py to
> /opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py
> INFO DriverRunner: Launch Command: "/opt/bitnami/java/bin/java" "-cp"
> "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M"
> "-Dspark.eventLog.enabled=true" "-Dspark.app.name=Spark REST API - Bronze
> Load" "-Dspark.driver.supervise=false" "-Dspark.master=spark://spark:7077"
> "-Dspark.submit.deployMode=client"
> "org.apache.spark.deploy.worker.DriverWrapper"
> "spark://[email protected]:35803"
> "/opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py"
> "org.apache.spark.deploy.SparkSubmit" "/opt/bitnami/spark/hostfiles/bronze.py
> --filename 'filename.json'"
> WARN Worker: Driver driver-20220118233748-0006 exited with failure
Is there something obvious I am missing? I originally considered trying to use
airflow, but was told that was overkill and to find another, simpler mechanism
for triggering our spark jobs (from outside of spark). Would be more
straightforward to launch a spark worker image via kubernetes and pass the
filename argument?
Thanks,
Mike
This electronic message may contain information that is Proprietary,
Confidential, or legally privileged or protected. It is intended only for the
use of the individual(s) and entity named in the message. If you are not an
intended recipient of this message, please notify the sender immediately and
delete the material from your computer. Do not deliver, distribute or copy this
message and do not disclose its contents or take any action in reliance on the
information it contains. Thank You.
