Hi all, I’d like to discuss an issue described here: https://issues.apache.org/jira/browse/SPARK-46912 It also has some old linked PR, but it’s stale and the changes are not quite suitable (because the keys are hardcoded)
I’ve made some debug/ research and found out that wrong env issue happens because of buildProcessBuilder method of org.apache.spark.deploy.worker.CommandUtils Process builder contains the correct env (including java home) but it’s replaced by localCommandEnv by keys. Which has env of the “submitter” machine (buildLocalCommand method substitutes only some library path envs) According to my research it only affects driver in Stand alone cluster (only cluster mode), but only if spark-submit is used (which internally uses ClientApplication and Client and sets env in onStart()) Executors only get two envs on start (OMP_NUM_THREADS and SPARK_USER), so the correct environments are set in buildProcessBuilder method. (Nothing is replaced) Submitting via REST is ok, because it doesn't provide any env variables by default (only via environmentVariables field of the request). I’d like to fix this bug, but I’d also like to discuss how to fix it. My suggestion would be: 1) Fully replace localCommand env by keys with System.env of the worker machine. And (optional) Make a similar(like REST) arg for spark submit command on StandAlone: —environment Or??? (We could also change JAVA_HOME resolving priority of buildJavaCommand method of org.apache.spark.launcher.AbstractCommandBuilder, but it would only fix java command path…the rest of the envs would be incorrect) As I can see driver local mode wouldn’t be affected, neither Yarn and K8s modes (because they have a separate clients for submitting) Regards, Melton
