Thomas Graves created SPARK-22151:
-------------------------------------
Summary: PYTHONPATH not picked up from the spark.yarn.appMasterEnv
properly
Key: SPARK-22151
URL: https://issues.apache.org/jira/browse/SPARK-22151
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 2.1.1
Reporter: Thomas Graves
the code looks at the env variables:
val pythonPathStr = (sys.env.get("PYTHONPATH") ++ pythonPath)
But when you set spark.yarn.appMasterEnv it puts it into the local env.
So the python path set in spark.yarn.appMasterEnv isn't properly set.
You can work around if you are running in cluster mode by setting it on the
client like:
PYTHONPATH=./addon/python/ spark-submit
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
