[jira] [Created] (ZEPPELIN-3991) Fail to bootstrap PySpark in yarn client mode

[Maziyar PANAHI (JIRA)]

Maziyar PANAHI created ZEPPELIN-3991:
----------------------------------------

             Summary: Fail to bootstrap PySpark in yarn client mode
                 Key: ZEPPELIN-3991
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3991
             Project: Zeppelin
          Issue Type: Bug
          Components: pySpark, python-interpreter
    Affects Versions: 0.9.0
         Environment: Cloudera/CDH 6.1

Spark 2.4

Hadoop 3.0

*zeppein-env.sh*
{code:java}
export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
export 
PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3

export PYSPARKPYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
export PYSPARKDRIVERPYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
{code}
            Reporter: Maziyar PANAHI


Hi,

PySpark fails with following error in Zeppelin 0.9.0:
{code:java}
ERROR [2019-02-06 22:23:40,599] ({FIFOScheduler-Worker-1} Job.java[run]:174) - 
Job failed
org.apache.zeppelin.interpreter.InterpreterException: Fail to bootstrap pyspark
at 
org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:124)
at 
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:593)
at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:502)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at 
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:121)
at 
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Fail to run bootstrap script: 
python/zeppelin_pyspark.py
at 
org.apache.zeppelin.python.PythonInterpreter.bootstrapInterpreter(PythonInterpreter.java:581)
at 
org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:122)
... 9 more
{code}
Full logs of YARN:

[https://gist.github.com/maziyarpanahi/3380e230246271217a2feb4512f5d665]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)