Thanks Jeff , yep that's what we have too in [1] - that's what we have currently in interpreter settings now. It doesn't work for some reason. We're running Zeppelin from ~May'18 snapshot - has anything changed since then?
Ruslan [1] LD_LIBRARY_PATH /opt/cloudera/parcels/Anaconda3/lib PATH /usr/java/latest/bin:/opt/cloudera/parcels/Anaconda3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/rdautkha/bin PYSPARK_DRIVER_PYTHON /opt/cloudera/parcels/Anaconda3/bin/python PYSPARK_PYTHON /opt/cloudera/parcels/Anaconda3/bin/python PYTHONHOME /opt/cloudera/parcels/Anaconda3 spark.executorEnv.LD_LIBRARY_PATH/ opt/cloudera/parcels/Anaconda3/lib spark.executorEnv.PYSPARK_PYTHON /opt/cloudera/parcels/Anaconda3/bin/python spark.pyspark.driver.python /opt/cloudera/parcels/Anaconda3/bin/python spark.pyspark.python /opt/cloudera/parcels/Anaconda3/bin/python spark.yarn.appMasterEnv.PYSPARK_PYTHON /opt/cloudera/parcels/Anaconda3/bin/python -- Ruslan Dautkhanov On Fri, Oct 26, 2018 at 9:10 PM Jeff Zhang <zjf...@gmail.com> wrote: > Hi Ruslan, > > I believe you can just set PYSPARK_PYTHON in spark interpreter setting to > switch between python2 and python3 > > > > Ruslan Dautkhanov <dautkha...@gmail.com>于2018年10月27日周六 上午2:26写道： > >> I'd like to give users ability to switch between Python2 and Python3 for >> their PySpark jobs. >> Was somebody able to set up something like this, so they can switch >> between python2 and python3 pyspark interpreters? >> >> For this experiment, created a new %py3spark interpreter, assigned to >> spark interpreter group. >> >> Added following options there for %py3spark: [1] >> /opt/cloudera/parcels/Anaconda3 is our Anaconda python3 home that's >> available on all worker nodes and on zeppelin server too. >> >> For default %pyspark interpreter it's very similar to [1], except all >> paths have "/opt/cloudera/parcels/Anaconda" instead of " >> /opt/cloudera/parcels/Anaconda3". >> >> Nevertheless, zeppelin_ipythonxxx/ipython_server.py >> seems catching environment variable from zeppelin-env.sh and not from >> interpreter settings. >> >> Zeppelin documentation reads that all uppercase variables will be >> treated as environment variables, so I assume it should overwrite what's >> in zeppelin-env.sh, no? >> >> It seems environment variables at interpreter level are broken - notice >> "pyspark" paragraph has "Anaconda3" and not "Anaconda" in PATH >> (highlighted). >> >> [image: image.png] >> >> >> >> [1] >> >> LD_LIBRARY_PATH /opt/cloudera/parcels/Anaconda3/lib >> PATH >> /usr/java/latest/bin:/opt/cloudera/parcels/Anaconda3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/rdautkha/bin >> PYSPARK_DRIVER_PYTHON /opt/cloudera/parcels/Anaconda3/bin/python >> PYSPARK_PYTHON /opt/cloudera/parcels/Anaconda3/bin/python >> PYTHONHOME /opt/cloudera/parcels/Anaconda3 >> >> spark.executorEnv.LD_LIBRARY_PATH/ opt/cloudera/parcels/Anaconda3/lib >> spark.executorEnv.PYSPARK_PYTHON >> /opt/cloudera/parcels/Anaconda3/bin/python >> spark.pyspark.driver.python /opt/cloudera/parcels/Anaconda3/bin/python >> spark.pyspark.python /opt/cloudera/parcels/Anaconda3/bin/python >> spark.yarn.appMasterEnv.PYSPARK_PYTHON >> /opt/cloudera/parcels/Anaconda3/bin/python >> >> -- >> Ruslan Dautkhanov >> >
