Since you're using YARN, you may also need to set SPARK_YARN_USER_ENV to
"PYSPARK_PYTHON=/your/desired/python/on/slave/nodes".
2014-09-04 9:59 GMT-07:00 Davies Liu :
> Hey Oleg,
>
> In pyspark, you MUST have the same version of Python in all the
> machines of the cluster,
> which means when you
Hey Oleg,
In pyspark, you MUST have the same version of Python in all the
machines of the cluster,
which means when you run `python` on these machines, all of them
should be the same
version ( 2.6 or 2.7).
With PYSPARK_PYTHON, you can run pyspark with a specified version of
Python. Also,
you shou
Hi ,
I am evaluating the PySpark.
I have hdp hortonworks installed with python 2.6.6. (I can't remove it
since it is used by hortonworks). I can successfully execute PySpark on
Yarn.
We need to use Anaconda packages , so I install anaconda. Anaconda is
installed with python 2.7.7 and it is a