Zeppelin doesn't need to take care of py4j, it is spark's responsibility. As long as you set SPARK_HOME, spark will find py4j properly.
Michaël Van de Borne <michael.vandebo...@gmail.com>于2018年1月23日周二 下午10:37写道： > As long as SPARK_HOME is concerned, this might work. > However, other variables, such as PYTHONPATH and SPARK_YARN_USER_ENV > should be defined in zeppelin-env.sh as well. > Without those variables, zeppelin cannot access, for instance, py4j or > pyspark. > And guess where py4j sits? in SPARK_HOME/python/lib/ > So it's a chicken-and-egg problem: if I define the SPARK_HOME in > zeppelin-env.sh, I cannot run both spark version. And if I don't, then > other paths are not defined, and zeppelin doesn't know where to look for > py4j and pyspark. > > 2018-01-23 12:45 GMT+01:00 Jeff Zhang <zjf...@gmail.com>: > >> >> Just remove SPARK_HOME in spark-env.sh, and instead define them in >> spark’s interpreter setting. You can create 2 spark interpreter, one for >> spark 1.6, another for spark 2. Only difference between them is the >> SPARK_HOME you defined in interpreter setting. >> >> >> >> Michaël Van de Borne <michael.vandebo...@gmail.com>于2018年1月23日周二 >> 下午6:42写道： >> >>> Hi list, >>> >>> I'd like my notebooks to support both spark 1.6 & spark 2. >>> I managed to get both versions working fine, with the SPARK_HOME >>> variable in zeppelin-env.sh. But just one at a time. >>> So I need to change the variable and restart zeppelin when I want to >>> swap spark versions. >>> Is it possible to somehow map a spark distribution to an interpreter >>> group so that I can get both versions supported at the same time? >>> >>> >>> Thank you, >>> >>> cheers, >>> >>> m. >>> >> >
