Hi Jeff,
I added one line into zeppelin-env.sh
export
PYTHONPATH=/usr/lib/spark/python:/usr/lib/spark/python/lib/py4j-src.zip:${PYTHONPATH}
# note spark's py4j is newer than zeppelin's
This modification is picked up however as far as I can tell some scripts
for python interpreter initialisation added its own py4j. I'm using
zeppelin 0.8.2 from AWS EMR.
Regards
-Lu Rui
On Fri, Apr 2, 2021 at 9:05 AM Jeff Zhang <zjf...@gmail.com> wrote:
> Could you share how do you include pyspark's py4j to python interpreter ?
>
> Rui Lu <r...@appannie.com> 于2021年3月25日周四 下午10:49写道：
>
>> Hi all,
>>
>> I’m trying to switch from pyspark interpreter to python interpreter and
>> ran into weird errors of py4j like “key error ‘x’” or “invalid command” or
>> so when creating spark session.
>>
>> A little digging reveals that zeppelin has its own py4j stuffed into
>> PYTHONPATH of python interpreter, the value of PYTHONPATH I get from within
>> python interpreter notebook is
>>
>>
>> '/usr/lib/zeppelin/interpreter/python/py4j-0.9.2/src:/usr/lib/spark/python:/usr/lib/spark/python/lib/py4j-0.10.7-src.zip'
>>
>> The latter parts with /usr/lib/spark are added into PYTHONPATH by me in
>> zeppelin-env.sh. However no matter how I arrange the order of paths in
>> PYTHONPATH, somehow zeppelin managed to set its py4j in the first place,
>> preventing python interpreter from finding the right py4j that comes with
>> spark. I suppose zeppelin manipulated PYTHONPATH after loading
>> zeppelin-env.sh.
>>
>> The reason I prefer python interpreter is that I would like to have fine
>> control over spark params per notebook (whereas pyspark uses the same set
>> of spark params for all notebooks)
>>
>> Has anyone run into the same issue or is there a workaround for this?
>>
>> Thanks!
>> --
>> Lu Rui
>>
>> *This email may contain or reference confidential information and is
>> intended only for the individual to whom it is addressed. Please refrain
>> from distributing, disclosing or copying this email and the information
>> contained within unless you are the intended recipient. If you received
>> this email in error, please notify us at le...@appannie.com
>> <le...@appannie.com>** immediately and remove it from your system.*
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
*This email may contain or reference confidential information and is
intended only for the individual to whom it is addressed. Please refrain
from distributing, disclosing or copying this email and the information
contained within unless you are the intended recipient. If you received
this email in error, please notify us at le...@appannie.com
<mailto:le...@appannie.com>** immediately and remove it from your system.*
