Re: PySpark PYTHONPATH issue after 0.8 to 0.9 upgrade

2021-03-11 Thread Vladimir Prus
I have figured the problem. - With the refactoring above, PYTHONPATH is set only in interpreter.sh and no longer set in PySparkInterpreter - I use sudo for impersonation - By default, sudo does not preserve environment, so I use "sudo -E" - But it *still* explicitly drops PYTHONPATH, as described i

PySpark PYTHONPATH issue after 0.8 to 0.9 upgrade

2021-03-11 Thread Vladimir Prus
Hi, we've upgraded from 0.8 to 0.9 and I observe that with the same interpreter settings, PySpark no longer works with: java.io.IOException: Fail to launch python process. Traceback (most recent call last): File "/tmp/1615477929423-0/zeppelin_python.py", line 20, in from