Github user yywwd commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@zjffdu I'm sorry to response slowly, because I tried to reproduce the
bugs. I thought it may caused by my codes, so I tried to use official PySpark
examples, the bugs still occurred. This is the code I used.
https://github.com/apache/incubator-livy/blob/master/examples/src/main/python/pi_app.py
**Note:** I comment the last line # client.stop(True), cause I won't want
to close the session after submitting just one job. The details about this bug
are as follows:
1. When I use default yarn mode, that is "yarn", official PySpark examples
and programmatic API work well. Using Livy interpreter in Zeppelin will throw
exception:` NoSuchMethodError: org.apache.spark.ui.SparkUI.appUIAddress()` for
the spark master.
2. ThenI change the yarn mode into "yarn-cluster" as SivaKaviyappa
suggested, Zeppelin works well. But the logs of this statement will hav a
warning: `"Warning: Master yarn-cluster is deprecated since 2.0. Please use
master \"yarn\" with specified deploy mode instead.`
However, Using programmatic API will have such bug:
2.1 I delete all Livy sessions, and run pi_app.py. It throw such
exception:
```
ReadTimeout: HTTPConnectionPool(host='172.31.5.251', port=8998): Read timed
out. (read timeout=10)
Traceback (most recent call last):
File "/home/ec2-user/wandongwu/livy_test_9/pi_app.py", line 35, in
<module>
pi = client.submit(pi_job).result()
File "/usr/local/lib/python2.7/site-packages/concurrent/futures/_base.py",
line 462, in result
return self.__get_result()
File
"/usr/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 414,
in __get_result
raise exception_type, self._exception, self._traceback
TypeError: raise: arg 3 must be a traceback or None
```
But I find It has started a new Livy session. So I edit configuration
parameter into `'http://<livy-server>:8998/sessions/0' 2`. It will throw
another exception:
```
Traceback (most recent call last):
File "/home/ec2-user/wandongwu/livy_test_9/pi_app.py", line 35, in
<module>
pi = client.submit(pi_job).result()
File
"/usr/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 462,
in result
return self.__get_result()
File
"/usr/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 414,
in __get_result
raise exception_type, self._exception, self._traceback
Exception: org.apache.livy.repl.PythonJobException: Client job
error:Traceback (most recent call last):
File
"/mnt/yarn/usercache/livy/appcache/application_1528945006613_0302/container_1528945006613_0302_01_000001/tmp/4991895008696585180",
line 159, in processBypassJob
deserialized_job = pickle.loads(serialized_job)
File "/usr/lib64/python2.7/pickle.py", line 1388, in loads
return Unpickler(file).load()
File "/usr/lib64/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File "/usr/lib64/python2.7/pickle.py", line 1096, in load_global
klass = self.find_class(module, name)
File "/usr/lib64/python2.7/pickle.py", line 1130, in find_class
__import__(module)
ImportError: No module named cloudpickle.cloudpickle
```
3. Then I change yarn mode into default mode, that is "yarn", programmatic
API can work well, but Zeppelin still cannot work.
---
