Try to add the following to log4j.properties for more log
log4j.logger.org.apache.zeppelin.interpreter=DEBUG
小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午5:13写道：
> Actually, there was not created the interpreter log file as said 'Cannot
> run interpreter' in the log. And it was run on local machine as MAC, i mean
> stand alone. i stopped every firewall to make it easy. And again,this
> happened just on Mac. On CentOS, it works fine.
> That makes me confusion. :-b
>
> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>
>>
>> Could you check the interpreter log ? The issue of not be able to
>> connecting to thrift port could happen due to network issue, like firewall
>> or you use VPN, etc.
>>
>>
>>
>> 小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>
>>> Hi,
>>>
>>> I show the all log on this error.There are 2 type error. The 1 is at the
>>> first run. The 2 is at the second run. Seems like could not established to
>>> Thrift callback port 51348 that was set at the first run.
>>> First of all, does it need to establish the callback port of Thrift
>>> server?
>>> The notebook script was just
>>> %pyspark
>>> print 'aaa'
>>> Any advice appreciate.
>>>
>>> 1.
>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
>>> 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) -
>>> Job 20180302-135047_99102973 started by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>> pyspark
>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
>>> InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
>>> [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>>> InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not
>>> found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
>>> Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>> 13:11:14,023] ({pool-2-thread-2}
>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>> callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049]
>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run
>>> interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10,
>>> -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g,
>>> spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter
>>> process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2}
>>> Job.java[run]:184) - Job failed
>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>> interpreter at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
>>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>> interpreter at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075]
>>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job
>>> 20180302-135047_99102973 is finished, status: ERROR, exception:
>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>> INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>> finished by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>
>>> 2.
>>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
>>> SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
>>> started by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>> pyspark
>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) -
>>> Job failed
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>> at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>>> at
>>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>> ... 13 more
>>> Caused by: org.apache.thrift.transport.TTransportException:
>>> java.net.ConnectException: Connection refused (Connection refused)
>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>>> ... 20 more
>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>> refused)
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net
>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>> at java.net
>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>> at java.net
>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:589)
>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>> ... 21 more
>>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
>>> NotebookServer.java[afterStatusChange]:2219) - Error
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>> at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>>> at
>>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>> ... 13 more
>>> Caused by: org.apache.thrift.transport.TTransportException:
>>> java.net.ConnectException: Connection refused (Connection refused)
>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>>> ... 20 more
>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>> refused)
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net
>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>> at java.net
>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>> at java.net
>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:589)
>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>> ... 21 more
>>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
>>> NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
>>> is finished, status: ERROR, exception:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused), result:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>> finished by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>
>>> 2018-03-02 15:07 GMT+09:00 小野圭二 <onoke...@gmail.com>:
>>>
>>>> Hi Jeff,
>>>>
>>>> No one graved 63039 so far on Mac.
>>>> Yes, this issue has been reproduced as changing the port NO randomly.
>>>>
>>>> Ono
>>>>
>>>>
>>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>>>>
>>>>>
>>>>> It is possible that 63039 is used by other processes, as Mac can not
>>>>> detect whether this port is used or not from my experience.
>>>>>
>>>>> And can you reproduce this issue consistently ? Because each time
>>>>> zeppelin will chose a random port as long as you don't specify the
>>>>> portRange, so if you reproduce it consistently, it might due to other
>>>>> issues.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 小野圭二 <onoke...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> I am so confusing now, let me know why if You know this, please.
>>>>>>
>>>>>> The env:
>>>>>> Mac: High Sierra ver.10.13.3
>>>>>> Zep: 0.7.3
>>>>>> Spark: 2.2.0
>>>>>> Java: 1.8
>>>>>> notebook script: %pyspark print 'aaa' (just it for
>>>>>> test)
>>>>>>
>>>>>> In this env, i got the error in log as
>>>>>>
>>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>>
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>
>>>>>>
>>>>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>>>>> prepares a Port for callback to Thrift.
>>>>>>
>>>>>>
>>>>>> INFO ({pool-2-thread-1}
>>>>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>>>>> callback will start. Port: 63039
>>>>>>
>>>>>>
>>>>>> I guess, the reason why the error was Zeppelin could not attache to
>>>>>> the Thrift PORT.
>>>>>>
>>>>>>
>>>>>> And my confusion is, when i run the same module on CentOS, it worked
>>>>>> fine.
>>>>>> CentOS: 7
>>>>>> Zep: same
>>>>>> Spark: same
>>>>>> Java: same
>>>>>> notebook script: same
>>>>>>
>>>>>> And there was no the Thift port description in the log file. From
>>>>>> this consequence, the fine working was cause of did not use Thrift port
>>>>>> on
>>>>>> CentOS.
>>>>>>
>>>>>>
>>>>>> Do You know actually there is different behavior between Mac and
>>>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Ono
>>>>>>
>>>>>
>>>>
>>>
>
