Interesting.
Yes, it might be that the Spark Context crashed.
I'll do few tests for that and update.
On Tue, Jun 23, 2015 at 6:36 AM, moon soo Lee <m...@apache.org> wrote:
> for broken pipe error,
>
> Zeppelin runs Spark driver (SparkContext) in separate JVM process and
> communicate through thrift protocol. The error is about socket
> disconnection between Zeppelin server daemon and Spark driver JVM process.
>
> The reason can be vary. One possibility is, once SparkContext is
> terminated by any reason, it sometimes terminates the process, too. Then
> Zeppelin server could raise such exception.
>
> Do you see any possibility of disconnection between Zeppelin server daemon
> and Spark driver JVM process in your environment?
>
> Thanks,
> moon
>
> On Mon, Jun 22, 2015 at 8:29 PM moon soo Lee <m...@apache.org> wrote:
>
>> Hi,
>>
>> Really appreciate for sharing the problem.
>> About the NPE, when do you get the exception? Are you getting the
>> exception when you trying to access Spark application ui?
>> If it is, any spark job (like .count(), .collect(), sparksql, etc) runs
>> without error?
>>
>> Thanks!
>> moon
>>
>> On Mon, Jun 22, 2015 at 5:19 AM Ophir Cohen <oph...@gmail.com> wrote:
>>
>>> OK,
>>> I think we can seperate the issues.
>>> Let's start from the NPE. For some reason (is that on perpuse?) when
>>> Zeppelin starts Spark it seems that it does not open the web ui.
>>> Trying to access port 4041 (this is the one it started on) yield the
>>> above NPE with the "WARN [2015-06-22 09:22:37,265] ({qtp622396559-74}
>>> ServletHandler.java[doHandle]:561) - /jobs/
>>> java.lang.NullPointerException" message.
>>>
>>> Any ideas?
>>>
>>> On Mon, Jun 22, 2015 at 12:34 PM, Ophir Cohen <oph...@gmail.com> wrote:
>>>
>>>> Hi Guys and great work!
>>>> In our company we decided to relay on Zeppelin as our tool to access
>>>> and visualize our data.
>>>> I'm working with Spark 1.3.1 on top of Hadoop yarn 2.4
>>>>
>>>> Lately we started to see many NullPointerExcpetion on the Spark
>>>> interpreter log (see below).
>>>> Sometimes it does not affect Zeppelin but sometimes we start to see
>>>> 'can't get status' error caused by broken pipe, this time on the
>>>> *zeppelin* log file (see below).
>>>> When the latest error starts - it stacks the Spark interpretor and it
>>>> enforce us to restart Zeppelin.
>>>>
>>>> I couldn't find when and why it's happening, it just happened once
>>>> awhile...
>>>> Any help will be more than appreciated!
>>>> 10x!
>>>>
>>>> *The NPE on zeppelin-interpreter-spark log file:*
>>>> WARN [2015-06-22 09:22:37,265] ({qtp622396559-74}
>>>> ServletHandler.java[doHandle]:561) - /jobs/
>>>> java.lang.NullPointerException
>>>> at
>>>> org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
>>>> at
>>>> org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
>>>> at
>>>> org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
>>>> at
>>>> org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
>>>> at
>>>> org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
>>>> at
>>>> org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>>> at
>>>> org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>>> at
>>>> org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>>> at org.spark-project.jetty.server.Server.handle(Server.java:370)
>>>> at
>>>> org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
>>>> at
>>>> org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
>>>> at
>>>> org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
>>>> at
>>>> org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>>>> at
>>>> org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>>> at
>>>> org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>>> at org.spark-project.jetty.io
>>>> .nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
>>>> at org.spark-project.jetty.io
>>>> .nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>>>> at
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>> at
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> *The broken pipe on zeppelin log file:*
>>>> ERROR [2015-06-21 13:10:31,368] ({Thread-120}
>>>> RemoteScheduler.java[getStatus]:245) - Can't get status information
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.SocketException: Broken pipe
>>>> at
>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>> at
>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
>>>> at
>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.send_getStatus(RemoteInterpreterService.java:319)
>>>> at
>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getStatus(RemoteInterpreterService.java:311)
>>>> at
>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.getStatus(RemoteScheduler.java:232)
>>>> at
>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.run(RemoteScheduler.java:183)
>>>> Caused by: java.net.SocketException: Broken pipe
>>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>>> at
>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>>> at
>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>>>> at
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>> at
>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>> at
>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>> ... 5 more
>>>>
>>>>
>>>
