Thanks Jun.
I grep the server log, and found 3 kinds of exception on broker.
1.
kafka.common.OffsetOutOfRangeException
I don't think this could cause broker closing the socket.
2.
579740110 [kafka-processor-6] ERROR kafka.network.Processor - Closing
socket for /10.47.7.56 because of error
java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:405)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:506)
at kafka.message.FileMessageSet.writeTo(FileMessageSet.scala:102)
at kafka.server.MessageSetSend.writeTo(MessageSetSend.scala:53)
at kafka.network.MultiSend.writeTo(Transmission.scala:91)
at kafka.network.Processor.write(SocketServer.scala:339)
at kafka.network.Processor.run(SocketServer.scala:216)
at java.lang.Thread.run(Thread.java:662)
3.
580861724 [kafka-processor-13] ERROR kafka.network.Processor - Closing
socket for /10.47.7.56 because of error
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:486)
at kafka.message.FileMessageSet.writeTo(FileMessageSet.scala:102)
at kafka.server.MessageSetSend.writeTo(MessageSetSend.scala:53)
at kafka.network.MultiSend.writeTo(Transmission.scala:91)
at kafka.network.Processor.write(SocketServer.scala:339)
at kafka.network.Processor.run(SocketServer.scala:216)
at java.lang.Thread.run(Thread.java:662)
probably the second exception may be the cause.
also, I have found the EOFException on consumer side doesn't cause stopping
fetching every time.
On Fri, May 3, 2013 at 11:50 PM, Jun Rao <jun...@gmail.com> wrote:
> The consumer hit the exception because the broker closed the socket. What
> does the broker log around the same time say? It should tell you the reason
> why the broker closed the socket.
>
> Thanks,
>
> Jun
>
>
> On Fri, May 3, 2013 at 2:34 AM, Helin Xiang <xkee...@gmail.com> wrote:
>
> > hi,
> >
> > We are doing some mirror-maker test。We currently use version 0.7.2。
> >
> > some times，the mirror maker will throw exceptions like this， and then
> > stopped fetching topics：
> >
> > java.io.EOFException: Received -1 when reading from channel, socket has
> > likely been closed.
> > at kafka.utils.Utils$.read(Utils.scala:539)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > at kafka.network.Receive$class.readCompletely(Transmission.scala:55)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > at
> kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
> > at
> > kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:125)
> > at
> kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
> > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > [2013-05-02 15:13:05,339] INFO stopping fetcher FetchRunnable-0 to host
> > kf.a.com (kafka.consumer.FetcherRunnable)
> >
> > I used jconsole to double check， and confirmed that the consumer in the
> > mirror maker stopped fetching logs brokers。I have to stop the mirror
> maker
> > and start it over， and it continued fetching logs。
> >
> > I start consumers using only 1 consumer thread。
> >
> > By the way，there are 16 mirror makers in the consume group。
> > I did some calculation and it can be assured that the partition numbers
> are
> > larger than or equal to the consumer threads。
> >
> > Thats all I got。So， I wonder if there was a bug issue for just mirror
> maker
> > or any applications using the high level consumers will behave like this？
> >
> >
> > thanks！
> >
> >
> > --
> > *Best Regards
> >
> > Helin Xiang*
> >
>
--
*Best Regards
向河林*
