Wondering if anyone has encountered similar issues.
Using Kafka 0.8.2.1.
Occasionally, we encounter a situation in which a consumer (including
kafka-console-consumer.sh) just hangs.
If I increment the offset to skip the offending message, things work fine again.
I have been able to identify the message offset and the data file (log file)
containing the message.
However, using kafka.tools.DumpLogSegments, I can dump the message using
commands like this -
/usr/hdp/current/kafka-broker/bin/kafka-run-class.sh
kafka.tools.DumpLogSegments --files 00000000007207840027.log --deep-iteration
/usr/hdp/current/kafka-broker/bin/kafka-run-class.sh
kafka.tools.DumpLogSegments --files 00000000007207840027.log --print-data-log
--deep-iteration
>From the DumLogSegments program, here's the checksum info that I get -
offset: 7207844652 position: 398291668 isvalid: true payloadsize: 59041 magic:
0 compresscodec: NoCompressionCodec crc: 186430976 keysize: 12
So it looks like the message is ok, since there's also a CRC checksum.
Has anyone encountered such an issue?
Is there any explanation or reason for the broker behavior?
I have the data/log file saved if there is any troubleshooting that can be done.
When the broker reads the message and it seems to hang forever, I have to kill
the console-consumer or our application consumer.
When I do that, here's what I see in the broker's log file
[2016-06-02 15:50:45,117] INFO Closing socket connection to /10.110.102.113.
(kafka.network.Processor)
[2016-06-02 15:50:45,139] INFO Closing socket connection to /10.110.102.113.
(kafka.network.Processor)
[2016-06-02 15:50:49,142] ERROR Closing socket for /10.110.100.46 because of
error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:375)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:347)
at kafka.network.Processor.run(SocketServer.scala:245)
at java.lang.Thread.run(Thread.java:724)
[2016-06-02 15:50:49,936] INFO Closing socket connection to /10.110.105.134.
(kafka.network.Processor)
[2016-06-02 15:50:51,591] INFO Closing socket connection to /10.110.102.113.
(kafka.network.Processor)
[2016-06-02 15:50:51,699] INFO Closing socket connection to /10.110.102.113.
(kafka.network.Processor)
This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s). Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender. If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.
