Thanks for the quick reply Danny. The message size as per the DumpLogSegments is around 59KB
I used a very high message.max.size and a high fetchsize of 1 MB (that's the message.max.size in the broker) and still the same hang behavior. Also tried a max-wait-ms so that the consumer does not "hang" - but still the same result. Here's what I used - kafka-simple-consumer-shell.sh --broker-list $HOSTNAME:9092 --fetchsize 1000000 --max-messages 10-max-wait-ms 10000 --offset 7207844650 --partition 0 --print-offsets --topic RtbBid --property message.max.size=1000000 -----Original Message----- From: Danny Bahir [mailto:dannyba...@gmail.com] Sent: Thursday, June 02, 2016 10:06 PM To: users@kafka.apache.org Subject: Re: Problematic messages in Kafka quoting from https://cwiki.apache.org/confluence/display/KAFKA/FAQ The high-level consumer will block if the next message available is larger than the maximum fetch size you have specified - One possibility of a stalled consumer is that the fetch size in the consumer is smaller than the largest message in the broker. You can use the DumpLogSegments tool to figure out the largest message size and set fetch.size in the consumer config accordingly. On Thu, Jun 2, 2016 at 3:41 PM, Thakrar, Jayesh < jthak...@conversantmedia.com> wrote: > Wondering if anyone has encountered similar issues. > > Using Kafka 0.8.2.1. > > Occasionally, we encounter a situation in which a consumer (including > kafka-console-consumer.sh) just hangs. > If I increment the offset to skip the offending message, things work > fine again. > > I have been able to identify the message offset and the data file (log > file) containing the message. > > However, using kafka.tools.DumpLogSegments, I can dump the message > using commands like this - > > /usr/hdp/current/kafka-broker/bin/kafka-run-class.sh > kafka.tools.DumpLogSegments --files 00000000007207840027.log > --deep-iteration > > /usr/hdp/current/kafka-broker/bin/kafka-run-class.sh > kafka.tools.DumpLogSegments --files 00000000007207840027.log > --print-data-log --deep-iteration > > From the DumLogSegments program, here's the checksum info that I get - > offset: 7207844652 position: 398291668 isvalid: true payloadsize: > 59041 > magic: 0 compresscodec: NoCompressionCodec crc: 186430976 keysize: 12 > > So it looks like the message is ok, since there's also a CRC checksum. > Has anyone encountered such an issue? > Is there any explanation or reason for the broker behavior? > I have the data/log file saved if there is any troubleshooting that > can be done. > > When the broker reads the message and it seems to hang forever, I have > to kill the console-consumer or our application consumer. > > When I do that, here's what I see in the broker's log file > > [2016-06-02 15:50:45,117] INFO Closing socket connection to / > 10.110.102.113. (kafka.network.Processor) > [2016-06-02 15:50:45,139] INFO Closing socket connection to / > 10.110.102.113. (kafka.network.Processor) > [2016-06-02 15:50:49,142] ERROR Closing socket for /10.110.100.46 > because of error (kafka.network.Processor) > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > at kafka.utils.Utils$.read(Utils.scala:375) > at > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) > at kafka.network.Processor.read(SocketServer.scala:347) > at kafka.network.Processor.run(SocketServer.scala:245) > at java.lang.Thread.run(Thread.java:724) > [2016-06-02 15:50:49,936] INFO Closing socket connection to / > 10.110.105.134. (kafka.network.Processor) > [2016-06-02 15:50:51,591] INFO Closing socket connection to / > 10.110.102.113. (kafka.network.Processor) > [2016-06-02 15:50:51,699] INFO Closing socket connection to / > 10.110.102.113. (kafka.network.Processor) > > > > > > > This email and any files included with it may contain privileged, > proprietary and/or confidential information that is for the sole use > of the intended recipient(s). Any disclosure, copying, distribution, > posting, or use of the information contained in or attached to this > email is prohibited unless permitted by the sender. If you have > received this email in error, please immediately notify the sender via > return email, telephone, or fax and destroy this original transmission > and its included files without reading or saving it in any manner. > Thank you. > This email and any files included with it may contain privileged, proprietary and/or confidential information that is for the sole use of the intended recipient(s). Any disclosure, copying, distribution, posting, or use of the information contained in or attached to this email is prohibited unless permitted by the sender. If you have received this email in error, please immediately notify the sender via return email, telephone, or fax and destroy this original transmission and its included files without reading or saving it in any manner. Thank you.
