InvalidMessageException: Message is corrupt, Consumer stuck

[Henry Cai]

Hi,

We are using the Kafka high-level consumer 8.1.1, somehow we got a
corrupted message in the topic.  We are not sure the root cause of this,
but the problem we are having now is the HL consumer is stuck in that
position:

kafka.message.InvalidMessageException: Message is corrupt (stored crc =
537685622, computed crc = 36513351)

        at kafka.message.Message.ensureValid(Message.scala:166)

        at
kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:102)

        at
kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)

        at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)

        at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)


If we try to ignore that exception and iterate to the next message, the
iterator couldn't pass that error state:

java.lang.IllegalStateException: Iterator is in failed state

        at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:54)


By looking at the code, looks like you can only calling
IteratorTemplate.resetState() to clear the state, but this is an internal
method, is this the right way to workaround this problem?