[
https://issues.apache.org/jira/browse/KAFKA-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526189#comment-15526189
]
Michael Coon commented on KAFKA-4224:
-------------------------------------
I think this catches the problem with IndexOutOfBounds, but would still yield a
caught exception at the NetworkClient level. Basically, the challenge is that
the exception is being caught in the NetworkClient and ultimately get back a
ConsumerRecords with 0 count. I have no way of knowing that there was a
problem, so I keep trying. Oddly, even though subsequent fetch requests should
theoretically move the position in the partition forward, I keep getting the
error over and over. I can keep moving the offset forward until success but
that seems like it's going to introduce other problems or potentially skip
records if things take too many attempts to come back.
> IndexOutOfBounds in RecordsIterator causes infinite loop in NetworkClient
> -------------------------------------------------------------------------
>
> Key: KAFKA-4224
> URL: https://issues.apache.org/jira/browse/KAFKA-4224
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.10.0.1
> Reporter: Michael Coon
>
> For whatever reason, I seem to have a corrupted message that is returned from
> a broker that puts the consumer into an infinite loop. The
> org.apache.kafka.client.consumer.internals.Fetcher (line 590) is getting the
> next record from the RecordsIterator or MemoryRecords but when it attempts to
> decode the record, it throws "IndexOutOfBounds" exception. Unfortunately,
> that exception is merely logged and the Fetcher goes on to get the next
> message. But the exception apparently does not move the underlying buffer
> read forward in such a way that it would actually go and get the next record.
> The result: it keeps trying to read the corrupted record but can't make
> progress.
> I offer two potential solutions:
> 1) throw the exception up to me and let me figure out whether I want to skip
> forward in offsets
> 2) Make sure the underlying RecordsIterator actually moves forward on
> exceptions so that progress can be made when corrupted messages are found.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
