[
https://issues.apache.org/jira/browse/KAFKA-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526908#comment-15526908
]
Michael Coon commented on KAFKA-4224:
-------------------------------------
Line 278 NetworkClient attempts to parse the record by calling the Fetcher
response callback. It wraps that callback in try/catch and catches "Exception"
and just logs it. It does NOT throw it back to me. What would be awesome is if
the fact that a corrupted record were found, that some
"CorruptedRecordException" would be thrown back to me and I could interrogate
that exception for the corrupted offset/topic/partition and either skip it, log
it, do something magical, whatever. That gives me the ability to handle the
actual situation instead of guessing what I should be doing with the problem.
The real challenge is that with a consumer assigned multiple partitions, that
consumer needs to check some offset progress for all partitions and then
attempt to advance those that have not made progress--which may or may not be
due to corrupted records. It's not clean/efficient/clear at all.
> IndexOutOfBounds in RecordsIterator causes infinite loop in NetworkClient
> -------------------------------------------------------------------------
>
> Key: KAFKA-4224
> URL: https://issues.apache.org/jira/browse/KAFKA-4224
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.10.0.1
> Reporter: Michael Coon
>
> For whatever reason, I seem to have a corrupted message that is returned from
> a broker that puts the consumer into an infinite loop. The
> org.apache.kafka.client.consumer.internals.Fetcher (line 590) is getting the
> next record from the RecordsIterator or MemoryRecords but when it attempts to
> decode the record, it throws "IndexOutOfBounds" exception. Unfortunately,
> that exception is merely logged and the Fetcher goes on to get the next
> message. But the exception apparently does not move the underlying buffer
> read forward in such a way that it would actually go and get the next record.
> The result: it keeps trying to read the corrupted record but can't make
> progress.
> I offer two potential solutions:
> 1) throw the exception up to me and let me figure out whether I want to skip
> forward in offsets
> 2) Make sure the underlying RecordsIterator actually moves forward on
> exceptions so that progress can be made when corrupted messages are found.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
