[
https://issues.apache.org/jira/browse/KAFKA-9867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085254#comment-17085254
]
Bradley Peterson commented on KAFKA-9867:
-----------------------------------------
After upgrade to 2.4.1, I see the same error logged slightly differently.
{noformat}
[2020-04-16 20:53:14,748] WARN [kafka-log-cleaner-thread-0]: Unexpected
exception thrown when cleaning log Log(dir=/var/lib/kafka/xxxxx-10,
topic=xxxxx, partition=10, highWatermark=1191729811,
lastStableOffset=1191729811, logStartOffset=0, logEndOffset=1191729811).
Marking its partition (xxxxx-10) as uncleanable (kafka.log.LogCleaner)
kafka.log.LogCleaningException: Attempt to append an offset (279327173) to
position 4077 no larger than the last offset appended (279327173) to
/var/lib/kafka/xxxxx-10/00000000000278373839.index.cleaned.
at
kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:349)
at
kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:325)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:314)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Caused by: org.apache.kafka.common.errors.InvalidOffsetException: Attempt to
append an offset (279327173) to position 4077 no larger than the last offset
appended (279327173) to
/var/lib/kafka/xxxxx-10/00000000000278373839.index.cleaned.
{noformat}
> Log cleaner throws InvalidOffsetException for some partitions
> -------------------------------------------------------------
>
> Key: KAFKA-9867
> URL: https://issues.apache.org/jira/browse/KAFKA-9867
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Affects Versions: 2.3.0
> Environment: AWS EC2 with data on EBS volumes.
> Reporter: Bradley Peterson
> Priority: Major
>
> About half the partitions for one topic are marked "uncleanable". This is
> consistent across broker replicas – if a partition is uncleanable on one
> broker, its replicas are also uncleanable.
> The log-cleaner log seems to suggest out of order offsets. We've seen corrupt
> indexes before, so I removed the indexes from the affected segments and let
> Kafka rebuild them, but it hit the same error.
> I don't know when the error first occurred because we've restarted the
> brokers and rotated logs, but it is possible the broker's crashed at some
> point.
> How can I repair these partitions?
> {noformat}
> [2020-04-09 00:14:16,979] INFO Cleaner 0: Cleaning segment 223293473 in log
> xxxxx-9 (largest timestamp Wed Nov 13 20:35:57 UTC 2019
> ) into 223293473, retaining deletes. (kafka.log.LogCleaner)
> [2020-04-09 00:14:29,486] INFO Cleaner 0: Cleaning segment 226762315 in log
> xxxxx-9 (largest timestamp Wed Nov 06 18:17:10 UTC 2019
> ) into 223293473, retaining deletes. (kafka.log.LogCleaner)
> [2020-04-09 00:14:29,502] WARN [kafka-log-cleaner-thread-0]: Unexpected
> exception thrown when cleaning log Log(/var/lib/kafka/xxxxx
> -9). Marking its partition (xxxxx-9) as uncleanable (kafka.log.LogCleaner)
> org.apache.kafka.common.errors.InvalidOffsetException: Attempt to append an
> offset (226765178) to position 941 no larger than the last offset appended
> (228774155) to /var/lib/kafka/xxxxx-9/00000000000223293473.index.cleaned.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
