[
https://issues.apache.org/jira/browse/KAFKA-8362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176934#comment-17176934
]
Luke Chen commented on KAFKA-8362:
----------------------------------
OK, after further investigation today, I found I was wrong. The root cause is
the *alterCheckpointDir* method doesn't remove the source dir data. It called
*updateCheckpoints* with None update data, which just update nothing to the
source dir file, but not removing the data. So, after the fix, the
*allCleanerCheckpoints* will work as expected (since the source dir is
removed). Submitted patch now. Thanks.
> LogCleaner gets stuck after partition move between log directories
> ------------------------------------------------------------------
>
> Key: KAFKA-8362
> URL: https://issues.apache.org/jira/browse/KAFKA-8362
> Project: Kafka
> Issue Type: Bug
> Components: jbod, log cleaner
> Reporter: Julio Ng
> Assignee: Luke Chen
> Priority: Major
>
> When a partition is moved from one directory to another, their checkpoint
> entry in cleaner-offset-checkpoint file is not removed from the source
> directory.
> As a consequence when we read the last firstDirtyOffset, we might get a stale
> value from the old checkpoint file.
> Basically, we need clean up the entry from the check point file in the source
> directory when the move is completed
> The current issue is that the code in LogCleanerManager:
> {noformat}
> /**
> * @return the position processed for all logs.
> */
> def allCleanerCheckpoints: Map[TopicPartition, Long] = {
> inLock(lock) {
> checkpoints.values.flatMap(checkpoint => {
> try {
> checkpoint.read()
> } catch {
> case e: KafkaStorageException =>
> error(s"Failed to access checkpoint file ${checkpoint.file.getName}
> in dir ${checkpoint.file.getParentFile.getAbsolutePath}", e)
> Map.empty[TopicPartition, Long]
> }
> }).toMap
> }
> }{noformat}
> collapses the offsets when multiple entries exist for the topicPartition
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
