[
https://issues.apache.org/jira/browse/HUDI-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492468#comment-17492468
]
Danny Chen commented on HUDI-3435:
----------------------------------
After this change, when the compaction metadata commits successfully but the
data set commit state switch fails, the metadata table may bookkeep the
compaction files which has been rolledback(removed), but the odds are far less
than the case that metadata and data set commits never happens.
And the compaction files are actually idempotent.
Have no good solution to fix this completely, maybe we should check the archive
timeline for accurate check whether the instant to rollback is archived.
> Do not throw exception when instant to rollback does not exist in metadata
> table active timeline
> ------------------------------------------------------------------------------------------------
>
> Key: HUDI-3435
> URL: https://issues.apache.org/jira/browse/HUDI-3435
> Project: Apache Hudi
> Issue Type: Improvement
> Components: core
> Reporter: Danny Chen
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> See the stacktrace:
> {code:xml}
> Caused by: org.apache.hudi.exception.HoodieMetadataException: The instant
> [20220214211929120__deltacommit__COMPLETED] required to sync rollback of
> 20220214211929120 has been archived
> at
> org.apache.hudi.metadata.HoodieTableMetadataUtil.lambda$processRollbackMetadata$10(HoodieTableMetadataUtil.java:224)
> at java.util.HashMap$Values.forEach(HashMap.java:982)
> at
> java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1082)
> at
> org.apache.hudi.metadata.HoodieTableMetadataUtil.processRollbackMetadata(HoodieTableMetadataUtil.java:201)
> at
> org.apache.hudi.metadata.HoodieTableMetadataUtil.convertMetadataToRecords(HoodieTableMetadataUtil.java:178)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:653)
> at
> org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$2(BaseActionExecutor.java:77)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
> at
> org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:77)
> at
> org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:244)
> at
> org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:122)
> at
> org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:144)
> at
> org.apache.hudi.table.HoodieFlinkMergeOnReadTable.rollback(HoodieFlinkMergeOnReadTable.java:132)
> at
> org.apache.hudi.table.HoodieTable.rollbackInflightCompaction(HoodieTable.java:499)
> at
> org.apache.hudi.util.CompactionUtil.lambda$rollbackCompaction$1(CompactionUtil.java:163)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
> at
> org.apache.hudi.util.CompactionUtil.rollbackCompaction(CompactionUtil.java:161)
> at
> org.apache.hudi.sink.compact.CompactionPlanOperator.open(CompactionPlanOperator.java:73)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
