Livia Zhu created SPARK-52637:
---------------------------------
Summary: Fix VERSIONID_MISMATCH error when RocksDB compaction
leads to incorrect file mapping during load
Key: SPARK-52637
URL: https://issues.apache.org/jira/browse/SPARK-52637
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.0.0
Reporter: Livia Zhu
We found a bug leading to checkpoint corruption with RocksDB VersionID Mismatch
error due to local file mappings not being cleared correctly when native
RocksDB does a compaction resulting in the original SST file not being deleted.
This resulted in the DFS file UUID being reused for 2 SST files that were not
the same, leading the version ID mismatch.
We need a change that purges from the local file mappings any mapping that was
created by a version equal or greater than the one loaded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
