dylanwong250 commented on code in PR #52202:
URL: https://github.com/apache/spark/pull/52202#discussion_r2323436252
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##########
@@ -1032,7 +1031,7 @@ case class RocksDBCheckpointMetadata(
/** Helper class for [[RocksDBCheckpointMetadata]] */
object RocksDBCheckpointMetadata {
- val VERSION = SQLConf.get.stateStoreCheckpointFormatVersion
+ val VERSION = 1
Review Comment:
I want to point out this change. I believe using `val VERSION =
SQLConf.get.stateStoreCheckpointFormatVersion` is incorrect in this case and I
was seeing a few test failures. The `VERSION` refers to the checkpoint format
version not the metadata format. Additionally, since this is inside an object
as a val it will be instantiated once and is not the current value of
SQLConf.get.stateStoreCheckpointFormatVersion. I think this VERSION may be
shared incorrectly also.
I also experimented with having VERSION equal to the current value. I ran
into a few issues with the maintenance threads having a different version than
the streaming query they were ran in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
