zecookiez commented on code in PR #50123:
URL: https://github.com/apache/spark/pull/50123#discussion_r2006503821
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -1472,6 +1475,8 @@ class RocksDB(
log"Current lineage: ${MDC(LogKeys.LINEAGE, lineageManager)}")
// Compare and update with the version that was just uploaded.
lastUploadedSnapshotVersion.updateAndGet(v => Math.max(snapshot.version,
v))
+ // Report snapshot upload event to the coordinator.
+ eventListener.foreach(_.reportSnapshotUploaded(snapshot.version))
Review Comment:
Yeah this is tricky, because the coordinator needs a way to distinguish
between the two cases {all partitions are lagging} and {messages are disabled},
and the checks needed for this are different between RocksDB and HDFS stores
because of the changelog config.
I'll add a short comment to explain the nuances behind it :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
