zecookiez commented on code in PR #50030: URL: https://github.com/apache/spark/pull/50030#discussion_r1970205725
########## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala: ########## @@ -419,6 +432,10 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with private val loadedMapCacheHitCount: LongAdder = new LongAdder private val loadedMapCacheMissCount: LongAdder = new LongAdder + // This is updated when the maintenance task writes the snapshot file and read by the task Review Comment: I may be wrong, but the only way for HDFS to execute `writeSnapshot` and make a snapshot upload is from the doMaintenance task. Is it also possible for the task thread to go through that path too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
