zecookiez commented on code in PR #50123: URL: https://github.com/apache/spark/pull/50123#discussion_r2023905676
########## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala: ########## @@ -55,6 +56,39 @@ private case class GetLocation(storeId: StateStoreProviderId) private case class DeactivateInstances(runId: UUID) extends StateStoreCoordinatorMessage +/** + * This message is used to report a state store has just finished uploading a snapshot, + * along with the timestamp in milliseconds and the snapshot version. + */ +private case class ReportSnapshotUploaded( + providerId: StateStoreProviderId, + version: Long, + timestamp: Long) Review Comment: I have this right now to reflect the actual upload timestamp more closely in case RPC calls get delayed, but since we also report the state store version loaded from checkpoints at the loading phase I figured this would be helpful in distinguishing between: - reporting the snapshot version just uploaded vs - reporting the snapshot version just loaded but was uploaded at a previous unknown time if this case is not necessary (since we have another way to detect cases from avail now) I can remove this if needed :+1: EDIT: removed this as this seemed to be unnecessary when verified in testing, timestamps reflect current time of upload / loading snapshots -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
