zecookiez commented on code in PR #49816:
URL: https://github.com/apache/spark/pull/49816#discussion_r1953571050
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -320,11 +322,22 @@ trait StateStoreWriter
* the driver after this SparkPlan has been executed and metrics have been
updated.
*/
def getProgress(): StateOperatorProgress = {
+ // Still publish the instance metrics that are marked as unpopulated,
+ // as they represent state store instances that never uploaded a snapshot.
+ val instanceMetrics = stateStoreInstanceMetrics.map { entry =>
+ entry._1 -> (if (longMetric(entry._1).isZero) -1L else
longMetric(entry._1).value)
+ }
+ // Only keep the smallest N instance metrics to report to driver.
Review Comment:
This makes sense, I've just added some configuration options to the instance
metrics class to help generalize this to some more custom metrics that we may
want to add in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
