neilramaswamy opened a new pull request, #48862: URL: https://github.com/apache/spark/pull/48862
### What changes were proposed in this pull request? These changes make the following changes to metrics in TWS: - `allUpdatesTimeMs` now captures the time it takes to process all the new data with the user's stateful processor. - `timerProcessingTimeMs` was added to capture the time it takes to process all the user's timers. - `allRemovalsTimeMs` now captures the time it takes to do TTL cleanup at the end of a micro-batch. - `commitTimeMs` now captures _only_ the time it takes to commit the state, not the TTL cleanup. With these metrics, a user can have a fairly clear picture of where time is being spent in a micro-batch that uses TWS: ### Why are the changes needed? The metrics today misrepresent what they're actually measuring. ### Does this PR introduce _any_ user-facing change? Yes. Metrics for TWS are changing. However, since TWS is `private[sql]`, this shouldn't impact any real users. ### How was this patch tested? We don't have any way to test these metrics in _any_ stateful operator for streaming today. ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
