neilramaswamy opened a new pull request, #48862:
URL: https://github.com/apache/spark/pull/48862

   ### What changes were proposed in this pull request?
   
   These changes make the following changes to metrics in TWS:
   
   - `allUpdatesTimeMs` now captures the time it takes to process all the new 
data with the user's stateful processor.
   - `timerProcessingTimeMs` was added to capture the time it takes to process 
all the user's timers.
   - `allRemovalsTimeMs` now captures the time it takes to do TTL cleanup at 
the end of a micro-batch.
   - `commitTimeMs` now captures _only_ the time it takes to commit the state, 
not the TTL cleanup.
   
   With these metrics, a user can have a fairly clear picture of where time is 
being spent in a micro-batch that uses TWS:
   
   
   ### Why are the changes needed?
   
   The metrics today misrepresent what they're actually measuring.
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Metrics for TWS are changing. However, since TWS is `private[sql]`, 
this shouldn't impact any real users.
   
   
   ### How was this patch tested?
   
   We don't have any way to test these metrics in _any_ stateful operator for 
streaming today.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to