Haotian Sun created SPARK-59993:
-----------------------------------
Summary: Collect scalar pandas UDF phase timing metrics
Key: SPARK-59993
URL: https://issues.apache.org/jira/browse/SPARK-59993
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Haotian Sun
Add task-local timing accumulation in the Python worker and report input
conversion, UDF execution, and output conversion durations for scalar pandas
UDFs, together with task and batch coverage counters.
Decode numeric worker reports into a generic map and register the new SQL
metrics so their values are available in the Spark SQL UI. Keep lifecycle
timing calculations and metric updates in the Python runner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
