Zhen Wang created SPARK-59487:
---------------------------------
Summary: Add task-level spans to correlate Spark tasks with
executor profiling samples
Key: SPARK-59487
URL: https://issues.apache.org/jira/browse/SPARK-59487
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Zhen Wang
The executor profiler records samples associated with executor threads, but the
profiling output currently lacks an explicit mapping between Spark task
attempts and those threads over time. Since each executor thread can run
multiple tasks sequentially, the thread identity alone is insufficient to
determine which task produced a profiling sample. This makes it difficult to
connect a slow task observed in the Spark UI with its corresponding profile.
Async-profiler introduced the Span API in [PR
#1755|https://github.com/async-profiler/async-profiler/pull/1755]. A span
records an execution interval on a thread with a user-defined tag, providing
the thread and time information needed to associate profiling samples with a
task attempt.
Add task-level spans to {{{}ProfilerExecutorPlugin{}}}, recording each task
attempt’s interval with its stage, partition, task attempt identifiers, and
completion status. These spans establish the task-to-thread mapping within the
same JFR recording, allowing users to identify and analyze samples belonging to
a specific task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
