[
https://issues.apache.org/jira/browse/SPARK-47917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-47917.
----------------------------------
Resolution: Invalid
Resolving as Invalid — this is a usage/how-to question rather than a specific
Spark defect or actionable change. Usage questions are best directed to
[email protected] (https://spark.apache.org/community.html) or Stack
Overflow (tag apache-spark). Findings from triage: The ticket is JIRA type
"Question" and its body is an explicit how-to request: the reporter describes
their organization's own external cost-segregation system and ends with "Do you
have any suggestions on how I can use the Spark event system?" There is no bug,
no reproducer, no expected-vs-actual behavior, no proposed code change, and 0
comments. It asks for advice on consuming already-emitted Spark metrics
(tracking per-task execution history across AQE stage-id changes to compute a
failure/goodput cost ratio) — a user-side analytics question, not a Spark
defect or a specific actionable impr
Please reopen with a concrete reproducer or a specific proposed change if this
is actually a bug or an actionable improvement.
> Accounting the impact of failures in spark jobs
> -----------------------------------------------
>
> Key: SPARK-47917
> URL: https://issues.apache.org/jira/browse/SPARK-47917
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 3.5.1
> Reporter: Faiz Halde
> Priority: Minor
>
> Hello,
>
> In my organization, we have an accounting system for spark jobs that uses the
> task execution time to determine how much time a spark job uses the executors
> for and we use it as a way to segregate cost. We sum all the task times per
> job and apply proportions. Our clusters follow a 1 task per core model & this
> works well.
>
> A job goes through several failures during its run, due to executor failure,
> node failure ( spot interruptions ), and spark retries tasks & sometimes
> entire stages.
>
> We now want to account for this failure and determine what % of a job's total
> task time is due to these retries. Basically, if a job with failures &
> retries has a total task time of X, there is a X' representing the goodput of
> this job – i.e. a hypothetical run of the job with 0 failures & retries. In
> this case, ( X-X' ) / X quantifies the cost of failures.
>
> This form of accounting requires tracking execution history of each task i.e.
> tasks that compute the same logical partition of some RDD. This was quite
> easy with AQE disabled as stage ids never changed, but with AQE enabled
> that's no longer the case.
>
> Do you have any suggestions on how I can use the Spark event system?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
