[
https://issues.apache.org/jira/browse/SPARK-49102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-49102.
----------------------------------
Resolution: Invalid
Resolving as Invalid — this is a usage/how-to question rather than a specific
Spark defect or actionable change. Usage questions are best directed to
[email protected] (https://spark.apache.org/community.html) or Stack
Overflow (tag apache-spark). Findings from triage: This is a
usage/how-does-it-work question (JIRA type "Question", 0 comments), not a Spark
defect or requested change. The user asks how the Executors-tab "Memory Used"
is calculated and why their custom SparkListener sum does not match. The
behavior is well-defined and confirmed in the current code:
AppStatusListener.updateExecutorMemoryDiskInfo
(AppStatusListener.scala:1196-1210) sets exec.memoryUsed by accumulating
SparkListenerBlockUpdated memory deltas (storage memory of cached blocks), with
usedOnHeap/usedOffHeap tracked separately; api.scala exposes this via
ExecutorSummary.memoryUsed /
Please reopen with a concrete reproducer or a specific proposed change if this
is actually a bug or an actionable improvement.
> Memory Used in Spark UI using the Custom spark listener
> -------------------------------------------------------
>
> Key: SPARK-49102
> URL: https://issues.apache.org/jira/browse/SPARK-49102
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 3.3.1
> Reporter: Bhargav M Gowda
> Priority: Major
>
> Hi, I was trying to write a custom spark listener to get the metrics which
> are available in Spark UI. I'm kinda lost at memory used part of the
> executors tab. How is it calculated when i tried to take sum of all on heap
> and off heap gave a very large number compared to the one in UI. Please guide
> me here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
