Hi Eric, A few links, in case they can be useful for your troubleshooting:
The Spark Web UI is documented in Spark 3.x documentation, although you can use most of it for Spark 2.4 too: https://spark.apache.org/docs/latest/web-ui.html Spark memory management is documented at https://spark.apache.org/docs/latest/tuning.html#memory-management-overview Additional resource: see also this diagram https://canali.web.cern.ch/docs/SparkExecutorMemory.png and https://db-blog.web.cern.ch/blog/luca-canali/2020-08-spark3-memory-monitoring Best, Luca From: Eric Beabes <mailinglist...@gmail.com> Sent: Wednesday, January 6, 2021 00:20 To: spark-user <user@spark.apache.org> Subject: Understanding Executors UI [image.png] Not sure if this image will go through. (Never sent an email to this mailing list with an image). I am trying to understand this 'Executors' UI in Spark 2.4. I have a Stateful Structured Streaming job with 'State timeout' set to 10 minutes. When the load on the system is low a message gets written to Kafka immediately after the State times out BUT under heavy load it takes over 40 minutes to get a message on the output topic. Trying to debug this issue & see if performance can be improved. Questions: 1) I am requesting 3.2 TB of memory but it seems the job keeps using only 598.5 GB as per the values in 'Storage Memory' as well as 'On Heap Storage Memory'. Wondering if this is a Cluster issue OR am I not setting values correctly? 2) Where can I find documentation to understand different 'Tabs' in the Spark UI? (Sorry, Googling didn't help. I will keep searching.) Any pointers would be appreciated. Thanks.
