I believe it’s a spark Ui issue which do not display correct value. I believe it is resolved for spark 3.0.
Thanks Amit On Fri, Jan 8, 2021 at 4:00 PM Luca Canali <luca.can...@cern.ch> wrote: > You report 'Storage Memory': 3.3TB/ 598.5 GB -> The first number is the > memory used for storage, the second one is the available memory (for > storage) in the unified memory pool. > > The used memory shown in your webui snippet is indeed quite high (higher > than the available memory!? ), you can probably profit by drilling down on > that to understand better what is happening. > > For example look at the details per executor (the numbers you reported are > aggregated values), then also look at the “storage tab” for a list of > cached RDDs with details. > > In case, Spark 3.0 has improved memory instrumentation and improved > instrumentation for streaming, so you can you profit from testing there too. > > > > > > *From:* Eric Beabes <mailinglist...@gmail.com> > *Sent:* Friday, January 8, 2021 04:23 > *To:* Luca Canali <luca.can...@cern.ch> > *Cc:* spark-user <user@spark.apache.org> > *Subject:* Re: Understanding Executors UI > > > > So when I see this for 'Storage Memory': *3.3TB/ 598.5 GB* *- it's > telling me that Spark is using 3.3 TB of memory & 598.5 GB is used for > caching data, correct?* What I am surprised about is that these numbers > don't change at all throughout the day even though the load on the system > is low after 5pm PST. > > > > I would expect the "Memory used" to be lower than 3.3Tb after 5pm PST. > > > > Does Spark 3.0 do a better job of memory management? Wondering if > upgrading to Spark 3.0 would improve performance? > > > > > > On Wed, Jan 6, 2021 at 2:29 PM Luca Canali <luca.can...@cern.ch> wrote: > > Hi Eric, > > > > A few links, in case they can be useful for your troubleshooting: > > > > The Spark Web UI is documented in Spark 3.x documentation, although you > can use most of it for Spark 2.4 too: > https://spark.apache.org/docs/latest/web-ui.html > > > > Spark memory management is documented at > https://spark.apache.org/docs/latest/tuning.html#memory-management-overview > > > Additional resource: see also this diagram > https://canali.web.cern.ch/docs/SparkExecutorMemory.png and > https://db-blog.web.cern.ch/blog/luca-canali/2020-08-spark3-memory-monitoring > > > > Best, > > Luca > > > > *From:* Eric Beabes <mailinglist...@gmail.com> > *Sent:* Wednesday, January 6, 2021 00:20 > *To:* spark-user <user@spark.apache.org> > *Subject:* Understanding Executors UI > > > > [image: image.png] > > > > > > Not sure if this image will go through. (Never sent an email to this > mailing list with an image). > > > > I am trying to understand this 'Executors' UI in Spark 2.4. I have a > Stateful Structured Streaming job with 'State timeout' set to 10 minutes. > When the load on the system is low a message gets written to Kafka > immediately after the State times out BUT under heavy load it takes over 40 > minutes to get a message on the output topic. Trying to debug this issue & > see if performance can be improved. > > > > Questions: > > > > 1) I am requesting 3.2 TB of memory but it seems the job keeps using only > 598.5 GB as per the values in 'Storage Memory' as well as 'On Heap Storage > Memory'. Wondering if this is a Cluster issue OR am I not setting values > correctly? > > 2) Where can I find documentation to understand different 'Tabs' in the > Spark UI? (Sorry, Googling didn't help. I will keep searching.) > > > > Any pointers would be appreciated. Thanks. > > > >
