Hello,
we send a lot of small jobs to Spark (up to 500 in a second). When profiling
I see Throwable.getStackTrace() in the top of memory profiler which is
caused by SparkContext.getCallSite - this is memory consuming.
we use Java API, I tried to call SparkContext.setCallSite("-") before
submitting a job. After that I see less calls of Throwable.getStackTrace()
but some of them are still there. I think it is because
SparkContext.setCallSite sets thread local variable and in some cases
SparkContext.getCallSite is called from a Spark thread.
Is there any way to disable CallSite completely? We dont need this
information and we want to improve performance by disabling CallSite.
Thank You
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-getCallSite-is-in-the-top-of-profiler-by-memory-allocation-tp22716.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
