Set spark.*.retained* configs to 0 when the UI is disabled?

I was recently debugging an OOM exception one of my coworkers was struggling with and found that `SQLListener._stageIdToStageMetrics` was the culprit. The UI was disabled in this case, but stats were still accumulating for jobs, stages, and tasks. The job my coworker was running had over 40k tasks

Re: Set spark.*.retained* configs to 0 when the UI is disabled?

There are some APIs (SparkStatusTracker) that expose job and stage data even when the UI is disabled. I don't think tasks, or SQL stuff, are exposed without the UI though, and maybe the SQL listener doesn't even need to be installed in that case. (Similar for other listeners that don't do anything

Re: Set spark.*.retained* configs to 0 when the UI is disabled?

Thanks for the feedback Marcelo! Are you referring to SPARK-20421 and SPARK-18085 ? If I can lend a hand in this, just let me know. -- Sent from: http://apache-spark-developers-list.100

Re: Set spark.*.retained* configs to 0 when the UI is disabled?

On Fri, Oct 13, 2017 at 12:49 PM, Craig Ingram wrote: > Are you referring to SPARK-20421 > > and SPARK-18085 ? If I > can lend a hand in this, just let me know. Yes, I'm referring to that pr