I believe I may be able to reproduce this now, it seems like it may be
something to do with many jobs at once:
Spark 2.3.1
> spark-shell --conf spark.ui.retainedJobs=1
scala> import scala.concurrent._
scala> import scala.concurrent.ExecutionContext.Implicits.global
scala> for (i <- 0 until 50000) { Future { println(sc.parallelize(0 until
i).collect.length) } }
On Mon, Oct 22, 2018 at 11:25 AM Marcelo Vanzin <van...@cloudera.com> wrote:
> Just tried on 2.3.2 and worked fine for me. UI had a single job and a
> single stage (+ the tasks related to that single stage), same thing in
> memory (checked with jvisualvm).
>
> On Sat, Oct 20, 2018 at 6:45 PM Marcelo Vanzin <van...@cloudera.com>
> wrote:
> >
> > On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown
> > <patrick.barry.br...@gmail.com> wrote:
> > > I recently upgraded to spark 2.3.1 I have had these same settings in
> my spark submit script, which worked on 2.0.2, and according to the
> documentation appear to not have changed:
> > >
> > > spark.ui.retainedTasks=1
> > > spark.ui.retainedStages=1
> > > spark.ui.retainedJobs=1
> >
> > I tried that locally on the current master and it seems to be working.
> > I don't have 2.3 easily in front of me right now, but will take a look
> > Monday.
> >
> > --
> > Marcelo
>
>
>
> --
> Marcelo
>
