Hi, (redirecting to users as it has nothing to do with Spark project development)
Monitor jobs and stages using SparkListener and submit cleanup jobs where a condition holds. Jacek On 20 Jan 2017 3:57 a.m., "Keith Chapman" <keithgchap...@gmail.com> wrote: > Hi , > > Is it possible for an executor (or slave) to know when an actual job ends? > I'm running spark on a cluster (with yarn) and my workers create some > temporary files that I would like to clean up once the job ends. Is there a > way for the worker to detect that a job has finished? I tried doing it in > the JobProgressListener but it does not seem to work in a cluster. The > event is not triggered in the worker. > > Regards, > Keith. > > http://keith-chapman.com >
