Hi Jacek, I've looked at SparkListener and tried it, I see it getting fired on the master but I don't see it getting fired on the workers in a cluster.
Regards, Keith. http://keith-chapman.com On Fri, Jan 20, 2017 at 11:09 AM, Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > (redirecting to users as it has nothing to do with Spark project > development) > > Monitor jobs and stages using SparkListener and submit cleanup jobs where > a condition holds. > > Jacek > > On 20 Jan 2017 3:57 a.m., "Keith Chapman" <keithgchap...@gmail.com> wrote: > >> Hi , >> >> Is it possible for an executor (or slave) to know when an actual job >> ends? I'm running spark on a cluster (with yarn) and my workers create some >> temporary files that I would like to clean up once the job ends. Is there a >> way for the worker to detect that a job has finished? I tried doing it in >> the JobProgressListener but it does not seem to work in a cluster. The >> event is not triggered in the worker. >> >> Regards, >> Keith. >> >> http://keith-chapman.com >> >
