Hi , Is it possible for an executor (or slave) to know when an actual job ends? I'm running spark on a cluster (with yarn) and my workers create some temporary files that I would like to clean up once the job ends. Is there a way for the worker to detect that a job has finished? I tried doing it in the JobProgressListener but it does not seem to work in a cluster. The event is not triggered in the worker.
Regards, Keith. http://keith-chapman.com
