Re: spark.cleaner.ttl for 1.4.1

2015-11-30 Thread Josh Rosen
AFAIK the ContextCleaner should perform all of the cleaning *as long as garbage collection is performed frequently enough on the driver*. See https://issues.apache.org/jira/browse/SPARK-7689 and https://github.com/apache/spark/pull/6220#issuecomment-102950055 for discussion of this technicality. O

spark.cleaner.ttl for 1.4.1

2015-11-30 Thread Michal Čizmazia
Does *spark.cleaner.ttl *still need to be used for Spark *1.4.1 *long-running streaming jobs? Or does *ContextCleaner* alone do all the cleaning?