Spark 1.0.0 introduced the ContextCleaner to replace the MetadataCleaner
API for this exact issue. The ContextClenaer automatically cleans up your
RDD metadata once the RDD gets garbage collected on the driver.
On Wed, Jul 9, 2014 at 3:31 AM, wrote:
> Hi,
>
>
>
> Does anyone know if it is possi
Hi,
Does anyone know if it is possible to call the MetadaCleaner on demand? i.e.
rather than set spark.cleaner.ttl and have this run periodically, I'd like to
run it on demand. The problem with periodic cleaning is that it can remove rdd
that we still require (some calcs are short, others very
