That’s really useful, thanks.
From: Andrew Ash [mailto:and...@andrewash.com]
Sent: 09 January 2015 22:42
To: England, Michael (IT/UK)
Cc: raghavendra.pan...@gmail.com; user
Subject: Re: Cleaning up spark.local.dir automatically
That's a worker setting which cleans up the files left behi
:
> Thanks, I imagine this will kill any cached RDDs if their files are
> beyond the ttl?
>
>
>
> Thanks
>
>
>
>
>
> *From:* Raghavendra Pandey [mailto:raghavendra.pan...@gmail.com]
> *Sent:* 09 January 2015 15:29
> *To:* England, Michael (I
Thanks, I imagine this will kill any cached RDDs if their files are beyond the
ttl?
Thanks
From: Raghavendra Pandey [mailto:raghavendra.pan...@gmail.com]
Sent: 09 January 2015 15:29
To: England, Michael (IT/UK); user@spark.apache.org
Subject: Re: Cleaning up spark.local.dir automatically
You
You may like to look at spark.cleaner.ttl configuration which is infinite
by default. Spark has that configuration to delete temp files time to time.
On Fri Jan 09 2015 at 8:34:10 PM wrote:
> Hi,
>
>
>
> Is there a way of automatically cleaning up the spark.local.dir after a
> job has been run?
