Re: Shuffle TTLs

It’s a good suggestion, however I don’t think there is a mechanism for TTLs in notebooks and most things in notebooks might not be safe to recompute, unlike if we delete shuffle files. Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/

Re: Shuffle TTLs

Thanks for bringing this up. Wouldn't it be better for the notebooks to control when these DFs/RDDs expire so they can do fine granular control? On Wed, Oct 16, 2024 at 7:51 AM Holden Karau wrote: > Hi Spark Devs, > > So back in Spark 1.X we had shuffle TTLs, but they did not take into > account