I'm doing coalesce with shuffle, cache and then do thousands of iterations.
I noticed that sometimes Spark would for no particular reason perform partial coalesce again after running for a long time - and there was no exception or failure on the worker's part. Why is this happening?
