Hello everyone I have a question about Shuffle Spills. From the introduction to amplab spark internals each task output could be saved to disk for 'redundancy'
if I set spark.shuffle.spill to false would this behavior be eliminated and make it in a way that it will never spill to disk ? Thank you -- LinkedIn: http://linkedin.com/in/fmilo Twitter: @fabmilo Github: http://github.com/Mistobaan/ ----------------------- Simplicity, consistency, and repetition - that's how you get through. (Jack Welch) Perfection must be reached by degrees; she requires the slow hand of time (Voltaire) The best way to predict the future is to invent it (Alan Kay)
