Hi Fernando, There is a limit of 2GB on blocks for shuffle, since you say the job fails while doing shuffle of 200GB data, it might be due to this. These links give more idea about this: http://apache-spark-developers-list.1001551.n3.nabble.com/Re-2GB-limit-for-partitions-td10435.html https://issues.apache.org/jira/browse/SPARK-5928
Thanks, Pratyush -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
