Hello All, We have a Spark cluster where driver and master are running on the same node. We are using Spark Standalone cluster manager. If the number of nodes (and the partitions) are increased, the same dataset that used to run to completion on lesser number of nodes is now giving an out of memory on the driver.
For example, a dataset that runs on 32 nodes with number of partitions set to 256 completes whereas the same dataset when run on 64 nodes with number of partitions as 512 gives an OOM on the driver side. >From what I read in the Spark documentation and other articles, following are the responsibilities of the driver/master. 1) create spark context 2) build DAG of operations 3) schedule tasks I am guessing that 1) and 2) should not change w.r.t number of nodes/partitions. So is it that since the driver has to keep track of lot more tasks, that it gives an OOM? What could be the possible reasons behind the driver-side OOM when the number of partitions are increased? Regards, Raghava.
