I have two nodes with 96G ram 16 cores, my setup is as follows:
conf = (SparkConf()
.setMaster("yarn-cluster")
.set("spark.executor.memory", "30G")
.set("spark.cores.max", 32)
.set("spark.executor.instances", 2)
.set("spark.executor.cores", 8)
.set("spark.akka.timeout", 10000)
.set("spark.akka.askTimeout", 100)
.set("spark.akka.frameSize", 500)
.set("spark.cleaner.ttl", 86400)
.set("spark.tast.maxFailures", 16)
.set("spark.worker.timeout", 150)
thanks a lot,
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Any-issues-with-repartition-tp13462p15674.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
