Hi, This post might be a duplicate with updates from another one (by me), sorry in advance
I have an HDP 2.3 cluster running Spark 1.3.1 on 6 nodes (edge + master + 4 workers) Each worker has 8 cores and 40G of RAM available in Yarn That makes a total of 160GB and 32 cores I'm running a job with the following parameters : --master yarn-client --num-executors 12 (-> 3 / node) --executor-cores 2 --executor-memory 12G I don't know if it's optimal but it should run (right ?) However I end up with spark setting up 2 executors using 1 core & 6.2G each Plus, my job is doing a cartesian product so I end up with a pretty big DataFrame that inevitably ends on a GC exception... It used to run on HDP2.2 / Spark 1.2.1 but I can't find any way to run it now Any Idea ? Thanks a lot Cesar -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-1-on-Yarn-not-using-all-given-capacity-tp24955.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
