Considering posting the question on vendor's forum. HDP 2.3 comes with Spark 1.4 if I remember correctly.
On Tue, Oct 6, 2015 at 9:05 AM, czoo <[email protected]> wrote: > Hi, > > This post might be a duplicate with updates from another one (by me), sorry > in advance > > I have an HDP 2.3 cluster running Spark 1.3.1 on 6 nodes (edge + master + 4 > workers) > Each worker has 8 cores and 40G of RAM available in Yarn > > That makes a total of 160GB and 32 cores > > I'm running a job with the following parameters : > --master yarn-client > --num-executors 12 (-> 3 / node) > --executor-cores 2 > --executor-memory 12G > > I don't know if it's optimal but it should run (right ?) > > However I end up with spark setting up 2 executors using 1 core & 6.2G each > > Plus, my job is doing a cartesian product so I end up with a pretty big > DataFrame that inevitably ends on a GC exception... > It used to run on HDP2.2 / Spark 1.2.1 but I can't find any way to run it > now > > Any Idea ? > > Thanks a lot > > Cesar > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-1-on-Yarn-not-using-all-given-capacity-tp24955.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
