Hi, Sorry this is probably a silly question but I have a standard CDH 5.4.2 config with Spark 1.3 and I'm trying to setup Spark dynamic allocation which was introduced in CDH 5.4.x and Spark 1.2.
According to the doc <https://spark.apache.org/docs/1.2.0/job-scheduling.html#dynamic-resource-allocation> I should set "spark.dynamicAllocation.enabled" to true but I cannot find this parameter in CM in the spark config section. After checking in the top search field in CM, it was found in the Hive section : "HiveServer2 Default Group" was specified next to it and ticked. Is this OK and i can assume it's enabled or should it mention "gateway/spark default group" instead of hiveServer2? The same goes with other related spark dynamic allocation parameters such as "spark.dynamicAllocation.minExecutors" (set to 1), and "spark.dynamicAllocation.initialExecutors" (set to 1) However I cannot find "spark.dynamicAllocation.maxExecutors" (Should i add this one to a safety valve ?) As things stand I can see it is not active as I tested a long spark job that was constantly using the default 2 executors in Spark-shell. I assume this is partly because a number of other settings mentioned in the doc should be setup like "yarn.nodemanager.aux-services" and all related parameters that should be added to "yarn-site.xml" (safety valve?) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-dynamic-allocation-config-tp25266.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
