Alonso, I could totally be misunderstanding something or missing a piece of the puzzle however remove .setMaster. If you do that it will run with whatever the CDH VM is setup for which in the out of the box default case is YARN and Client.
val sparkConf = new SparkConf().setAppName(“Some App thingy thing”) From the Spark 1.6.0 Scala API Documentation: https://spark.apache.org/docs/1.6.0/api/scala/index.html#org.apache.spark.SparkConf “ Configuration for a Spark application. Used to set various Spark parameters as key-value pairs. Most of the time, you would create a SparkConf object with new SparkConf(), which will load values from any spark.* Java system properties set in your application as well. In this case, parameters you set directly on the SparkConf object take priority over system properties. For unit tests, you can also call new SparkConf(false) to skip loading external settings and get the same configuration no matter what the system properties are. All setter methods in this class support chaining. For example, you can write new SparkConf().setMaster("local").setAppName("My app"). Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime. “ David Newberger From: Alonso Isidoro Roman [mailto:alons...@gmail.com] Sent: Friday, June 3, 2016 10:37 AM To: David Newberger Cc: user@spark.apache.org Subject: Re: About a problem running a spark job in a cdh-5.7.0 vmware image. Thank you David, so, i would have to change the way that i am creating SparkConf object, isn't? I can see in this link<http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_running_spark_on_yarn.html#concept_ysw_lnp_h5> that the way to run a spark job using YARN is using this kind of command: spark-submit --class org.apache.spark.examples.SparkPi --master yarn \ --deploy-mode client SPARK_HOME/lib/spark-examples.jar 10 Can i use this way programmatically? maybe changing setMaster? to something like setMaster("yarn:quickstart.cloudera:8032")? I have seen the port in this guide: http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_ports_cdh5.html
