Try running it like this:
sudo -u hdfs spark-submit --class org.apache.spark.examples.SparkPi
--deploy-mode cluster --master yarn
hdfs:///user/spark/spark-examples-1.2.0-cdh5.3.2-hadoop2.5.0-cdh5.3.2.jar 10
Caveats:
1) Make sure the permissions of /user/nick is 775 or 777.
2) No need for hostnam
I think the jar file has to be local. In HDFS is not supported yet in Spark.
See this answer:
http://stackoverflow.com/questions/28739729/spark-submit-not-working-when-application-jar-is-in-hdfs
> Date: Sun, 29 Mar 2015 22:34:46 -0700
> From: n.e.trav...@gmail.com
> To: user@spark.apache.org
> Sub
What happens when you do:
sc.textFile("hdfs://path/to/the_file.txt")
Thanks
Best Regards
On Mon, Mar 30, 2015 at 11:04 AM, Nick Travers
wrote:
> Hi List,
>
> I'm following this example here
> <
> https://github.com/databricks/learning-spark/tree/master/mini-complete-example
> >
> with the fol
