Hi Andrew,
Thanks for the response. I believe I have HDFS set up correctly: all my
slaves can access it fine, and I can access the files I'm storing there
with ~/persistent-hdfs/bin/hadoop -ls, etc. However, when I do spark-submit
--master local, I still get the following error:
14/07/18 16:20:58
Hi Chris,
Did you ever figure this out? It should just work provided that your HDFS
is set up correctly. If you don't call setMaster, it actually uses the
spark://[master-node-ip]:7077 by default (this is configured in your
conf/spark-env.sh). However, even if you use a local master, it should
sti
You can try the following in the spark-shell:
1. Run it in *Clustermode* by going inside the spark directory:
$ SPARK_MASTER=spark://masterip:7077 ./bin/spark-shell
val textFile = sc.textFile("hdfs://masterip/data/blah.csv")
textFile.take(10).foreach(println)
2. Now try running in *Localmode:
