I have made a little progress - by downloading a prebuilt version of Spark I can call spark-shell.cmd and bring up a spark shell. In the shell things run. Next I go to my development environment and try to run JavaWordCount i try -Dspark.master=spark://local[*]:55519 -Dspark.master=spark://Asterix:7707 (Asterix is my machine) end many other combinations
I can hit a web page http://asterix:4040/environment/ and see many details about a presumably running spark master but the incantation to allow a simple job like JavaWordCount is escaping me Oh yes - I am running on Windows 8 Any help would be appreciated starting with how do I know a spark master is running and what port it is on On Sat, Aug 16, 2014 at 7:33 PM, Manu Suryavansh <[email protected] > wrote: > Hi, > > I have built spark-1.0.0 on Windows using Java 7/8 and I have been able to > run several examples - here are my notes - > http://ml-nlp-ir.blogspot.com/2014/04/building-spark-on-windows-and-cloudera.html > on how to build from source and run examples in spark shell. > > > Regards, > Manu > > > On Sat, Aug 16, 2014 at 12:14 PM, Steve Lewis <[email protected]> > wrote: > >> I want to look at porting a Hadoop problem to Spark - eventually I want >> to run on a Hadoop 2.0 cluster but while I am learning and porting I want >> to run small problems in my windows box. >> I installed scala and sbt. >> I download Spark and in the spark directory can say >> mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package >> which succeeds >> I tried >> sbt/sbt assembly >> which fails with errors >> >> In the documentation >> <https://spark.apache.org/docs/latest/spark-standalone.html>it says >> >> *Note:* The launch scripts do not currently support Windows. To run a >> Spark cluster on Windows, start the master and workers by hand. >> with no indication of how to do this. >> >> I can build and run samples (say JavaWordCount) to the point where they >> fail because a master cannot be found (none is running) >> >> I want to know how to get a spark master and a slave or two running on my >> windows box so I can look at the samples and start playing with Spark >> >> Does anyone have a windows instance running?? >> Please DON'T SAY I SHOULD RUN LINUX! if it is supposed to work on >> windows someone should have tested it and be willing to state how. >> >> >> >> > > > -- > Manu Suryavansh > -- Steven M. Lewis PhD 4221 105th Ave NE Kirkland, WA 98033 206-384-1340 (cell) Skype lordjoe_com
