Hi, Diana, You don’t need to use spark-distributed sbt
just download sbt from its official website and set your PATH to the right place Best, -- Nan Zhu On Monday, March 24, 2014 at 4:30 PM, Diana Carroll wrote: > Yeah, that's exactly what I did. Unfortunately it doesn't work: > > $SPARK_HOME/sbt/sbt package > awk: cmd. line:1: fatal: cannot open file `./project/build.properties' for > reading (No such file or directory) > Attempting to fetch sbt > /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or > directory > /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or > directory > Our attempt to download sbt locally to sbt/sbt-launch-.jar failed. Please > install sbt manually from http://www.scala-sbt.org/ > > > > > On Mon, Mar 24, 2014 at 4:25 PM, Ognen Duzlevski <og...@plainvanillagames.com > (mailto:og...@plainvanillagames.com)> wrote: > > You can use any sbt on your machine, including the one that comes with > > spark. For example, try: > > > > ~/path_to_spark/sbt/sbt compile > > ~/path_to_spark/sbt/sbt run <arguments> > > > > Or you can just add that to your PATH by: > > > > export $PATH=$PATH:~/path_to_spark/sbt > > > > To make it permanent, you can add it to your ~/.bashrc or ~/.bash_profile > > or ??? depending on the system you are using. If you are on Windows, sorry, > > I can't offer any help there ;) > > > > Ognen > > > > > > On 3/24/14, 3:16 PM, Diana Carroll wrote: > > > Thanks Ongen. > > > > > > Unfortunately I'm not able to follow your instructions either. In > > > particular: > > > > > > > > sbt compile > > > > sbt run <arguments if any> > > > > > > This doesn't work for me because there's no program on my path called > > > "sbt". The instructions in the Quick Start guide are specific that I > > > should call "$SPARK_HOME/sbt/sbt". I don't have any other executable on > > > my system called "sbt". > > > > > > Did you download and install sbt separately? In following the Quick > > > Start guide, that was not stated as a requirement, and I'm trying to run > > > through the guide word for word. > > > > > > Diana > > > > > > > > > On Mon, Mar 24, 2014 at 4:12 PM, Ognen Duzlevski > > > <og...@plainvanillagames.com (mailto:og...@plainvanillagames.com)> wrote: > > > > Diana, > > > > > > > > Anywhere on the filesystem you have read/write access (you need not be > > > > in your spark home directory): > > > > > > > > mkdir myproject > > > > cd myproject > > > > mkdir project > > > > mkdir target > > > > mkdir -p src/main/scala > > > > cp $mypath/$mymysource.scala src/main/scala/ > > > > cp $mypath/myproject.sbt . > > > > > > > > Make sure that myproject.sbt has the following in it: > > > > > > > > name := "I NEED A NAME!" > > > > > > > > version := "I NEED A VERSION!" > > > > > > > > scalaVersion := "2.10.3" > > > > > > > > libraryDependencies += "org.apache.spark" % "spark-core_2.10" % > > > > "0.9.0-incubating" > > > > > > > > If you will be using Hadoop/HDFS functionality you will need the below > > > > line also > > > > > > > > libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.2.0" > > > > > > > > The above assumes you are using Spark 0.9 and Scala 2.10.3. If you are > > > > using 0.8.1 - adjust appropriately. > > > > > > > > That's it. Now you can do > > > > > > > > sbt compile > > > > sbt run <arguments if any> > > > > > > > > You can also do > > > > sbt package to produce a jar file of your code which you can then add > > > > to the SparkContext at runtime. > > > > > > > > In a more complicated project you may need to have a bit more involved > > > > hierarchy like com.github.dianacarroll which will then translate to > > > > src/main/scala/com/github/dianacarroll/ where you can put your multiple > > > > .scala files which will then have to be a part of a package > > > > com.github.dianacarroll (you can just put that as your first line in > > > > each of these scala files). I am new to Java/Scala so this is how I do > > > > it. More educated Java/Scala programmers may tell you otherwise ;) > > > > > > > > You can get more complicated with the sbt project subrirectory but you > > > > can read independently about sbt and what it can do, above is the bare > > > > minimum. > > > > > > > > Let me know if that helped. > > > > Ognen > > > > > > > > > > > > On 3/24/14, 2:44 PM, Diana Carroll wrote: > > > > > Has anyone successfully followed the instructions on the Quick Start > > > > > page of the Spark home page to run a "standalone" Scala application? > > > > > I can't, and I figure I must be missing something obvious! > > > > > > > > > > I'm trying to follow the instructions here as close to "word for > > > > > word" as possible: > > > > > http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala > > > > > > > > > > 1. The instructions don't say what directory to create my test > > > > > application in, but later I'm instructed to run "sbt/sbt" so I > > > > > conclude that my working directory must be $SPARK_HOME. (Temporarily > > > > > ignoring that it is a little weird to be working directly in the > > > > > Spark distro.) > > > > > > > > > > 2. Create $SPARK_HOME/mysparktest/src/main/scala/SimpleApp.scala. > > > > > Copy&paste in the code from the instructions exactly, replacing > > > > > YOUR_SPARK_HOME with my spark home path. > > > > > > > > > > 3. Create $SPARK_HOME/mysparktest/simple.sbt. Copy&paste in the sbt > > > > > file from the instructions > > > > > > > > > > 4. From the $SPARK_HOME I run "sbt/sbt package". It runs through > > > > > the ENTIRE Spark project! This takes several minutes, and at the > > > > > end, it says "Done packaging". unfortunately, there's nothing in the > > > > > $SPARK_HOME/mysparktest/ folder other than what I already had there. > > > > > > > > > > (Just for fun, I also did what I thought was more logical, which is > > > > > set my working directory to $SPARK_HOME/mysparktest, and but > > > > > $SPARK_HOME/sbt/sbt package, but that was even less successful: I got > > > > > an error: > > > > > awk: cmd. line:1: fatal: cannot open file > > > > > `./project/build.properties' for reading (No such file or directory) > > > > > Attempting to fetch sbt > > > > > /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or > > > > > directory > > > > > /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or > > > > > directory > > > > > Our attempt to download sbt locally to sbt/sbt-launch-.jar failed. > > > > > Please install sbt manually from http://www.scala-sbt.org/ > > > > > > > > > > > > > > > So, help? I'm sure these instructions work because people are > > > > > following them every day, but I can't tell what they are supposed to > > > > > do. > > > > > > > > > > Thanks! > > > > > Diana > > > > > > > > > > > -- “A distributed system is one in which the failure of a computer you > > didn’t even know existed can render your own computer unusable” -- Leslie > > Lamport
