Hi Fernando, Here's a <https://github.com/sigmoidanalytics/Test> simple log parser/analyser written in scala (you can run it without spark-shell/submit). https://github.com/sigmoidanalytics/Test
Basically to run a spark job without spark-submit or shell you need a build file <https://github.com/sigmoidanalytics/Test/blob/master/build.sbt> which will pull in all the dependecies, and the main program <https://github.com/sigmoidanalytics/Test/blob/master/src/main/scala/LogAnalyser.scala#L35> in which you will specify your cluster details while creating the SparkContext. Thanks Best Regards On Wed, Dec 31, 2014 at 10:54 PM, Fernando O. <fot...@gmail.com> wrote: > Before jumping into a sea of dependencies and bash files: > Does anyone have an example of how to run a spark job without using > spark-submit or shell ? > > On Tue, Dec 30, 2014 at 3:23 PM, Fernando O. <fot...@gmail.com> wrote: > >> Hi all, >> I'm investigating spark for a new project and I'm trying to use >> spark-jobserver because... I need to reuse and share RDDs and from what I >> read in the forum that's the "standard" :D >> >> Turns out that spark-jobserver doesn't seem to work on yarn, or at least >> it does not on 1.1.1 >> >> My config is spark 1.1.1 (moving to 1.2.0 soon), hadoop 2.6 (which seems >> compatible with 2.4 from spark point of view... at least I was able to run >> spark-submit and shell tasks both in yarn-client and yarn-cluster modes) >> >> >> >> >> going back to my original point, I did some changes in spark-jobserver >> and how I can submit a job but I get: >> >> .... >> [2014-12-30 18:20:19,769] INFO e.spark.deploy.yarn.Client [] >> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample] >> - Max mem capabililty of a single resource in this cluster 15000 >> [2014-12-30 18:20:19,770] INFO e.spark.deploy.yarn.Client [] >> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample] >> - Preparing Local resources >> [2014-12-30 18:20:20,041] INFO e.spark.deploy.yarn.Client [] >> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample] >> - Prepared Local resources Map(__spark__.jar -> resource { scheme: "file" >> port: -1 file: >> "/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar" >> } size: 343226 timestamp: 1416429031000 type: FILE visibility: PRIVATE) >> >> [...] >> >> [2014-12-30 18:20:20,139] INFO e.spark.deploy.yarn.Client [] >> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample] >> - Yarn AM launch context: >> [2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client [] >> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample] >> - class: org.apache.spark.deploy.yarn.ExecutorLauncher >> [2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client [] >> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample] >> - env: Map(CLASSPATH -> >> $PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:$PWD/__app__.jar:$PWD/*, >> SPARK_YARN_CACHE_FILES_FILE_SIZES -> 343226, SPARK_YARN_STAGING_DIR -> >> .sparkStaging/application_1419963137232_0001/, >> SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_USER -> ec2-user, >> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> >> 1416429031000, SPARK_YARN_CACHE_FILES -> >> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar#__spark__.jar) >> >> [...] >> >> [2014-12-30 18:03:04,474] INFO YarnClientSchedulerBackend [] >> [akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample] >> - Application report from ASM: >> appMasterRpcPort: -1 >> appStartTime: 1419962580444 >> yarnAppState: FAILED >> >> [2014-12-30 18:03:04,475] ERROR .jobserver.JobManagerActor [] >> [akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample] >> - Failed to create context ebac0153-spark.jobserver.WordCountExample, >> shutting down actor >> org.apache.spark.SparkException: Yarn application already ended,might be >> killed or not able to launch application master. >> at >> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApp(YarnClientSchedulerBackend.scala:117) >> at >> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:93) >> >> >> >> In the hadoop console I can get the detailed issue >> >> Diagnostics: File >> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar >> does not exist >> java.io.FileNotFoundException: File >> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar >> does not exist >> >> now... it seems like spark is actually use a file I used for launching >> the task in other nodes >> >> Can anyone point me in the right direction of where that might be being >> set? >> >> >
