Re: Programatically running of the Spark Jobs.

2014-09-04 Thread Vicky Kak
I get this error when i run it from IDE *** Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED at org.apache.spark.scheduler.DAG

Re: Programatically running of the Spark Jobs.

2014-09-04 Thread Vicky Kak
I don't want to use YARN or Mesos, just trying the standalone spark cluster. We need a way to do seamless submission with the API which I don't see. To my surprise I was hit by this issue when i tried running the submit from another machine, it is crazy that I have to submit the job from the worked

Re: Programatically running of the Spark Jobs.

2014-09-04 Thread Guru Medasani
I am able to run Spark jobs and Spark Streaming jobs successfully via YARN on a CDH cluster. When you mean YARN isn’t quite there yet, you mean to submit the jobs programmatically? or just in general? On Sep 4, 2014, at 1:45 AM, Matt Chu wrote: > https://github.com/spark-jobserver/spark-jo

Re: Programatically running of the Spark Jobs.

2014-09-04 Thread ericacm
Ahh - that probably explains an issue I am seeing. I am a brand new user and I tried running the SimpleApp class that is on the Quick Start page (http://spark.apache.org/docs/latest/quick-start.html). When I use conf.setMaster("local") then I can run the class directly from my IDE. But when I tr

Re: Programatically running of the Spark Jobs.

2014-09-04 Thread Vicky Kak
er > > > > *From:* Matt Chu [mailto:m...@kabam.com] > *Sent:* Thursday, September 04, 2014 2:46 AM > *To:* Vicky Kak > *Cc:* user > *Subject:* Re: Programatically running of the Spark Jobs. > > > > https://github.com/spark-jobserver/spark-jobserver > > &

RE: Programatically running of the Spark Jobs.

2014-09-04 Thread Ruebenacker, Oliver A
Hello, Can this be used as a library from within another application? Thanks! Best, Oliver From: Matt Chu [mailto:m...@kabam.com] Sent: Thursday, September 04, 2014 2:46 AM To: Vicky Kak Cc: user Subject: Re: Programatically running of the Spark Jobs. https://github.com/spark

Re: Programatically running of the Spark Jobs.

2014-09-03 Thread Matt Chu
https://github.com/spark-jobserver/spark-jobserver Ooyala's Spark jobserver is the current de facto standard, IIUC. I just added it to our prototype stack, and will begin trying it out soon. Note that you can only do standalone or Mesos; YARN isn't quite there yet. (The repo just moved from https