+1 as well for being able to submit jobs programmatically without using shell script.
we also experience issues of submitting jobs programmatically without using spark-submit. In fact, even in the Hadoop World, I rarely used "hadoop jar" to submit jobs in shell. On Wed, Jul 9, 2014 at 9:47 AM, Robert James <srobertja...@gmail.com> wrote: > +1 to be able to do anything via SparkConf/SparkContext. Our app > worked fine in Spark 0.9, but, after several days of wrestling with > uber jars and spark-submit, and so far failing to get Spark 1.0 > working, we'd like to go back to doing it ourself with SparkConf. > > As the previous poster said, a few scripts should be able to give us > the classpath and any other params we need, and be a lot more > transparent and debuggable. > > On 7/9/14, Surendranauth Hiraman <suren.hira...@velos.io> wrote: > > Are there any gaps beyond convenience and code/config separation in using > > spark-submit versus SparkConf/SparkContext if you are willing to set your > > own config? > > > > If there are any gaps, +1 on having parity within SparkConf/SparkContext > > where possible. In my use case, we launch our jobs programmatically. In > > theory, we could shell out to spark-submit but it's not the best option > for > > us. > > > > So far, we are only using Standalone Cluster mode, so I'm not > knowledgeable > > on the complexities of other modes, though. > > > > -Suren > > > > > > > > On Wed, Jul 9, 2014 at 8:20 AM, Koert Kuipers <ko...@tresata.com> wrote: > > > >> not sure I understand why unifying how you submit app for different > >> platforms and dynamic configuration cannot be part of SparkConf and > >> SparkContext? > >> > >> for classpath a simple script similar to "hadoop classpath" that shows > >> what needs to be added should be sufficient. > >> > >> on spark standalone I can launch a program just fine with just SparkConf > >> and SparkContext. not on yarn, so the spark-launch script must be doing > a > >> few things extra there I am missing... which makes things more difficult > >> because I am not sure its realistic to expect every application that > >> needs > >> to run something on spark to be launched using spark-submit. > >> On Jul 9, 2014 3:45 AM, "Patrick Wendell" <pwend...@gmail.com> wrote: > >> > >>> It fulfills a few different functions. The main one is giving users a > >>> way to inject Spark as a runtime dependency separately from their > >>> program and make sure they get exactly the right version of Spark. So > >>> a user can bundle an application and then use spark-submit to send it > >>> to different types of clusters (or using different versions of Spark). > >>> > >>> It also unifies the way you bundle and submit an app for Yarn, Mesos, > >>> etc... this was something that became very fragmented over time before > >>> this was added. > >>> > >>> Another feature is allowing users to set configuration values > >>> dynamically rather than compile them inside of their program. That's > >>> the one you mention here. You can choose to use this feature or not. > >>> If you know your configs are not going to change, then you don't need > >>> to set them with spark-submit. > >>> > >>> > >>> On Wed, Jul 9, 2014 at 10:22 AM, Robert James <srobertja...@gmail.com> > >>> wrote: > >>> > What is the purpose of spark-submit? Does it do anything outside of > >>> > the standard val conf = new SparkConf ... val sc = new SparkContext > >>> > ... ? > >>> > >> > > > > > > -- > > > > SUREN HIRAMAN, VP TECHNOLOGY > > Velos > > Accelerating Machine Learning > > > > 440 NINTH AVENUE, 11TH FLOOR > > NEW YORK, NY 10001 > > O: (917) 525-2466 ext. 105 > > F: 646.349.4063 > > E: suren.hiraman@v <suren.hira...@sociocast.com>elos.io > > W: www.velos.io > > >
