when i write a general spark application i use SparkConf/SparkContext, not Client.scala for Yarn
On Wed, Jul 9, 2014 at 9:39 PM, Andrew Or <and...@databricks.com> wrote: > I don't see why using SparkSubmit.scala as your entry point would be any > different, because all that does is invoke the main class of Client.scala > (e.g. for Yarn) after setting up all the class paths and configuration > options. (Though I haven't tried this myself) > > > 2014-07-09 9:40 GMT-07:00 Ron Gonzalez <zlgonza...@yahoo.com>: > > I am able to use Client.scala or LauncherExecutor.scala as my programmatic >> entry point for Yarn. >> >> Thanks, >> Ron >> >> Sent from my iPad >> >> On Jul 9, 2014, at 7:14 AM, Jerry Lam <chiling...@gmail.com> wrote: >> >> +1 as well for being able to submit jobs programmatically without using >> shell script. >> >> we also experience issues of submitting jobs programmatically without >> using spark-submit. In fact, even in the Hadoop World, I rarely used >> "hadoop jar" to submit jobs in shell. >> >> >> >> On Wed, Jul 9, 2014 at 9:47 AM, Robert James <srobertja...@gmail.com> >> wrote: >> >>> +1 to be able to do anything via SparkConf/SparkContext. Our app >>> worked fine in Spark 0.9, but, after several days of wrestling with >>> uber jars and spark-submit, and so far failing to get Spark 1.0 >>> working, we'd like to go back to doing it ourself with SparkConf. >>> >>> As the previous poster said, a few scripts should be able to give us >>> the classpath and any other params we need, and be a lot more >>> transparent and debuggable. >>> >>> On 7/9/14, Surendranauth Hiraman <suren.hira...@velos.io> wrote: >>> > Are there any gaps beyond convenience and code/config separation in >>> using >>> > spark-submit versus SparkConf/SparkContext if you are willing to set >>> your >>> > own config? >>> > >>> > If there are any gaps, +1 on having parity within >>> SparkConf/SparkContext >>> > where possible. In my use case, we launch our jobs programmatically. In >>> > theory, we could shell out to spark-submit but it's not the best >>> option for >>> > us. >>> > >>> > So far, we are only using Standalone Cluster mode, so I'm not >>> knowledgeable >>> > on the complexities of other modes, though. >>> > >>> > -Suren >>> > >>> > >>> > >>> > On Wed, Jul 9, 2014 at 8:20 AM, Koert Kuipers <ko...@tresata.com> >>> wrote: >>> > >>> >> not sure I understand why unifying how you submit app for different >>> >> platforms and dynamic configuration cannot be part of SparkConf and >>> >> SparkContext? >>> >> >>> >> for classpath a simple script similar to "hadoop classpath" that shows >>> >> what needs to be added should be sufficient. >>> >> >>> >> on spark standalone I can launch a program just fine with just >>> SparkConf >>> >> and SparkContext. not on yarn, so the spark-launch script must be >>> doing a >>> >> few things extra there I am missing... which makes things more >>> difficult >>> >> because I am not sure its realistic to expect every application that >>> >> needs >>> >> to run something on spark to be launched using spark-submit. >>> >> On Jul 9, 2014 3:45 AM, "Patrick Wendell" <pwend...@gmail.com> >>> wrote: >>> >> >>> >>> It fulfills a few different functions. The main one is giving users a >>> >>> way to inject Spark as a runtime dependency separately from their >>> >>> program and make sure they get exactly the right version of Spark. So >>> >>> a user can bundle an application and then use spark-submit to send it >>> >>> to different types of clusters (or using different versions of >>> Spark). >>> >>> >>> >>> It also unifies the way you bundle and submit an app for Yarn, Mesos, >>> >>> etc... this was something that became very fragmented over time >>> before >>> >>> this was added. >>> >>> >>> >>> Another feature is allowing users to set configuration values >>> >>> dynamically rather than compile them inside of their program. That's >>> >>> the one you mention here. You can choose to use this feature or not. >>> >>> If you know your configs are not going to change, then you don't need >>> >>> to set them with spark-submit. >>> >>> >>> >>> >>> >>> On Wed, Jul 9, 2014 at 10:22 AM, Robert James < >>> srobertja...@gmail.com> >>> >>> wrote: >>> >>> > What is the purpose of spark-submit? Does it do anything outside of >>> >>> > the standard val conf = new SparkConf ... val sc = new SparkContext >>> >>> > ... ? >>> >>> >>> >> >>> > >>> > >>> > -- >>> > >>> > SUREN HIRAMAN, VP TECHNOLOGY >>> > Velos >>> > Accelerating Machine Learning >>> > >>> > 440 NINTH AVENUE, 11TH FLOOR >>> > NEW YORK, NY 10001 >>> > O: (917) 525-2466 ext. 105 >>> > F: 646.349.4063 >>> > E: suren.hiraman@v <suren.hira...@sociocast.com>elos.io >>> > W: www.velos.io >>> > >>> >> >> >
