Praveen, the mode in which you run spark (standalone, yarn, mesos) is determined when you create SparkContext <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext>. You are right that spark-submit and spark-shell create different SparkContexts.
In general, resource sharing is the task of the cluster scheduler and there are only 3 choices by default right now. On Sun, Feb 14, 2016 at 9:13 PM, praveen S <mylogi...@gmail.com> wrote: > Even i was trying to launch spark jobs from webservice : > > But I thought you could run spark jobs in yarn mode only through > spark-submit. Is my understanding not correct? > > Regards, > Praveen > On 15 Feb 2016 08:29, "Sabarish Sasidharan" < > sabarish.sasidha...@manthan.com> wrote: > >> Yes you can look at using the capacity scheduler or the fair scheduler >> with YARN. Both allow using full cluster when idle. And both allow >> considering cpu plus memory when allocating resources which is sort of >> necessary with Spark. >> >> Regards >> Sab >> On 13-Feb-2016 10:11 pm, "Eugene Morozov" <evgeny.a.moro...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I have several instances of the same web-service that is running some ML >>> algos on Spark (both training and prediction) and do some Spark unrelated >>> job. Each web-service instance creates their on JavaSparkContext, thus >>> they're seen as separate applications by Spark, thus they're configured >>> with separate limits of resources such as cores (I'm not concerned about >>> the memory as much as about cores). >>> >>> With this set up, say 3 web service instances, each of them has just 1/3 >>> of cores. But it might happen, than only one instance is going to use >>> Spark, while others are busy with Spark unrelated. I'd like in this case >>> all Spark cores be available for the one that's in need. >>> >>> Ideally I'd like Spark cores just be available in total and the first >>> app who needs it, takes as much as required from the available at the >>> moment. Is it possible? I believe Mesos is able to set resources free if >>> they're not in use. Is it possible with YARN? >>> >>> I'd appreciate if you could share your thoughts or experience on the >>> subject. >>> >>> Thanks. >>> -- >>> Be well! >>> Jean Morozov >>> >> -- Alex Kozlov ale...@gmail.com
