Which version of Spark are you using?
Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi> On Mon, Mar 10, 2014 at 6:49 PM, abhinav chowdary < abhinav.chowd...@gmail.com> wrote: > for any one who is interested to know about job server from Ooyala.. we > started using it recently and been working great so far.. > On Feb 25, 2014 9:23 PM, "Ognen Duzlevski" <og...@nengoiksvelzud.com> > wrote: > >> In that case, I must have misunderstood the following (from >> http://spark.incubator.apache.org/docs/0.8.1/job-scheduling.html). >> Apologies. Ognen >> >> "Inside a given Spark application (SparkContext instance), multiple >> parallel jobs can run simultaneously if they were submitted from separate >> threads. By “job”, in this section, we mean a Spark action (e.g. save, >> collect) and any tasks that need to run to evaluate that action. Spark’s >> scheduler is fully thread-safe and supports this use case to enable >> applications that serve multiple requests (e.g. queries for multiple >> users). >> >> By default, Spark’s scheduler runs jobs in FIFO fashion. Each job is >> divided into “stages” (e.g. map and reduce phases), and the first job gets >> priority on all available resources while its stages have tasks to launch, >> then the second job gets priority, etc. If the jobs at the head of the >> queue don’t need to use the whole cluster, later jobs can start to run >> right away, but if the jobs at the head of the queue are large, then later >> jobs may be delayed significantly. >> >> Starting in Spark 0.8, it is also possible to configure fair sharing >> between jobs. Under fair sharing, Spark assigns tasks between jobs in a >> “round robin” fashion, so that all jobs get a roughly equal share of >> cluster resources. This means that short jobs submitted while a long job is >> running can start receiving resources right away and still get good >> response times, without waiting for the long job to finish. This mode is >> best for multi-user settings. >> >> To enable the fair scheduler, simply set the spark.scheduler.mode to FAIR >> before creating a SparkContext:" >> On 2/25/14, 12:30 PM, Mayur Rustagi wrote: >> >> fair scheduler merely reorders tasks .. I think he is looking to run >> multiple pieces of code on a single context on demand from customers...if >> the code & order is decided then fair scheduler will ensure that all tasks >> get equal cluster time :) >> >> >> >> Mayur Rustagi >> Ph: +919632149971 >> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >> https://twitter.com/mayur_rustagi >> >> >> >> On Tue, Feb 25, 2014 at 10:24 AM, Ognen Duzlevski < >> og...@nengoiksvelzud.com> wrote: >> >>> Doesn't the fair scheduler solve this? >>> Ognen >>> >>> >>> On 2/25/14, 12:08 PM, abhinav chowdary wrote: >>> >>> Sorry for not being clear earlier >>> how do you want to pass the operations to the spark context? >>> this is partly what i am looking for . How to access the active spark >>> context and possible ways to pass operations >>> >>> Thanks >>> >>> >>> >>> On Tue, Feb 25, 2014 at 10:02 AM, Mayur Rustagi < >>> mayur.rust...@gmail.com> wrote: >>> >>>> how do you want to pass the operations to the spark context? >>>> >>>> >>>> Mayur Rustagi >>>> Ph: +919632149971 >>>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >>>> https://twitter.com/mayur_rustagi >>>> >>>> >>>> >>>> On Tue, Feb 25, 2014 at 9:59 AM, abhinav chowdary < >>>> abhinav.chowd...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> I am looking for ways to share the sparkContext, meaning i need >>>>> to be able to perform multiple operations on the same spark context. >>>>> >>>>> Below is code of a simple app i am testing >>>>> >>>>> def main(args: Array[String]) { >>>>> println("Welcome to example application!") >>>>> >>>>> val sc = new SparkContext("spark://10.128.228.142:7077", "Simple >>>>> App") >>>>> >>>>> println("Spark context created!") >>>>> >>>>> println("Creating RDD!") >>>>> >>>>> Now once this context is created i want to access this to submit >>>>> multiple jobs/operations >>>>> >>>>> Any help is much appreciated >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Warm Regards >>> Abhinav Chowdary >>> >>> >>> >> >>
