[Spark Context]: How to add on demand jobs to an existing spark context?

I want to run different jobs on demand with same spark context, but i don't know how exactly i can do this. I try to get current context, but seems it create a new spark context(with new executors). I call spark-submit to add new jobs. I run code on Amazon EMR(3 instances, 4 core & 16GB ram / in

Re: [Spark Context]: How to add on demand jobs to an existing spark context?

t; I think you are loking for livy or spark jobserver >> >> On Wed, 8 Feb 2017 at 12:37 am, Cosmin Posteuca < >> cosmin.poste...@gmail.com> wrote: >> >>> I want to run different jobs on demand with same spark context, but i >>> don't know how exa

Re: [Spark Context]: How to add on demand jobs to an existing spark context?

you >> own app like the one I wrote a year ago https://github.com/elppc/akka- >> spark-experiments >> It combines Akka actors and a shared Spark context to serve concurrent >> subsecond jobs >> >> >> 2017-02-07 15:28 GMT+01:00 ayan guha : >> >

Re: [Spark Context]: How to add on demand jobs to an existing spark context?

; > (note that there is an issue with using zeppelin in it and I have raised > it as an issue to AWS and they are looking into it now) > > Regards, > Gourav Sengupta > > On Tue, Feb 7, 2017 at 10:37 PM, Michael Segel > wrote: > >> Why couldn’t you use the spark th

Re: [Spark Context]: How to add on demand jobs to an existing spark context?

EMR cluster. > > > Regards, > Gourav > > On Wed, Feb 8, 2017 at 11:10 AM, Cosmin Posteuca < > cosmin.poste...@gmail.com> wrote: > > I tried to run some test on EMR on yarn cluster mode. > > I have a cluster with 16 cores(8 processors with 2 threads each). If i run >

[Spark Launcher] How to launch parallel jobs?

Hi, I think i don't understand enough how to launch jobs. I have one job which takes 60 seconds to finish. I run it with following command: spark-submit --executor-cores 1 \ --executor-memory 1g \ --driver-memory 1g \ --master yarn \ --deploy-m

Re: [Spark Launcher] How to launch parallel jobs?

hing in more threads, so now more tasks can >> execute in parallel. >> >> 2017-02-13 7:05 GMT-08:00 Cosmin Posteuca : >> >>> Hi, >>> >>> I think i don't understand enough how to launch jobs. >>> >>> I have one job which takes

Re: [Spark Launcher] How to launch parallel jobs?

usage. > > > > BTW, it doesn’t matter how much memory your program wants but how much it > reserves. In your example it will not take the 50MB of the test but the > ~1.5GB (after overhead) per executor. > > Hope this helps, > > Assaf. > > > > *From:* Cosmi

Re: [Spark Launcher] How to launch parallel jobs?

core each, take 60 seconds Why is it happen? why is non deterministic? Thanks 2017-02-14 10:29 GMT+02:00 Cosmin Posteuca : > Memory seems to be enough. My cluster has 22.5 gb total memory and my job > use 6.88 gb. If i run twice this job, they will use 13.75 gb, but sometimes > the clus

Re: Spark Worker can't find jar submitted programmatically

Hi Zoran, I think you are looking for --jars parameter/argument to spark-submit When using spark-submit, the application jar along with any jars included > with the --jars option will be automatically transferred to the cluster. > URLs supplied after --jars must be separated by commas. ( > http:/

Spark don't run all code when is submit to yarn-cluster mode.

Hi, I have the following problem: After SparkSession is initialized i create a task: val task = new Runnable { } where i make a REST API, and from it's response i read some data from internet/ES/Hive. This task is running to every 5 second with Akka scheduler: scheduler.schedule( Duration(0, T