You can set the max cores for the first submitted job such that it does not take all the resources from the master. See http://spark.apache.org/docs/latest/submitting-applications.html
# Run on a Spark standalone cluster in client deploy mode ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://207.184.161.138:7077 \ --executor-memory 20G \ *--total-executor-cores 100 \* /path/to/examples.jar \ 1000 On Mon, Oct 19, 2015 at 4:26 PM, Augustus Hong <augus...@branchmetrics.io> wrote: > Hi All, > > Would it be possible to run multiple spark streaming jobs on a single > master at the same time? > > I currently have one master node and several worker nodes in the > standalone mode, and I used spark-submit to submit multiple spark streaming > jobs. > > From what I observed, it seems like only the first submitted job would get > resources and run. Jobs submitted afterwards will have the status > "Waiting", and will only run after the first one is finished or killed. > > I tried limiting each executor to only 1 core(each worker machine has 8 > cores), but the same things happens that only one job will be run, even > though there are a lot of idle cores. > > Best, > Augustus > > > > -- > [image: Branch Metrics mobile deep linking] <http://branch.io/>* Augustus > Hong* > Data Analytics | Branch Metrics > m 650-391-3369 | e augus...@branch.io >
