You’ll want to setup the FAIR scheduler as described here: https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application
From: yael aharon <[email protected]<mailto:[email protected]>> Date: Friday, February 12, 2016 at 2:00 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Allowing parallelism in spark local mode Hello, I have an application that receives requests over HTTP and uses spark in local mode to process the requests. Each request is running in its own thread. It seems that spark is queueing the jobs, processing them one at a time. When 2 requests arrive simultaneously, the processing time for each of them is almost doubled. I tried setting spark.default.parallelism, spark.executor.cores, spark.driver.cores but that did not change the time in a meaningful way. Am I missing something obvious? thanks, Yael
