Thanks Ted, so 1) is off from the table, can I go with 2), yarn-cluster mode? As the driver is running as a Yarn container, it's should be OK for my usercase, isn't it?
Anfernee On Tue, Dec 1, 2015 at 4:48 PM, Ted Yu <[email protected]> wrote: > For #1, looks like the config is used in test suites: > > .set("spark.driver.allowMultipleContexts", "true") > ./sql/core/src/test/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala > .set("spark.driver.allowMultipleContexts", "true") > > ./sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala > .set("spark.driver.allowMultipleContexts", "false") > val conf = new SparkConf().set("spark.driver.allowMultipleContexts", > "false") > .set("spark.driver.allowMultipleContexts", "true") > .set("spark.driver.allowMultipleContexts", "true")) > ./core/src/test/scala/org/apache/spark/SparkContextSuite.scala > > FYI > > On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <[email protected]> wrote: > >> Hi, >> >> I have a doubt regarding yarn-cluster mode and spark.driver. >> allowMultipleContexts for below usercases. >> >> I have a long running backend server where I will create a short-lived >> Spark job in response to each user request, base on the fact that by >> default multiple Spark Context cannot be created in the same JVM, looks >> like I have 2 choices >> >> 1) enable spark.driver.allowMultipleContexts >> >> 2) run my jobs in yarn-cluster mode instead yarn-client >> >> For 1) I cannot find any official document, so looks like it's not >> encouraged, isn't it? >> For 2), I want to make sure yarn-cluster will NOT hit such >> limitation(single SparkContext per VM), apparently I have to something in >> driver side to push the result set back to my application. >> >> Thanks >> >> -- >> --Anfernee >> > > -- --Anfernee
