On Tue, Dec 1, 2015 at 9:43 PM, Anfernee Xu wrote:
> But I have a single server(JVM) that is creating SparkContext, are you
> saying Spark supports multiple SparkContext in the same JVM? Could you
> please clarify on this?
I'm confused. Nothing you said so far requires multiple contexts. From
you
Thanks Marcelo,
But I have a single server(JVM) that is creating SparkContext, are you
saying Spark supports multiple SparkContext in the same JVM? Could you
please clarify on this?
Thanks
Anfernee
On Tue, Dec 1, 2015 at 8:14 PM, Marcelo Vanzin wrote:
> On Tue, Dec 1, 2015 at 3:32 PM, Anferne
On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu wrote:
> I have a long running backend server where I will create a short-lived Spark
> job in response to each user request, base on the fact that by default
> multiple Spark Context cannot be created in the same JVM, looks like I have
> 2 choices
>
> 2
Yep, you shouldn't enable *spark.driver.allowMultipleContexts* since it has
the potential to cause extremely difficult-to-debug task failures; it was
originally introduced as an escape-hatch to allow users whose workloads
happened to work "by accident" to continue using multiple active contexts,
bu
Looks like #2 is better choice.
On Tue, Dec 1, 2015 at 4:51 PM, Anfernee Xu wrote:
> Thanks Ted, so 1) is off from the table, can I go with 2), yarn-cluster
> mode? As the driver is running as a Yarn container, it's should be OK for
> my usercase, isn't it?
>
> Anfernee
>
> On Tue, Dec 1, 2015 a
Thanks Ted, so 1) is off from the table, can I go with 2), yarn-cluster
mode? As the driver is running as a Yarn container, it's should be OK for
my usercase, isn't it?
Anfernee
On Tue, Dec 1, 2015 at 4:48 PM, Ted Yu wrote:
> For #1, looks like the config is used in test suites:
>
> .se
For #1, looks like the config is used in test suites:
.set("spark.driver.allowMultipleContexts", "true")
./sql/core/src/test/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala
.set("spark.driver.allowMultipleContexts", "true")
./sql/core/src/test/scala/org/apache/spark/sql/exec
