RE: SPARK-2243 Support multiple SparkContexts in the same JVM

2014-12-18 Thread Anton Brazhnyk
Am I right on this? -Original Message- From: Sean Owen [mailto:so...@cloudera.com] Sent: Thursday, December 18, 2014 2:04 AM To: Anton Brazhnyk Cc: user@spark.apache.org Subject: Re: SPARK-2243 Support multiple SparkContexts in the same JVM Yes, although once you have multiple ClassL

Re: SPARK-2243 Support multiple SparkContexts in the same JVM

2014-12-18 Thread Sean Owen
Yes, although once you have multiple ClassLoaders, you are operating as if in multiple JVMs for most intents and purposes. I think the request for this kind of functionality comes from use cases where multiple ClassLoaders wouldn't work, like, wanting to have one app (in one ClassLoader) managing m

Re: SPARK-2243 Support multiple SparkContexts in the same JVM

2014-12-17 Thread Marcelo Vanzin
Hi Anton, That could solve some of the issues (I've played with that a little bit). But there are still some areas where this would be sub-optimal, because Spark still uses system properties in some places and those are global, not per-class loader. (SparkSubmit is the biggest offender here, but