GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/2161
[SPARK-2608][Core] Fixed command line option passing issue over Mesos via
SPARK_EXECUTOR_OPTS
This is another try after #2145 to fix
[SPARK-2608](https://issues.apache.org/jira/browse/SPARK-2608).
### Basic Idea
The basic idea is to pass `extraJavaOpts` and `extraLibraryPath` together
via environment variable `SPARK_EXECUTOR_OPTS`. This variable is recognized by
`spark-class` and not used anywhere else. In this way, we still launch Mesos
executors with `spark-class`/`spark-executor`, but avoids the executor side
Spark home issue.
### Known Issue
Quoted string with spaces is not allowed in either `extraJavaOpts` or
`extraLibraryPath` when using Spark over Mesos. The reason is that Mesos passes
the whole command line as a single string argument to `sh -c` to start the
executor, and this makes shell string escaping non-trivial to handle. This
should be fixed in a later release.
### Background
Classes in package `org.apache.spark.deploy` shouldn't be used as they
assume Spark is deployed in standalone mode, and give wrong executor side Spark
home directory. Please refer to comments in #2145 for more details.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark mesos-fix-with-env-var
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2161.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2161
----
commit 117407654c0fd92d8dd39b0dcc802e6baac0a668
Author: Cheng Lian <[email protected]>
Date: 2014-08-26T06:34:07Z
Draft fix for CoarseMesosSchedulerBackend
commit ba59190d41b5d779235404e25eb7d4b7d57f5eb7
Author: Cheng Lian <[email protected]>
Date: 2014-08-27T06:12:25Z
Added fine grained Mesos executor support
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
