Ah, great to know this is already being fixed. Thanks Patrick, I have
marked my JIRA as a duplicate.
2014-08-07 21:42 GMT-07:00 Patrick Wendell :
> Andrew - I think your JIRA may duplicate existing work:
> https://github.com/apache/spark/pull/1513
>
>
> On Thu, Aug 7, 2014 at 7:55 PM, Andrew Or
Andrew - I think your JIRA may duplicate existing work:
https://github.com/apache/spark/pull/1513
On Thu, Aug 7, 2014 at 7:55 PM, Andrew Or wrote:
> @Cody I took a quick glance at the Mesos code and it appears that we
> currently do not even pass extra java options to executors except in coarse
@Cody I took a quick glance at the Mesos code and it appears that we
currently do not even pass extra java options to executors except in coarse
grained mode, and even in this mode we do not pass them to executors
correctly. I have filed a related JIRA here:
https://issues.apache.org/jira/browse/SP
Thanks Marcelo, I have moved the changes to a new PR to describe the
problems more clearly: https://github.com/apache/spark/pull/1845
@Gary Yeah, the goal is to get this into 1.1 as a bug fix.
2014-08-07 17:30 GMT-07:00 Gary Malouf :
> Can this be cherry-picked for 1.1 if everything works out?
Can this be cherry-picked for 1.1 if everything works out? In my opinion,
it could be qualified as a bug fix.
On Thu, Aug 7, 2014 at 5:47 PM, Marcelo Vanzin wrote:
> Andrew has been working on a fix:
> https://github.com/apache/spark/pull/1770
>
> On Thu, Aug 7, 2014 at 2:35 PM, Cody Koeninger
Andrew has been working on a fix:
https://github.com/apache/spark/pull/1770
On Thu, Aug 7, 2014 at 2:35 PM, Cody Koeninger wrote:
> Just wanted to check in on this, see if I should file a bug report
> regarding the mesos argument propagation.
>
>
> On Thu, Jul 31, 2014 at 8:35 AM, Cody Koeninger
Just wanted to check in on this, see if I should file a bug report
regarding the mesos argument propagation.
On Thu, Jul 31, 2014 at 8:35 AM, Cody Koeninger wrote:
> 1. I've tried with and without escaping equals sign, it doesn't affect the
> results.
>
> 2. Yeah, exporting SPARK_SUBMIT_OPTS fr
1. I've tried with and without escaping equals sign, it doesn't affect the
results.
2. Yeah, exporting SPARK_SUBMIT_OPTS from spark-env.sh works for getting
system properties set in the local shell (although not for executors).
3. We're using the default fine-grained mesos mode, not setting
spark
The third issue may be related to this:
https://issues.apache.org/jira/browse/SPARK-2022
We can take a look at this during the bug fix period for the 1.1
release next week. If we come up with a fix we can backport it into
the 1.0 branch also.
On Wed, Jul 30, 2014 at 11:31 PM, Patrick Wendell wro
Thanks for digging around here. I think there are a few distinct issues.
1. Properties containing the '=' character need to be escaped.
I was able to load properties fine as long as I escape the '='
character. But maybe we should document this:
== spark-defaults.conf ==
spark.foo a\=B
== shell ==
In addition, spark.executor.extraJavaOptions does not seem to behave as I
would expect; java arguments don't seem to be propagated to executors.
$ cat conf/spark-defaults.conf
spark.master
mesos://zk://etl-01.mxstg:2181,etl-02.mxstg:2181,etl-03.mxstg:2181/masters
spark.executor.extraJavaOptions
Either whitespace or equals sign are valid properties file formats.
Here's an example:
$ cat conf/spark-defaults.conf
spark.driver.extraJavaOptions -Dfoo.bar.baz=23
$ ./bin/spark-shell -v
Using properties file: /opt/spark/conf/spark-defaults.conf
Adding default property: spark.driver.extraJavaOpt
Cody - in your example you are using the '=' character, but in our
documentation and tests we use a whitespace to separate the key and
value in the defaults file.
docs: http://spark.apache.org/docs/latest/configuration.html
spark.driver.extraJavaOptions -Dfoo.bar.baz=23
I'm not sure if the java
Hi Cody,
Could you file a bug for this if there isn't one already?
For system properties SparkSubmit should be able to read those
settings and do the right thing, but that obviously won't work for
other JVM options... the current code should work fine in cluster mode
though, since the driver is a
14 matches
Mail list logo
