Just wanted to check in on this, see if I should file a bug report regarding the mesos argument propagation.
On Thu, Jul 31, 2014 at 8:35 AM, Cody Koeninger <c...@koeninger.org> wrote: > 1. I've tried with and without escaping equals sign, it doesn't affect the > results. > > 2. Yeah, exporting SPARK_SUBMIT_OPTS from spark-env.sh works for getting > system properties set in the local shell (although not for executors). > > 3. We're using the default fine-grained mesos mode, not setting > spark.mesos.coarse, so it doesn't seem immediately related to that ticket. > Should I file a bug report? > > > On Thu, Jul 31, 2014 at 1:33 AM, Patrick Wendell <pwend...@gmail.com> > wrote: > >> The third issue may be related to this: >> https://issues.apache.org/jira/browse/SPARK-2022 >> >> We can take a look at this during the bug fix period for the 1.1 >> release next week. If we come up with a fix we can backport it into >> the 1.0 branch also. >> >> On Wed, Jul 30, 2014 at 11:31 PM, Patrick Wendell <pwend...@gmail.com> >> wrote: >> > Thanks for digging around here. I think there are a few distinct issues. >> > >> > 1. Properties containing the '=' character need to be escaped. >> > I was able to load properties fine as long as I escape the '=' >> > character. But maybe we should document this: >> > >> > == spark-defaults.conf == >> > spark.foo a\=B >> > == shell == >> > scala> sc.getConf.get("spark.foo") >> > res2: String = a=B >> > >> > 2. spark.driver.extraJavaOptions, when set in the properties file, >> > don't affect the driver when running in client mode (always the case >> > for mesos). We should probably document this. In this case you need to >> > either use --driver-java-options or set SPARK_SUBMIT_OPTS. >> > >> > 3. Arguments aren't propagated on Mesos (this might be because of the >> > other issues, or a separate bug). >> > >> > - Patrick >> > >> > On Wed, Jul 30, 2014 at 3:10 PM, Cody Koeninger <c...@koeninger.org> >> wrote: >> >> In addition, spark.executor.extraJavaOptions does not seem to behave >> as I >> >> would expect; java arguments don't seem to be propagated to executors. >> >> >> >> >> >> $ cat conf/spark-defaults.conf >> >> >> >> spark.master >> >> >> mesos://zk://etl-01.mxstg:2181,etl-02.mxstg:2181,etl-03.mxstg:2181/masters >> >> spark.executor.extraJavaOptions -Dfoo.bar.baz=23 >> >> spark.driver.extraJavaOptions -Dfoo.bar.baz=23 >> >> >> >> >> >> $ ./bin/spark-shell >> >> >> >> scala> sc.getConf.get("spark.executor.extraJavaOptions") >> >> res0: String = -Dfoo.bar.baz=23 >> >> >> >> scala> sc.parallelize(1 to 100).map{ i => ( >> >> | java.net.InetAddress.getLocalHost.getHostName, >> >> | System.getProperty("foo.bar.baz") >> >> | )}.collect >> >> >> >> res1: Array[(String, String)] = Array((dn-01.mxstg,null), >> >> (dn-01.mxstg,null), (dn-01.mxstg,null), (dn-01.mxstg,null), >> >> (dn-01.mxstg,null), (dn-01.mxstg,null), (dn-01.mxstg,null), >> >> (dn-01.mxstg,null), (dn-01.mxstg,null), (dn-01.mxstg,null), >> >> (dn-01.mxstg,null), (dn-01.mxstg,null), (dn-02.mxstg,null), >> >> (dn-02.mxstg,null), ... >> >> >> >> >> >> >> >> Note that this is a mesos deployment, although I wouldn't expect that >> to >> >> affect the availability of spark.driver.extraJavaOptions in a local >> spark >> >> shell. >> >> >> >> >> >> On Wed, Jul 30, 2014 at 4:18 PM, Cody Koeninger <c...@koeninger.org> >> wrote: >> >> >> >>> Either whitespace or equals sign are valid properties file formats. >> >>> Here's an example: >> >>> >> >>> $ cat conf/spark-defaults.conf >> >>> spark.driver.extraJavaOptions -Dfoo.bar.baz=23 >> >>> >> >>> $ ./bin/spark-shell -v >> >>> Using properties file: /opt/spark/conf/spark-defaults.conf >> >>> Adding default property: >> spark.driver.extraJavaOptions=-Dfoo.bar.baz=23 >> >>> >> >>> >> >>> scala> System.getProperty("foo.bar.baz") >> >>> res0: String = null >> >>> >> >>> >> >>> If you add double quotes, the resulting string value will have double >> >>> quotes. >> >>> >> >>> >> >>> $ cat conf/spark-defaults.conf >> >>> spark.driver.extraJavaOptions "-Dfoo.bar.baz=23" >> >>> >> >>> $ ./bin/spark-shell -v >> >>> Using properties file: /opt/spark/conf/spark-defaults.conf >> >>> Adding default property: >> spark.driver.extraJavaOptions="-Dfoo.bar.baz=23" >> >>> >> >>> scala> System.getProperty("foo.bar.baz") >> >>> res0: String = null >> >>> >> >>> >> >>> Neither one of those affects the issue; the underlying problem in my >> case >> >>> seems to be that bin/spark-class uses the SPARK_SUBMIT_OPTS and >> >>> SPARK_JAVA_OPTS environment variables, but nothing parses >> >>> spark-defaults.conf before the java process is started. >> >>> >> >>> Here's an example of the process running when only >> spark-defaults.conf is >> >>> being used: >> >>> >> >>> $ ps -ef | grep spark >> >>> >> >>> 514 5182 2058 0 21:05 pts/2 00:00:00 bash >> ./bin/spark-shell -v >> >>> >> >>> 514 5189 5182 4 21:05 pts/2 00:00:22 >> /usr/local/java/bin/java >> >>> -cp >> >>> >> ::/opt/spark/conf:/opt/spark/lib/spark-assembly-1.0.1-hadoop2.3.0-mr1-cdh5.0.2.jar:/etc/hadoop/conf-mx >> >>> -XX:MaxPermSize=128m -Djava.library.path= -Xms512m -Xmx512m >> >>> org.apache.spark.deploy.SparkSubmit spark-shell -v --class >> >>> org.apache.spark.repl.Main >> >>> >> >>> >> >>> Here's an example of it when the command line --driver-java-options is >> >>> used (and thus things work): >> >>> >> >>> >> >>> $ ps -ef | grep spark >> >>> 514 5392 2058 0 21:15 pts/2 00:00:00 bash >> ./bin/spark-shell -v >> >>> --driver-java-options -Dfoo.bar.baz=23 >> >>> >> >>> 514 5399 5392 80 21:15 pts/2 00:00:06 >> /usr/local/java/bin/java >> >>> -cp >> >>> >> ::/opt/spark/conf:/opt/spark/lib/spark-assembly-1.0.1-hadoop2.3.0-mr1-cdh5.0.2.jar:/etc/hadoop/conf-mx >> >>> -XX:MaxPermSize=128m -Dfoo.bar.baz=23 -Djava.library.path= -Xms512m >> >>> -Xmx512m org.apache.spark.deploy.SparkSubmit spark-shell -v >> >>> --driver-java-options -Dfoo.bar.baz=23 --class >> org.apache.spark.repl.Main >> >>> >> >>> >> >>> >> >>> >> >>> On Wed, Jul 30, 2014 at 3:43 PM, Patrick Wendell <pwend...@gmail.com> >> >>> wrote: >> >>> >> >>>> Cody - in your example you are using the '=' character, but in our >> >>>> documentation and tests we use a whitespace to separate the key and >> >>>> value in the defaults file. >> >>>> >> >>>> docs: http://spark.apache.org/docs/latest/configuration.html >> >>>> >> >>>> spark.driver.extraJavaOptions -Dfoo.bar.baz=23 >> >>>> >> >>>> I'm not sure if the java properties file parser will try to interpret >> >>>> the equals sign. If so you might need to do this. >> >>>> >> >>>> spark.driver.extraJavaOptions "-Dfoo.bar.baz=23" >> >>>> >> >>>> Do those work for you? >> >>>> >> >>>> On Wed, Jul 30, 2014 at 1:32 PM, Marcelo Vanzin <van...@cloudera.com >> > >> >>>> wrote: >> >>>> > Hi Cody, >> >>>> > >> >>>> > Could you file a bug for this if there isn't one already? >> >>>> > >> >>>> > For system properties SparkSubmit should be able to read those >> >>>> > settings and do the right thing, but that obviously won't work for >> >>>> > other JVM options... the current code should work fine in cluster >> mode >> >>>> > though, since the driver is a different process. :-) >> >>>> > >> >>>> > >> >>>> > On Wed, Jul 30, 2014 at 1:12 PM, Cody Koeninger < >> c...@koeninger.org> >> >>>> wrote: >> >>>> >> We were previously using SPARK_JAVA_OPTS to set java system >> properties >> >>>> via >> >>>> >> -D. >> >>>> >> >> >>>> >> This was used for properties that varied on a >> >>>> per-deployment-environment >> >>>> >> basis, but needed to be available in the spark shell and workers. >> >>>> >> >> >>>> >> On upgrading to 1.0, we saw that SPARK_JAVA_OPTS had been >> deprecated, >> >>>> and >> >>>> >> replaced by spark-defaults.conf and command line arguments to >> >>>> spark-submit >> >>>> >> or spark-shell. >> >>>> >> >> >>>> >> However, setting spark.driver.extraJavaOptions and >> >>>> >> spark.executor.extraJavaOptions in spark-defaults.conf is not a >> >>>> replacement >> >>>> >> for SPARK_JAVA_OPTS: >> >>>> >> >> >>>> >> >> >>>> >> $ cat conf/spark-defaults.conf >> >>>> >> spark.driver.extraJavaOptions=-Dfoo.bar.baz=23 >> >>>> >> >> >>>> >> $ ./bin/spark-shell >> >>>> >> >> >>>> >> scala> System.getProperty("foo.bar.baz") >> >>>> >> res0: String = null >> >>>> >> >> >>>> >> >> >>>> >> $ ./bin/spark-shell --driver-java-options "-Dfoo.bar.baz=23" >> >>>> >> >> >>>> >> scala> System.getProperty("foo.bar.baz") >> >>>> >> res0: String = 23 >> >>>> >> >> >>>> >> >> >>>> >> Looking through the shell scripts for spark-submit and >> spark-class, I >> >>>> can >> >>>> >> see why this is; parsing spark-defaults.conf from bash could be >> >>>> brittle. >> >>>> >> >> >>>> >> But from an ergonomic point of view, it's a step back to go from a >> >>>> >> set-it-and-forget-it configuration in spark-env.sh, to requiring >> >>>> command >> >>>> >> line arguments. >> >>>> >> >> >>>> >> I can solve this with an ad-hoc script to wrap spark-shell with >> the >> >>>> >> appropriate arguments, but I wanted to bring the issue up to see >> if >> >>>> anyone >> >>>> >> else had run into it, >> >>>> >> or had any direction for a general solution (beyond parsing java >> >>>> properties >> >>>> >> files from bash). >> >>>> > >> >>>> > >> >>>> > >> >>>> > -- >> >>>> > Marcelo >> >>>> >> >>> >> >>> >> > >
