Awesome, thanks Jeff. On Sun, Jul 25, 2021 at 11:24 AM Jeff Zhang <zjf...@gmail.com> wrote:
> Hi Lior， > > It would be fixed in https://github.com/apache/zeppelin/pull/4127 > > > Lior Chaga <lio...@taboola.com> 于2021年7月25日周日 下午3:58写道： > >> After a couple of attempts of code fixes, when every time I seemed to >> make things work just to find out the next step in the process breaks, I've >> found the most simple solution - put them extraJavaOptions in >> spark-defaults.conf (instead of keeping them in interpreter settings) >> >> >> >> On Sun, Jul 11, 2021 at 1:30 PM Lior Chaga <lio...@taboola.com> wrote: >> >>> Thanks Jeff, >>> So I should escape the whitespaces? Is there a ticket for it? couldn't >>> find one >>> >>> On Sun, Jul 11, 2021 at 1:10 PM Jeff Zhang <zjf...@gmail.com> wrote: >>> >>>> I believe this is due to SparkInterpreterLauncher doesn't support >>>> parameters with whitespace. (It would use whitespace as delimiter to >>>> separate parameters), this is a known issue >>>> >>>> Lior Chaga <lio...@taboola.com> 于2021年7月11日周日 下午4:14写道： >>>> >>>>> So after adding the quotes in both SparkInterpreterLauncher >>>>> and interpreter.sh, interpreter is still failing with same error of >>>>> Unrecognized option. >>>>> But the weird thing is that if I copy the command supposedly executed >>>>> from zeppelin (as it is printed to log) and run it directly in shell, the >>>>> interpreter process is properly running. So my guess is that the forked >>>>> process command that is created, is not really identical to the one that >>>>> is >>>>> logged. >>>>> >>>>> This is how my cmd looks like (censored a bit): >>>>> >>>>> /usr/local/spark/bin/spark-submit >>>>> --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer >>>>> --driver-class-path >>>>> :/zeppelin/local-repo/spark/*:/zeppelin/interpreter/spark/*:::/zeppelin/inter >>>>> preter/zeppelin-interpreter-shaded-0.10.0-SNAPSHOT.jar:/zeppelin/interpreter/spark/spark-interpreter-0.10.0-SNAPSHOT.jar:/etc/hadoop/conf >>>>> >>>>> *--driver-java-options " -DSERVICENAME=zeppelin_docker >>>>> -Dfile.encoding=UTF-8 >>>>> -Dlog4j.configuration=file:///zeppelin/conf/log4j.properties >>>>> -Dlog4j.configurationFile=file:///zeppelin/conf/log4j2.properties >>>>> -Dzeppelin.log.file=/var/log/zeppelin/zeppelin-interpreter-spark-shared_process--zeppelin-test-spark3-7d74d5df4-2g8x5.log" >>>>> * >>>>> --conf spark.driver.host=10.135.120.245 >>>>> --conf "spark.dynamicAllocation.minExecutors=1" >>>>> --conf "spark.shuffle.service.enabled=true" >>>>> --conf "spark.sql.parquet.int96AsTimestamp=true" >>>>> --conf "spark.ui.retainedTasks=10000" >>>>> --conf "spark.executor.heartbeatInterval=600s" >>>>> --conf "spark.ui.retainedJobs=100" >>>>> --conf "spark.sql.ui.retainedExecutions=10" >>>>> --conf "spark.hadoop.cloneConf=true" >>>>> --conf "spark.debug.maxToStringFields=200000" >>>>> --conf "spark.executor.memory=70g" >>>>> --conf >>>>> "spark.executor.extraClassPath=../mysql-connector-java-8.0.18.jar:../guava-19.0.jar" >>>>> >>>>> --conf "spark.hadoop.fs.permissions.umask-mode=000" >>>>> --conf "spark.memory.storageFraction=0.1" >>>>> --conf "spark.scheduler.mode=FAIR" >>>>> --conf "spark.sql.adaptive.enabled=true" >>>>> --conf >>>>> "spark.master=mesos://zk://zk003:2181,zk004:2181,zk006:2181,/mesos-zeppelin" >>>>> >>>>> --conf "spark.driver.memory=15g" >>>>> --conf "spark.io.compression.codec=lz4" >>>>> --conf "spark.executor.uri= >>>>> https://artifactory.company.com/artifactory/static/spark/spark-dist/spark-3.1.2.2-hadoop-2.7-zulu"; >>>>> - >>>>> -conf "spark.ui.retainedStages=500" >>>>> --conf "spark.mesos.uris= >>>>> https://artifactory.company.com/artifactory/static/spark/spark-executor/jars/mysql-connector-java-8.0.18.jar,https://artifactory.company.com/artifactory/static/spark/spark-executor/jars/guava-19.0.jar"; >>>>> >>>>> --conf "spark.driver.maxResultSize=8g" >>>>> *--conf "spark.executor.extraJavaOptions=-DSERVICENAME=Zeppelin >>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=2015 >>>>> -XX:-OmitStackTraceInFastThrow -Dcom.sun.management.jmxremote >>>>> -Dcom.sun.management.jmxremote.port=55745 >>>>> -Dcom.sun.management.jmxremote.authenticate=false >>>>> -Dcom.sun.management.jmxremote.ssl=false -verbose:gc >>>>> -Dlog4j.configurationFile=/etc/config/log4j2-executor-config.xml >>>>> -XX:+UseG1GC -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps >>>>> -XX:+PrintGCTimeStamps -XX:+PrintFlagsFinal -XX:+PrintReferenceGC >>>>> -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy >>>>> -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark >>>>> -XX:+PrintStringDeduplicationStatistics -XX:+UseStringDeduplication >>>>> -XX:InitiatingHeapOccupancyPercent=35 >>>>> -Dhttps.proxyHost=proxy.service.consul -Dhttps.proxyPort=3128" * >>>>> --conf "spark.dynamicAllocation.enabled=true" >>>>> --conf "spark.default.parallelism=1200" >>>>> --conf >>>>> "spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2" >>>>> --conf >>>>> "spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS" >>>>> >>>>> --conf "spark.app.name=zeppelin_docker_spark3" >>>>> --conf "spark.shuffle.service.port=7337" >>>>> --conf "spark.memory.fraction=0.75" >>>>> --conf "spark.mesos.coarse=true" >>>>> --conf "spark.ui.port=4041" >>>>> --conf "spark.dynamicAllocation.executorIdleTimeout=60s" >>>>> --conf "spark.sql.shuffle.partitions=1200" >>>>> --conf "spark.sql.parquet.outputTimestampType=TIMESTAMP_MILLIS" >>>>> --conf "spark.dynamicAllocation.cachedExecutorIdleTimeout=120s" >>>>> --conf "spark.network.timeout=1200s" >>>>> --conf "spark.cores.max=600" >>>>> --conf >>>>> "spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem" >>>>> >>>>> --conf "spark.worker.timeout=150000" >>>>> *--conf >>>>> "spark.driver.extraJavaOptions=-Dhttps.proxyHost=proxy.service.consul >>>>> -Dhttps.proxyPort=3128 >>>>> -Dlog4j.configuration=file:/usr/local/spark/conf/log4j.properties >>>>> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver >>>>> -Djavax.jdo.option.ConnectionPassword=2eebb22277 >>>>> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://proxysql-backend.service.consul.company.com:6033/hms?useSSL=false&databaseTerm=SCHEMA&nullDatabaseMeansCurrent=true >>>>> <http://proxysql-backend.service.consul.company.com:6033/hms?useSSL=false&databaseTerm=SCHEMA&nullDatabaseMeansCurrent=true> >>>>> -Djavax.jdo.option.ConnectionUserName=hms_rw" * >>>>> --conf "spark.files.overwrite=true" >>>>> /zeppelin/interpreter/spark/spark-interpreter-0.10.0-SNAPSHOT.jar >>>>> 10.135.120.245 >>>>> 36419 >>>>> spark-shared_process : >>>>> >>>>> >>>>> >>>>> *Error: Unrecognized option: >>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=2015* >>>>> >>>>> Will continue tackling it... >>>>> >>>>> On Thu, Jul 8, 2021 at 4:49 PM Jeff Zhang <zjf...@gmail.com> wrote: >>>>> >>>>>> Thanks Lior for the investigation. >>>>>> >>>>>> >>>>>> Lior Chaga <lio...@taboola.com> 于2021年7月8日周四 下午8:31写道： >>>>>> >>>>>>> Ok, I think I found the issue. It's not only that the quotations are >>>>>>> missing from the --conf param, they are also missing from >>>>>>> the --driver-java-options, which is concatenated to >>>>>>> the INTERPRETER_RUN_COMMAND in interpreter.sh >>>>>>> >>>>>>> I will fix it in my build, but would like a confirmation that this >>>>>>> is indeed the issue (and I'm not missing anything), so I'd open a pull >>>>>>> request. >>>>>>> >>>>>>> On Thu, Jul 8, 2021 at 3:05 PM Lior Chaga <lio...@taboola.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I'm trying to run zeppelin using local spark interpreter. >>>>>>>> Basically everything works, but if I try to set >>>>>>>> `spark.driver.extraJavaOptions` or `spark.executor.extraJavaOptions` >>>>>>>> containing several arguments, I get an exception. >>>>>>>> For instance, for providing `-DmyParam=1 -DmyOtherParam=2`, I'd get: >>>>>>>> Error: Unrecognized option: -DmyOtherParam=2 >>>>>>>> >>>>>>>> I noticed that the spark submit looks as follow: >>>>>>>> >>>>>>>> spark-submit --class >>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer >>>>>>>> --driver-class-path >>>>>>>> .... *--conf spark.driver.extraJavaOptions=-DmyParam=1 >>>>>>>> -DmyOtherParam=2* >>>>>>>> >>>>>>>> So I tried to patch SparkInterpreterLauncher to add quotation marks >>>>>>>> (like in the example from spark documentation - >>>>>>>> https://spark.apache.org/docs/latest/configuration.html#dynamically-loading-spark-properties >>>>>>>> ) >>>>>>>> >>>>>>>> I see that the quotation marks were added: *--conf >>>>>>>> "spark.driver.extraJavaOptions=-DmyParam=1 -DmyOtherParam=2"* >>>>>>>> But I still get the same error. >>>>>>>> >>>>>>>> Any idea how I can make it work? >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards >>>>>> >>>>>> Jeff Zhang >>>>>> >>>>> >>>> >>>> -- >>>> Best Regards >>>> >>>> Jeff Zhang >>>> >>> > > -- > Best Regards > > Jeff Zhang >
