Thanks a lot for creating the issue. It seems I am not allowed to. As I understand it, the environment variable is supposed to be SPARK_MASTER.
On Sun, May 17, 2020 at 11:56 AM Alex Ott <alex...@gmail.com> wrote: > Ok, I've created a JIRA for it: > https://issues.apache.org/jira/browse/ZEPPELIN-4821 and working on patch > > I'm not sure about environment variable name - it's simply MASTER, should > it be `SPARK_MASTER`, or it's a requirement of CDH and other Hadoop > distributions to have it as MASTER? > > On Sat, May 16, 2020 at 3:45 PM Patrik Iselind <patrik....@gmail.com> > wrote: > >> Hi Alex, >> >> Thanks a lot for helping out with this. >> >> You're correct, but it doesn't seem that it's the >> interpreter-settings.json for Spark interpreter that is being used. It's >> conf/interpreter.json. In this file both 0.8.2 and 0.9.0 have >> ```partial-json >> "spark": { >> "id": "spark", >> "name": "spark", >> "group": "spark", >> "properties": { >> "SPARK_HOME": { >> "name": "SPARK_HOME", >> "value": "", >> "type": "string", >> "description": "Location of spark distribution" >> }, >> "master": { >> "name": "master", >> "value": "local[*]", >> "type": "string", >> "description": "Spark master uri. local | yarn-client | >> yarn-cluster | spark master address of standalone mode, ex) >> spark://master_host:7077" >> }, >> ``` >> That "master" should be "spark.master". >> >> By adding an explicit spark.master with the value "local[*]" I can use >> all cores as expected. Without this and printing sc.master I get "local". >> With the addition of the spark.master property set to "local[*]" and >> printing sc.master I get "local[*]". My conclusion is >> that conf/interpreter.json isn't in sync with the interpreter-settings.json >> for Spark interpreter. >> >> Best regards, >> Patrik Iselind >> >> >> On Sat, May 16, 2020 at 11:22 AM Alex Ott <alex...@gmail.com> wrote: >> >>> Spark master is set to `local[*]` by default. Here is corresponding piece >>> form interpreter-settings.json for Spark interpreter: >>> >>> "master": { >>> "envName": "MASTER", >>> "propertyName": "spark.master", >>> "defaultValue": "local[*]", >>> "description": "Spark master uri. local | yarn-client | >>> yarn-cluster | spark master address of standalone mode, ex) >>> spark://master_host:7077", >>> "type": "string" >>> }, >>> >>> >>> Patrik Iselind at "Sun, 10 May 2020 20:31:08 +0200" wrote: >>> PI> Hi Jeff, >>> >>> PI> I've tried the release from >>> http://zeppelin.apache.org/download.html, both in a docker and without >>> a docker. They both have the same issue as >>> PI> previously described. >>> >>> PI> Can I somehow set spark.master to "local[*]" in zeppelin, perhaps >>> using some environment variable? >>> >>> PI> When is the next Zeppelin 0.9.0 docker image planned to be released? >>> >>> PI> Best Regards, >>> PI> Patrik Iselind >>> >>> PI> On Sun, May 10, 2020 at 9:26 AM Jeff Zhang <zjf...@gmail.com> >>> wrote: >>> >>> PI> Hi Patric, >>> PI> >>> PI> Do you mind to try the 0.9.0-preview, it might be an issue of >>> docker container. >>> PI> >>> PI> http://zeppelin.apache.org/download.html >>> >>> PI> Patrik Iselind <patrik....@gmail.com> 于2020年5月10日周日上午2:30写道： >>> PI> >>> PI> Hello Jeff, >>> PI> >>> PI> Thank you for looking into this for me. >>> PI> >>> PI> Using the latest pushed docker image for 0.9.0 (image >>> ID 92890adfadfb, built 6 weeks ago), I still see the same issue. My image >>> has >>> PI> the digest "apache/zeppelin@sha256 >>> :0691909f6884319d366f5d3a5add8802738d6240a83b2e53e980caeb6c658092". >>> PI> >>> PI> If it's not on the tip of master, could you guys please >>> release a newer 0.9.0 image? >>> PI> >>> PI> Best Regards, >>> PI> Patrik Iselind >>> >>> PI> On Sat, May 9, 2020 at 4:03 PM Jeff Zhang <zjf...@gmail.com> >>> wrote: >>> PI> >>> PI> This might be a bug of 0.8, I tried that in 0.9 (master >>> branch), it works for me. >>> PI> >>> PI> print(sc.master) >>> PI> print(sc.defaultParallelism) >>> PI> >>> PI> --- >>> PI> local[*] 8 >>> >>> PI> Patrik Iselind <patrik....@gmail.com> >>> 于2020年5月9日周六下午8:34写道： >>> PI> >>> PI> Hi, >>> PI> >>> PI> First comes some background, then I have some >>> questions. >>> PI> >>> PI> Background >>> PI> I'm trying out Zeppelin 0.8.2 based on the Docker >>> image. My Docker file looks like this: >>> PI> >>> PI> ```Dockerfile >>> PI> FROM apache/zeppelin:0.8.2 >>> >>> PI> >>> PI> # Install Java and some tools >>> PI> RUN apt-get -y update &&\ >>> PI> DEBIAN_FRONTEND=noninteractive \ >>> PI> apt -y install vim python3-pip >>> PI> >>> PI> RUN python3 -m pip install -U pyspark >>> PI> >>> PI> ENV PYSPARK_PYTHON python3 >>> PI> ENV PYSPARK_DRIVER_PYTHON python3 >>> PI> ``` >>> PI> >>> PI> When I start a section like so >>> PI> >>> PI> ```Zeppelin paragraph >>> PI> %pyspark >>> PI> >>> PI> print(sc) >>> PI> print() >>> PI> print(dir(sc)) >>> PI> print() >>> PI> print(sc.master) >>> PI> print() >>> PI> print(sc.defaultParallelism) >>> PI> ``` >>> PI> >>> PI> I get the following output >>> PI> >>> PI> ```output >>> PI> <SparkContext master=local appName=Zeppelin> >>> ['PACKAGE_EXTENSIONS', '__class__', '__delattr__', '__dict__', '__dir__', >>> PI> '__doc__', '__enter__', '__eq__', '__exit__', >>> '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', >>> PI> '__hash__', '__init__', '__le__', '__lt__', >>> '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', >>> '__repr__', >>> PI> '__setattr__', '__sizeof__', '__str__', >>> '__subclasshook__', '__weakref__', '_accumulatorServer', >>> '_active_spark_context', >>> PI> '_batchSize', '_callsite', '_checkpointFile', >>> '_conf', '_dictToJavaMap', '_do_init', '_ensure_initialized', '_gateway', >>> PI> '_getJavaStorageLevel', '_initialize_context', >>> '_javaAccumulator', '_jsc', '_jvm', '_lock', '_next_accum_id', >>> PI> '_pickled_broadcast_vars', '_python_includes', >>> '_repr_html_', '_temp_dir', '_unbatched_serializer', 'accumulator', >>> 'addFile', >>> PI> 'addPyFile', 'appName', 'applicationId', >>> 'binaryFiles', 'binaryRecords', 'broadcast', 'cancelAllJobs', >>> 'cancelJobGroup', >>> PI> 'defaultMinPartitions', 'defaultParallelism', >>> 'dump_profiles', 'emptyRDD', 'environment', 'getConf', 'getLocalProperty', >>> PI> 'getOrCreate', 'hadoopFile', 'hadoopRDD', 'master', >>> 'newAPIHadoopFile', 'newAPIHadoopRDD', 'parallelize', 'pickleFile', >>> PI> 'profiler_collector', 'pythonExec', 'pythonVer', >>> 'range', 'runJob', 'sequenceFile', 'serializer', 'setCheckpointDir', >>> PI> 'setJobGroup', 'setLocalProperty', 'setLogLevel', >>> 'setSystemProperty', 'show_profiles', 'sparkHome', 'sparkUser', 'startTime', >>> PI> 'statusTracker', 'stop', 'textFile', 'uiWebUrl', >>> 'union', 'version', 'wholeTextFiles'] local 1 >>> PI> ``` >>> PI> >>> PI> This even though the "master" property in the >>> interpretter is set to "local[*]". I'd like to use all cores on my machine. >>> To >>> PI> do that I have to explicitly create the >>> "spark.master" property in the spark interpretter with the value >>> "local[*]", then I >>> PI> get >>> PI> >>> PI> ```new output >>> PI> <SparkContext master=local[*] appName=Zeppelin> >>> ['PACKAGE_EXTENSIONS', '__class__', '__delattr__', '__dict__', '__dir__', >>> PI> '__doc__', '__enter__', '__eq__', '__exit__', >>> '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', >>> PI> '__hash__', '__init__', '__le__', '__lt__', >>> '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', >>> '__repr__', >>> PI> '__setattr__', '__sizeof__', '__str__', >>> '__subclasshook__', '__weakref__', '_accumulatorServer', >>> '_active_spark_context', >>> PI> '_batchSize', '_callsite', '_checkpointFile', >>> '_conf', '_dictToJavaMap', '_do_init', '_ensure_initialized', '_gateway', >>> PI> '_getJavaStorageLevel', '_initialize_context', >>> '_javaAccumulator', '_jsc', '_jvm', '_lock', '_next_accum_id', >>> PI> '_pickled_broadcast_vars', '_python_includes', >>> '_repr_html_', '_temp_dir', '_unbatched_serializer', 'accumulator', >>> 'addFile', >>> PI> 'addPyFile', 'appName', 'applicationId', >>> 'binaryFiles', 'binaryRecords', 'broadcast', 'cancelAllJobs', >>> 'cancelJobGroup', >>> PI> 'defaultMinPartitions', 'defaultParallelism', >>> 'dump_profiles', 'emptyRDD', 'environment', 'getConf', 'getLocalProperty', >>> PI> 'getOrCreate', 'hadoopFile', 'hadoopRDD', 'master', >>> 'newAPIHadoopFile', 'newAPIHadoopRDD', 'parallelize', 'pickleFile', >>> PI> 'profiler_collector', 'pythonExec', 'pythonVer', >>> 'range', 'runJob', 'sequenceFile', 'serializer', 'setCheckpointDir', >>> PI> 'setJobGroup', 'setLocalProperty', 'setLogLevel', >>> 'setSystemProperty', 'show_profiles', 'sparkHome', 'sparkUser', 'startTime', >>> PI> 'statusTracker', 'stop', 'textFile', 'uiWebUrl', >>> 'union', 'version', 'wholeTextFiles'] local[*] 8 >>> PI> ``` >>> PI> This is what I want. >>> PI> >>> PI> The Questions >>> PI> @ Why is the "master" property not used in the >>> created SparkContext? >>> PI> @ How do I add the spark.master property to the >>> docker image? >>> PI> >>> PI> Any hint or support you can provide would be >>> greatly appreciated. >>> PI> >>> PI> Yours Sincerely, >>> PI> Patrik Iselind >>> >>> PI> -- >>> PI> Best Regards >>> PI> >>> PI> Jeff Zhang >>> >>> PI> -- >>> PI> Best Regards >>> PI> >>> PI> Jeff Zhang >>> >>> >>> >>> -- >>> With best wishes, Alex Ott >>> http://alexott.net/ >>> Twitter: alexott_en (English), alexott (Russian) >>> >> > > -- > With best wishes, Alex Ott > http://alexott.net/ > Twitter: alexott_en (English), alexott (Russian) >
