[jira] [Created] (ZEPPELIN-5681) Zeppelin image can't use spark on prem

[Dmitry Kravchuk (Jira)]

Dmitry Kravchuk created ZEPPELIN-5681:
-----------------------------------------

             Summary: Zeppelin image can't use spark on prem
                 Key: ZEPPELIN-5681
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-5681
             Project: Zeppelin
          Issue Type: Bug
          Components: docker, Interpreters, pySpark, spark, spark-submit, 
zeppelin-interpreter
    Affects Versions: 0.10.0
         Environment: * apache/zeppelin:0.10.0 docker image
 * spark 3.1.1 on prem
            Reporter: Dmitry Kravchuk
             Fix For: 0.10.0
         Attachments: image-2022-03-16-09-18-24-581.png, 
image-2022-03-16-09-19-19-093.png

Hello there!

I have kinda complex task - run zeppelin in docker at on prem cluster and make 
it possible to use on prem builded spark as a source for spark interpreter.

My docker-compose file looks like this:
{code:yaml}
version: '3'
services: 
 zeppelin: 
   container_name: zeppelin10
   restart: always
   build: 
      context: /opt/docker/zeppelin/build/.
      dockerfile: /opt/docker/zeppelin/build/Dockerfile
   image: apache/zeppelin:0.10.0
   ports: 
     - "18080:8080"
     - "18443:8443"
     - "18081:8081"
     - "44000-44653:44000-44653"
     - "44655-45000:44655-45000"
   environment: 
     ZEPPELIN_PORT: 8080
     ZEPPELIN_SSL_PORT: 443
     ZEPPELIN_MEM: "-Xms2048m -Xmx2048m -XX:MaxMetaspaceSize=1024m"
     ZEPPELIN_JAVA_OPTS: "-Dspark.executor.memory=8g -Dspark.cores.max=16"
     ZEPPELIN_LOG_DIR: '/opt/zeppelin/logs'
     PATH: '/opt/conda/envs/python_3_with_R/bin:/opt/spark/bin:$PATH'    
   volumes: 
     - s_var-log_bind:/var/log
     - notebook_bind:/opt/zeppelin/notebook
     - /opt/docker/zeppelin/root/opt/zeppelin/conf:/opt/zeppelin/conf
     - /opt/docker/zeppelin/root/opt/packages:/opt/packages
     - /opt/docker/zeppelin/root/opt/delta-core:/opt/delta-core
     - /opt/docker/zeppelin/root/etc/hadoop/conf:/etc/hadoop/conf
     - spark_bind:/opt/spark
     - python_bind:/opt/python-3.7.9
     - /home/zeppelin:/home/zeppelin

volumes: 
  spark_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/opt/spark"
  notebook_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/opt/zeppelin/notebook"
  log_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/opt/zeppelin/logs"
  s_var-log_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/var/log"
  python_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/opt/python-3.7.9"
{code}
Here is my interpreter for on prem spark mounted as volume:

!image-2022-03-16-09-18-24-581.png!

Here is zeppelin output when I'm trying to run default spark interpreter and my 
own:

!image-2022-03-16-09-19-19-093.png!

So it works in local mode but not in client.

Can you help?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)