Yes, my configuration files are at /opt/hadoop/conf...but the configuration was not correct. It's working now.
On Sun, Oct 9, 2016 at 3:54 PM Jeff Zhang <zjf...@gmail.com> wrote: > There're 2 processes, the first one is embedded mode. The second is the > normal spark launch mode. I am not sure which process you are using, can > you kill these processes and rerun again ? > > And are you sure you put hadoop conf files under /opt/hadoop/conf/ ? > > On Sun, Oct 9, 2016 at 3:14 PM, Xi Shen <davidshe...@gmail.com> wrote: > > > ps aux | grep RemoteInterpreterServer > > root 71 0.1 3.3 3528164 133668 ? Sl 06:21 0:04 > /usr/lib/jvm/java-8-openjdk-amd64//bin/java -Dfile.encoding=UTF-8 > -Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties > -Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-sh--cad975db8a21.log > -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp > ::/opt/zeppelin-0.6.1-bin-all/interpreter/sh/*::/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar > org.apache.zeppelin.interpreter.remote.*RemoteInterpreterServer *45877 > root 346 0.7 13.7 3612088 555092 ? Sl 06:28 0:18 > /usr/lib/jvm/java-8-openjdk-amd64//bin/java -cp > /opt/zeppelin-0.6.1-bin-all/interpreter/spark/*:/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar:/opt/spark/conf/:/opt/spark/jars/*:/opt/hadoop/conf/ > -Xmx1g -Dfile.encoding=UTF-8 > -Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties > -Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-spark--cad975db8a21.log > org.apache.spark.deploy.SparkSubmit --conf > spark.driver.extraClassPath=::/opt/zeppelin-0.6.1-bin-all/interpreter/spark/*::/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar > --conf spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8 > -Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties > -Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-spark--cad975db8a21.log > --class org.apache.zeppelin.interpreter.remote.*RemoteInterpreterServer > */opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar > 44320 > > I started my Zeppelin instance with > > /opt/zeppelin-0.6.1-bin-all/bin/zeppelin.sh --conf > /opt/zeppelin-0.6.1-bin-all/conf > > On Sun, Oct 9, 2016 at 2:54 PM Jeff Zhang <zjf...@gmail.com> wrote: > > Could you check the classpath of the interpreter process ? I suspect you > didn't export HADOOP_CONF_DIR correctly. > > Run the following command: > > ps aux | grep RemoteInterpreterServer > > On Sun, Oct 9, 2016 at 2:36 PM, Xi Shen <davidshe...@gmail.com> wrote: > > Hi, > > I followed http://zeppelin.apache.org/docs/latest/interpreter/spark.html, > and set up SPARK_HOME, HADOOP_CONF_DIR. > > My SPARK build is 2.0. My Zeppelin build is the 0.6.1 binary from the web. > > After I start Zeppelin, I went to the interpreter setting page and changed > Spark interpreter settings as follow: > > master: yarn > deploy-mode: client > > Then, I created a new notebook and executed: > > %spark > > spark.version > > The block never finishes...no error either. > > In the ./logs/zeppelin-interpreter-spark-*.log, I found the following *which > I think is the cause of my problem* > > INFO [2016-10-09 06:28:37,074] ({pool-2-thread-2} > Logging.scala[logInfo]:54) - Added JAR > *file:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar* > at spark://172.17.0.3:38775/jars/zeppelin-spark_2.11-0.6.1.jar with > timestamp 1475994517073 > INFO [2016-10-09 06:28:37,150] ({pool-2-thread-2} > Logging.scala[logInfo]:54) - Created default pool default, schedulingMode: > FIFO, minShare: 0, weight: 1 > INFO [2016-10-09 06:28:38,205] ({pool-2-thread-2} > RMProxy.java[createRMProxy]:98) - *Connecting to ResourceManager at > /0.0.0.0:8032 <http://0.0.0.0:8032>* > > Looks like Zeppelin is not using my Spark binary, and is not using my > hadoop configuration. > > What did I miss? > > -- > > > Thanks, > David S. > > > > > -- > Best Regards > > Jeff Zhang > > -- > > > Thanks, > David S. > > > > > -- > Best Regards > > Jeff Zhang > -- Thanks, David S.
