Re: Cloudera Spark 2.2

2017-08-04 Thread Benjamin Kim
Ruslan, Quick observation. I see that you put Scala 2.10. How come you didn't use Scala 2.11? Just curious. Cheers, Ben On Fri, Aug 4, 2017 at 8:36 AM Ruslan Dautkhanov wrote: > This should do: > > >> export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >> export HIVE_HOME=/opt/cloudera

Re: Cloudera Spark 2.2

2017-08-04 Thread Ruslan Dautkhanov
This should do: > export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 > export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive > export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop > export HADOOP_CONF_DIR=/etc/hadoop/conf > export HIVE_CONF_DIR=/etc/hive/conf > > mvn clean package -Dsk

Re: Cloudera Spark 2.2

2017-08-04 Thread Benjamin Kim
Hi Ruslan, Can you send me the steps you used to build it, especially the Maven command with the arguments? I will try to build it also. I do believe that the binaries are for official releases. Cheers, Ben On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov wrote: > It was built. I think binar

Re: Cloudera Spark 2.2

2017-08-02 Thread Ruslan Dautkhanov
It was built. I think binaries are only available for official releases? -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim wrote: > Did you build Zeppelin or download the binary? > > On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov > wrote: > >> We're using an ~April snapsho

Re: Cloudera Spark 2.2

2017-08-02 Thread Benjamin Kim
Did you build Zeppelin or download the binary? On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov wrote: > We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. > > Yes, we have that spark home in zeppelin-env.sh > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 4:31 PM, Ben

Re: Cloudera Spark 2.2

2017-08-02 Thread Ruslan Dautkhanov
We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. Yes, we have that spark home in zeppelin-env.sh -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim wrote: > Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in > zeppelin-env.sh to wha

Re: Cloudera Spark 2.2

2017-08-02 Thread Benjamin Kim
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below. On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov wrote: > You don't have to use spark2-shell and spark2-submit to use Spark 2. > That can be controled by setting SPARK_HOME using re

Re: Cloudera Spark 2.2

2017-08-02 Thread Ruslan Dautkhanov
You don't have to use spark2-shell and spark2-submit to use Spark 2. That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell. $ which spark-submit /usr/bin/spark-submit $ which spark-shell /usr/bin/spark-shell $ spark-shell Welcome to __ / _

Re: Cloudera Spark 2.2

2017-08-02 Thread Benjamin Kim
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to b

Re: Cloudera Spark 2.2

2017-08-01 Thread Ruslan Dautkhanov
Might need to recompile Zeppelin with Scala 2.11? Also Spark 2.2 now requires JDK8 I believe. -- Ruslan Dautkhanov On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: > Here is more. > > org.apache.zeppelin.interpreter.InterpreterException: WARNING: > User-defined SPARK_HOME (/opt/cloudera/p

Re: Cloudera Spark 2.2

2017-08-01 Thread Benjamin Kim
Here is more. org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). WARNING: Running spark-class from user-defined location

Re: Cloudera Spark 2.2

2017-08-01 Thread Jeff Zhang
Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. Benjamin Kim 于2017年8月2日周三 上午8:22写道： > Here is the error that was sent to me. > > org.apache.zeppelin.interpreter.InterpreterException: Exception in thread > "mai

Re: Cloudera Spark 2.2

2017-08-01 Thread Benjamin Kim
Here is the error that was sent to me. org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream Cheers, Ben On Tue,

Re: Cloudera Spark 2.2

2017-08-01 Thread Jeff Zhang
By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check. Jeff Zhang 于2017年8月2日周三 上午8:18写道： > > What's the error you see in log ? > > > Benjamin Kim 于2017年8月2日周三 上午8:18写道： > >> Has anyone

Re: Cloudera Spark 2.2

2017-08-01 Thread Jeff Zhang
What's the error you see in log ? Benjamin Kim 于2017年8月2日周三 上午8:18写道： > Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? > I can't get it to work. I downloaded the binary and set SPARK_HOME to > /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. > > Ch