Hi,
I can run the same code in SPARK_HOME/bin/spark-shell. However, it does not
allow me to execute the SQL command.
On Thursday, November 17, 2016 6:01 PM, moon soo Lee <[email protected]>
wrote:
Are you able to run the same code in SPARK_HOME/bin/spark-shell?
Thanks,moon
On Thu, Nov 17, 2016 at 9:47 AM Muhammad Rezaul Karim <[email protected]>
wrote:
Hi,
Thanks a lot. Yes, I have removed the metastore_db directory too.
On Thursday, November 17, 2016 5:38 PM, moon soo Lee <[email protected]>
wrote:
Although "export PATH=$PATH..." is not really necessary in zeppelin-env.sh, i
think your configuration looks okay.
Have you tried remove /home/asif/zeppelin-0.6.2-bin-all/metastore_db ?
Thanks,moon
On Wed, Nov 16, 2016 at 6:48 PM Muhammad Rezaul Karim <[email protected]>
wrote:
Hi Moon,
I have set those variables as follows (a partial view of the zeppelin-env.sh
file). Is this okay?
#!/bin/bash
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=$PATH:$JAVA_HOME/bin
export SPARK_HOME=/home/asif/spark-2.0.0-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin
# Options read in YARN client mode
export HADOOP_CONF_DIR=/home/asif/Downloads/hadoop-2.7.3/etc
export PATH=$PATH:$HADOOP_CONF_DIR/hadoop
On Thursday, November 17, 2016 1:45 AM, Md. Rezaul Karim
<[email protected]> wrote:
Hi Moon, No, I did not set those two environmental variables. I'm still how
to do that. I have installed Spark on my machine and have set the SPARK_HOME.
However, do I also need to install and configure the Hadoop. But the error is
all about the Hive. Am I some how wrong? A sample zeppelin-env.sh file would be
very helpful.
On Nov 16, 2016 10:47 PM, moon soo Lee <[email protected]> wrote:
Hi,
It's strange, Do you have SPARK_HOME or HADOOP_CONF_DIR defined in
conf/zeppelin-env.sh?
You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db,
start Zeppelin and try again.
Thanks,moon
On Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim <[email protected]>
wrote:
Hi,I am a new user of Apache Zeppelin and I am running a simple notebook app on
Zeppelin (version 0.6.2-bin-all) using Scala based on Spark.
My source code is as follows:
val bankText =
sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")case class
Bank(age:String, job:String, marital:String, education:String, balance:String)
val bank = bankText.map(s=>s.split(";")).map(
s=>Bank(s(0),
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5)
)
)
bank.collect()val mydf=bank.toDF()
mydf.registerTempTable("mydf");
Upto bank.collect() it works pretty well. However, I'm getting the following
error on Ubuntu 14.04 when trying to execute the last two lines of my code:
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at
org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:171)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
... 46 elided
Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
... 70 more
Caused by: java.lang.reflect.InvocationTargetException:
javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the
given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true,
username = APP. Terminating connection pool (set lazyInit to true if you expect
to start your database after your app). Original Exception: ------
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with
class loader
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the
next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 136 more
Caused by: java.sql.SQLException: Database at
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with
the current version of the software. The database was created by or upgraded by
version 10.11.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
... 133 more
Caused by: ERROR XSLAN: Database at
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with
the current version of the software. The database was created by or upgraded by
version 10.11.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
------
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 76 more
Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test
connection to the given database. JDBC url =
jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating
connection pool (set lazyInit to true if you expect to start your database
after your app). Original Exception: ------
java.sql.SQLException: Failed to start database 'metastore_db' with class
loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e,
see the next exception for details.
Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with
class loader
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the
next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 136 more
Caused by: java.sql.SQLException: Database at
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with
the current version of the software. The database was created by or upgraded by
version 10.11.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
... 133 more
Caused by: ERROR XSLAN: Database at
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with
the current version of the software. The database was created by or upgraded by
version 10.11.
Caused by: org.apache.derby.impl.jdbc.EmbedSQLException: Failed to start
database 'metastore_db' with class loader
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the
next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 136 more
Caused by: org.apache.derby.impl.jdbc.EmbedSQLException: Database at
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with
the current version of the software. The database was created by or upgraded by
version 10.11.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
... 133 more
Caused by: org.apache.derby.iapi.error.StandardException: Database at
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with
the current version of the software. The database was created by or upgraded by
version 10.11.
Thanks and Regards,
---------------------------------
Md. Rezaul Karim
PhD Researcher, Insight Centre for Data Analytics
National University of Ireland Galway
