Hi,
We're having a similar issue. We have a standalone cluster running 1.5.2
with Hive working fine having dropped hive-site.xml into the conf folder.
We've just updated to 1.6.0, using the same configuration. Now when
starting a spark-shell we get the following:
java.lang.RuntimeException: java.lang.RuntimeException: Unable to
instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreCli
ent
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at
org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)
at
org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
at
org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
at
org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
at
org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
at
org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
at
org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
at
org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
at
org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
at $iwC$$iwC.<init>(<console>:15)
at $iwC.<init>(<console>:24)
at <init>(<console>:26)
On stepping though the code and enabling debug it shows that
hive.metastore.uris is not set:
DEBUG ClientWrapper: Hive Config: hive.metastore.uris=
..So it looks like it's not finding hive-site.xml? Weirdly, if I remove
hive-site.xml the exception does not occur which implies that it WAS on the
classpath...
Dave
On Tue, 9 Feb 2016 at 22:26 Koert Kuipers <ko...@tresata.com> wrote:
> i do not have phoenix, but i wonder if its something related. will check
> my classpaths
>
> On Tue, Feb 9, 2016 at 5:00 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>
>> I got the same problem when I added the Phoenix plugin jar in the driver
>> and executor extra classpaths. Do you have those set too?
>>
>
>> On Feb 9, 2016, at 1:12 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>> yes its not using derby i think: i can see the tables in my actual hive
>> metastore.
>>
>> i was using a symlink to /etc/hive/conf/hive-site.xml for my
>> hive-site.xml which has a lot more stuff than just hive.metastore.uris
>>
>> let me try your approach
>>
>>
>>
>> On Tue, Feb 9, 2016 at 3:57 PM, Alexandr Dzhagriev <dzh...@gmail.com>
>> wrote:
>>
>>> I'm using spark 1.6.0, hive 1.2.1 and there is just one property in the
>>> hive-site.xml hive.metastore.uris Works for me. Can you check in the
>>> logs, that when the HiveContext is created it connects to the correct uri
>>> and doesn't use derby.
>>>
>>> Cheers, Alex.
>>>
>>> On Tue, Feb 9, 2016 at 9:39 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>>
>>>> hey thanks. hive-site is on classpath in conf directory
>>>>
>>>> i currently got it to work by changing this hive setting in
>>>> hive-site.xml:
>>>> hive.metastore.schema.verification=true
>>>> to
>>>> hive.metastore.schema.verification=false
>>>>
>>>> this feels like a hack, because schema verification is a good thing i
>>>> would assume?
>>>>
>>>> On Tue, Feb 9, 2016 at 3:25 PM, Alexandr Dzhagriev <dzh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Koert,
>>>>>
>>>>> As far as I can see you are using derby:
>>>>>
>>>>> Using direct SQL, underlying DB is DERBY
>>>>>
>>>>> not mysql, which is used for the metastore. That means, spark couldn't
>>>>> find hive-site.xml on your classpath. Can you check that, please?
>>>>>
>>>>> Thanks, Alex.
>>>>>
>>>>> On Tue, Feb 9, 2016 at 8:58 PM, Koert Kuipers <ko...@tresata.com>
>>>>> wrote:
>>>>>
>>>>>> has anyone successfully connected to hive metastore using spark
>>>>>> 1.6.0? i am having no luck. worked fine with spark 1.5.1 for me. i am on
>>>>>> cdh 5.5 and launching spark with yarn.
>>>>>>
>>>>>> this is what i see in logs:
>>>>>> 16/02/09 14:49:12 INFO hive.metastore: Trying to connect to metastore
>>>>>> with URI thrift://metastore.mycompany.com:9083
>>>>>> 16/02/09 14:49:12 INFO hive.metastore: Connected to metastore.
>>>>>>
>>>>>> and then a little later:
>>>>>>
>>>>>> 16/02/09 14:49:34 INFO hive.HiveContext: Initializing execution hive,
>>>>>> version 1.2.1
>>>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Inspected Hadoop
>>>>>> version: 2.6.0-cdh5.4.4
>>>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Loaded
>>>>>> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version
>>>>>> 2.6.0-cdh5.4.4
>>>>>> 16/02/09 14:49:34 WARN conf.HiveConf: HiveConf of name
>>>>>> hive.server2.enable.impersonation does not exist
>>>>>> 16/02/09 14:49:35 INFO metastore.HiveMetaStore: 0: Opening raw store
>>>>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>>>>>> 16/02/09 14:49:35 INFO metastore.ObjectStore: ObjectStore, initialize
>>>>>> called
>>>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property
>>>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored
>>>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property
>>>>>> datanucleus.cache.level2 unknown - will be ignored
>>>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified but
>>>>>> not present in CLASSPATH (or one of dependencies)
>>>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified but
>>>>>> not present in CLASSPATH (or one of dependencies)
>>>>>> 16/02/09 14:49:37 WARN conf.HiveConf: HiveConf of name
>>>>>> hive.server2.enable.impersonation does not exist
>>>>>> 16/02/09 14:49:37 INFO metastore.ObjectStore: Setting MetaStore
>>>>>> object pin classes with
>>>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
>>>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class
>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class
>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class
>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class
>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>> 16/02/09 14:49:40 INFO metastore.MetaStoreDirectSql: Using direct
>>>>>> SQL, underlying DB is DERBY
>>>>>> 16/02/09 14:49:40 INFO metastore.ObjectStore: Initialized ObjectStore
>>>>>> java.lang.RuntimeException: java.lang.RuntimeException: Unable to
>>>>>> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>>>>>> at
>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
>>>>>> at
>>>>>> org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)
>>>>>> at
>>>>>> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
>>>>>> at
>>>>>> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
>>>>>> at
>>>>>> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
>>>>>> at
>>>>>> org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:440)
>>>>>> at
>>>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272)
>>>>>> at
>>>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:271)
>>>>>> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>>>>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>>>>>> at
>>>>>> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>>>>>> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>>>>>> at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:271)
>>>>>> at
>>>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:97)
>>>>>> at
>>>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>> Method)
>>>>>> at
>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at
>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at org.apache.spark.repl.Main$.createSQLContext(Main.scala:89)
>>>>>> ... 47 elided
>>>>>> Caused by: java.lang.RuntimeException: Unable to instantiate
>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
>>>>>> at
>>>>>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
>>>>>> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
>>>>>> at
>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
>>>>>> ... 66 more
>>>>>> Caused by: java.lang.reflect.InvocationTargetException:
>>>>>> org.apache.hadoop.hive.metastore.api.MetaException: Version information
>>>>>> not
>>>>>> found in metastore.
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>> Method)
>>>>>> at
>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at
>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
>>>>>> ... 72 more
>>>>>> Caused by: org.apache.hadoop.hive.metastore.api.MetaException:
>>>>>> Version information not found in metastore.
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6664)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
>>>>>> at com.sun.proxy.$Proxy26.verifySchema(Unknown Source)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
>>>>>> at
>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
>>>>>> ... 77 more
>>>>>>
>>>>>> i seem to get this exact same error no matter what i set
>>>>>> spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars to. i
>>>>>> tried building different spark 1.6.0 jars (with hadoop provided, with
>>>>>> hadoop included), but no effect.
>>>>>>
>>>>>> any ideas?
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
