thanks for the reply. I am trying to use it without hive setup
(spark-standalone), so it prints something like this:
hive_ctx.sql("show tables").collect()
15/05/15 17:59:03 INFO HiveMetaStore: 0: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/05/15 17:59:03 INFO ObjectStore: ObjectStore, initialize called
15/05/15 17:59:04 INFO Persistence: Property datanucleus.cache.level2
unknown - will be ignored
15/05/15 17:59:04 INFO Persistence: Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/05/15 17:59:04 WARN Connection: BoneCP specified but not present in
CLASSPATH (or one of dependencies)
15/05/15 17:59:05 WARN Connection: BoneCP specified but not present in
CLASSPATH (or one of dependencies)
15/05/15 17:59:08 INFO BlockManagerMasterActor: Registering block manager
xxxx:42819 with 3.0 GB RAM, BlockManagerId(2, xxx, 42819)
[0/1844]
15/05/15 17:59:18 INFO ObjectStore: Setting MetaStore object pin classes
with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/05/15 17:59:18 INFO MetaStoreDirectSql: MySQL check failed, assuming we
are not on mysql: Lexical error at line 1, column 5. Encountered: "@"
(64), after : "".
15/05/15 17:59:20 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
15/05/15 17:59:20 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
15/05/15 17:59:28 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
15/05/15 17:59:29 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
15/05/15 17:59:31 INFO ObjectStore: Initialized ObjectStore
15/05/15 17:59:32 WARN ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording
the schema version 0.13.1aa
15/05/15 17:59:33 WARN MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-azure-file-system.properties,hadoop-metrics2.properties
15/05/15 17:59:33 INFO MetricsSystemImpl: Scheduled snapshot period at 10
second(s).
15/05/15 17:59:33 INFO MetricsSystemImpl: azure-file-system metrics system
started
15/05/15 17:59:33 INFO HiveMetaStore: Added admin role in metastore
15/05/15 17:59:34 INFO HiveMetaStore: Added public role in metastore
15/05/15 17:59:34 INFO HiveMetaStore: No user is added in admin role, since
config is empty
15/05/15 17:59:35 INFO SessionState: No Tez session required at this point.
hive.execution.engine=mr.
15/05/15 17:59:37 INFO HiveMetaStore: 0: get_tables: db=default pat=.*
15/05/15 17:59:37 INFO audit: ugi=testuser ip=unknown-ip-addr
cmd=get_tables: db=default pat=.*
not sure what to put in hive.metastore.uris in this case?
On Fri, May 15, 2015 at 2:52 PM, Yana Kadiyska <yana.kadiy...@gmail.com>
wrote:
> This should work. Which version of Spark are you using? Here is what I do
> -- make sure hive-site.xml is in the conf directory of the machine you're
> using the driver from. Now let's run spark-shell from that machine:
>
> scala> val hc= new org.apache.spark.sql.hive.HiveContext(sc)
> hc: org.apache.spark.sql.hive.HiveContext =
> org.apache.spark.sql.hive.HiveContext@6e9f8f26
>
> scala> hc.sql("show tables").collect
> 15/05/15 09:34:17 INFO metastore: Trying to connect to metastore with URI
> thrift://hostname.com:9083 <-- here should be a value from your
> hive-site.xml
> 15/05/15 09:34:17 INFO metastore: Waiting 1 seconds before next connection
> attempt.
> 15/05/15 09:34:18 INFO metastore: Connected to metastore.
> res0: Array[org.apache.spark.sql.Row] = Array([table1,false],
>
> scala> hc.getConf("hive.metastore.uris")
> res13: String = thrift://hostname.com:9083
>
> scala> hc.getConf("hive.metastore.warehouse.dir")
> res14: String = /user/hive/warehouse
>
>
>
> The first line tells you which metastore it's trying to connect to -- this
> should be the string specified under hive.metastore.uris property in your
> hive-site.xml file. I have not mucked with warehouse.dir too much but I
> know that the value of the metastore URI is in fact picked up from there as
> I regularly point to different systems...
>
>
> On Thu, May 14, 2015 at 6:26 PM, Tamas Jambor <jambo...@gmail.com> wrote:
>
>> I have tried to put the hive-site.xml file in the conf/ directory with,
>> seems it is not picking up from there.
>>
>>
>> On Thu, May 14, 2015 at 6:50 PM, Michael Armbrust <mich...@databricks.com
>> > wrote:
>>
>>> You can configure Spark SQLs hive interaction by placing a hive-site.xml
>>> file in the conf/ directory.
>>>
>>> On Thu, May 14, 2015 at 10:24 AM, jamborta <jambo...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> is it possible to set hive.metastore.warehouse.dir, that is internally
>>>> create by spark, to be stored externally (e.g. s3 on aws or wasb on
>>>> azure)?
>>>>
>>>> thanks,
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/store-hive-metastore-on-persistent-store-tp22891.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>
>>>>
>>>
>>
>
