Hi,
We're trying to get a Spark (1.6.1) job running on EMR (4.7.1) that's
connecting to the Hive metastore in another EMR cluster. A simplification
of what we're doing is below
val sparkConf = new SparkConf().setAppName("MyApp")
val sc = new SparkContext(sparkConf)
val sqlContext = new HiveContext(sc)
sqlContext.setConf("hive.metastore.uris", "thrift://other.emr.master:9083")
sqlContext.sql("use db1") //any statement using sql method.
sqlContext.table("db2.table1").show
The above works but if we remove the sql method then the table.show fails
with NoSuchTableException indicating it hasn't connected to the external
metastore.
So it seems something in the sql method code path does something to connect
to the metastore that table doesn't.
Does this sound familiar to anyone? Is there something we can do to avoid
having to call sql just to force it to connect/configure correctly?
Thanks,
Dave
