Probably a noob question.
But I am trying to create a hive table using spark-sql.
Here is what I am trying to do:
hc = HiveContext(sc)
hdf = hc.parquetFile(output_path)
data_types = hdf.dtypes
schema = "(" + " ,".join(map(lambda x: x[0] + " " + x[1], data_types)) +")"
hc.sql(" CREATE TABLE IF NOT EXISTS example.foo " + schema)
There is already a database called "example" in hive.
But I see an error:
An error occurred while calling o35.sql.
: org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution
Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:java.lang.IllegalArgumentException:
java.net.URISyntaxException: Relative path in absolute URI:
hdfs://some_path/foo)
Also, I was wondering on how to use saveAsTable(..) construct
hdf.saveAsTable(tablename) tries to store into default db? How do I specify
the database name (example in this case) while trying to store this table?
Thanks
--
Mohit
"When you want success as badly as you want the air, then you will get it.
There is no other secret of success."
-Socrates
