Hi Roberto
I have question regarding HiveContext .
when you create HiveContext where you define Hive connection properties ?
Suppose Hive is not in local machine i need to connect , how HiveConext will
know the data base info like url ,username and password ?
String username = "";
String password = "";
String url = "jdbc:hive2://quickstart.cloudera:10000/default";
On Friday, July 17, 2015 2:29 AM, Roberto Coluccio
<[email protected]> wrote:
Hello community,
I'm currently using Spark 1.3.1 with Hive support for outputting processed data
on an external Hive table backed on S3. I'm using a manual specification of the
delimiter, but I'd want to know if is there any "clean" way to write in CSV
format:
val sparkConf = new SparkConf()val sc = new SparkContext(sparkConf)val
hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)import
hiveContext.implicits._ hiveContext.sql( "CREATE EXTERNAL TABLE IF NOT EXISTS
table_name(field1 STRING, field2 STRING) ROW FORMAT DELIMITED FIELDS TERMINATED
BY ',' LOCATION '" + path_on_s3 + "'")hiveContext.sql(<an INSERT OVERWRITE
query to write into the above table>)
I also need the header of the table to be printed on each written file. I tried
with:
hiveContext.sql("set hive.cli.print.header=true")
But it didn't work.
Any hint?
Thank you.
Best regards,Roberto
