Re: How to insert data into 2000 partitions(directories) of ORC/parquet at a time using Spark SQL?

[Sree Eedupuganti]

Hi Spark users, i am new to spark. I am trying to connect hive using
SparkJavaContext. Unable to connect to the database. By executing the below
code i can see only "default" database. Can anyone help me out. What i need
is a sample program for Querying Hive results using SparkJavaContext. Need
to pass any values like this.

userDF.registerTempTable("userRecordsTemp")

sqlContext.sql("SET hive.default.fileformat=Orc  ")
sqlContext.sql("set hive.enforce.bucketing = true; ")
sqlContext.sql("set hive.enforce.sorting = true; ")

         public static void  main(String[] args ) throws Exception {
                  SparkConf sparkConf = new
SparkConf().setAppName("SparkSQL").setMaster("local");
                  SparkContext  ctx=new SparkContext(sparkConf);
                  HiveContext  hiveql=new
org.apache.spark.sql.hive.HiveContext(ctx);
                  DataFrame df=hiveql.sql("show databases");
                  df.show();
                  }

Any suggestions please....Thanks.