Re: how to give hdfs file path as argument to spark-submit

2017-02-17 Thread Jörn Franke
You should not use the scala API to load the file, but the data source API or the hadoop methods of Spark. > On 17 Feb 2017, at 10:12, nancy henry wrote: > > Hi All, > > > object Step1 { > def main(args: Array[String]) = { > > val sparkConf = new SparkConf().setAppName("my-app") >

how to give hdfs file path as argument to spark-submit

2017-02-17 Thread nancy henry
Hi All, object Step1 { def main(args: Array[String]) = { val sparkConf = new SparkConf().setAppName("my-app") val sc = new SparkContext(sparkConf) val hiveSqlContext: HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) hiveSqlContext.sql(scala.io.Source.fromFile(args(