Krishna: If you want to query ORC files, see the following JIRA: [SPARK-10623] [SQL] Fixes ORC predicate push-down
which is in the 1.5.1. release. FYI On Mon, Sep 28, 2015 at 9:42 AM, Fabien Martin <fabien.marti...@gmail.com> wrote: > Hi Krishna, > > - Take a lokk at > *http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/ > > <http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/>* > - Or you can specify your 1.5.x jar as the Spark one using something > like : > > --conf > "spark.yarn.jar=hdfs://master:8020/spark-assembly-1.5.0-hadoop2.6.0.jar" > > The main drawback is : > > *Known Issues* > > *Spark YARN ATS integration does not work in this tech preview. You will > not see the history of Spark jobs in the Jobs server after a job is > finished.* > > 2015-09-23 1:31 GMT+02:00 Zhan Zhang <zzh...@hortonworks.com>: > >> Hi Krishna, >> >> For the time being, you can download from upstream, and it should be >> running OK for HDP2.3. For hdp specific problem, you can ask in >> Hortonworks forum. >> >> Thanks. >> >> Zhan Zhang >> >> On Sep 22, 2015, at 3:42 PM, Krishna Sankar <ksanka...@gmail.com> wrote: >> >> Guys, >> >> - We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The >> current wisdom is that it will support the 1.4.x train (which is good, >> need >> DataFrame et al). >> - What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on >> HDP 2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ? >> >> Cheers & Thanks >> <k/> >> >> >> >
