the only way I managed this work (and I have not tried newer build yet) was Hive. 2.0.1 on Spark 1.3.1 engine
Connected to: Apache Hive (version 2.0.1 0: jdbc:hive2://rhes564:10099/default> select count(1) from passwd; INFO : Compiling command(queryId=hduser_20170617175040_5d4a85d9-a306-4cb3-a191-e675cdd21dcf): select count(1) from passwd INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hduser_20170617175040_5d4a85d9-a306-4cb3-a191-e675cdd21dcf); Time taken: 0.064 seconds INFO : Executing command(queryId=hduser_20170617175040_5d4a85d9-a306-4cb3-a191-e675cdd21dcf): select count(1) from passwd INFO : Query ID = hduser_20170617175040_5d4a85d9-a306-4cb3-a191-e675cdd21dcf INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : *Query Hive on Spark job[0] stages:* INFO : 0 INFO : 1 INFO : *Status: Running (Hive on Spark job[0])* HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 17 June 2017 at 17:33, Boris Lublinsky <boris.lublin...@lightbend.com> wrote: > You need to exlicitely build spark without Hive. Look at getting started > doc > > Get Outlook for Android <https://aka.ms/ghei36> > > > > > On Sat, Jun 17, 2017 at 5:26 AM -0400, "wuchang" <583424...@qq.com> wrote: > > I want to build hive and spark to make my hive based on spark engine.I >> choose Hive 2.3.0 and Spark 2.0.0， which is claimed to be *compatible* >> by hive official document.According to the hive officials document ,I >> have to build spark *without hive profile *to avoid the conflict between >> original hive and spark-integrated hive. Yes, I build successfully , but >> then the problem comes:*I cannot use spark-sql anymore* because *spark-sql >> relies on the hive library and my spark is a no-hive build.* >> >> [appuser@ab-10-11-22-209 spark]$ spark-sql java.lang.ClassNotFoundException: >> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver at >> java.net.URLClassLoader.findClass(URLClassLoader.java:381) at >> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at >> java.lang.ClassLoader.loadClass(ClassLoader.java:357) at >> java.lang.Class.forName0(Native Method) at >> java.lang.Class.forName(Class.java:348) >> at org.apache.spark.util.Utils$.classForName(Utils.scala:225) at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ >> deploy$SparkSubmit$$runMain(SparkSubmit.scala:686) at >> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at >> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at >> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Failed to >> load main class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver. >> You need to build Spark with *-Phive and -Phive-thriftserver.* >> How can I build and setup spark and to make hive on spark >> Work properly and my spark-sql、pyspark and spark-shell work properly? >> >> >> I don’t know the relationship between *spark-integrated hive* and *original >> hive. *Below is the spark-integrated hive jars: >> >> hive-beeline-1.2.1.spark2.jarhive-cli-1.2.1.spark2.jarhive-exec-1.2.1.spark2.jarhive-jdbc-1.2.1.spark2.jarhive-metastore-1.2.1.spark2.jarspark-hive_2.11-2.0.0.jarspark-hive-thriftserver_2.11-2.0.0.jar >> >> >> It seems that Spark 2.0.0 relies on hive 1.2.1. >> >> >>
