Thanks a lot, That worked great!
Thanks, Lars On Tue, Jun 3, 2014 at 12:17 PM, Yin Huai <huaiyin....@gmail.com> wrote: > Hello Lars, > > Can you check the value of "hive.security.authenticator.manager" in > hive-site.xml? I guess the value is > "org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator". This class was > introduced in hive 0.13, but Spark SQL is based on hive 0.12 right now. Can > you change the value of > "hive.security.authenticator.manager" to > "org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator" and try > again? > > Thanks, > > Yin > > > On Tue, Jun 3, 2014 at 10:44 AM, Lars Selsaas < > lars.sels...@thinkbiganalytics.com> wrote: > >> Hi, >> >> I've installed Spark 1.0.0 on a HDP 2.1 >> >> I moved the hive-site.xml file into the conf directory for Spark in an >> attempt to connect Spark with my existing Hive. >> >> Below is the full log from me starting Spark till I get the error. It >> seems to be building the assembly with hive so that part should be good. Is >> the Hive version HDP 2.1 not support or any other hints to what I'm doing >> wrong? >> >> >> [root@sandbox bin]# ./spark-shell >> >> Spark assembly has been built with Hive, including Datanucleus jars on >> classpath >> >> 14/06/03 10:30:21 INFO SecurityManager: Using Spark's default log4j >> profile: org/apache/spark/log4j-defaults.properties >> >> 14/06/03 10:30:21 INFO SecurityManager: Changing view acls to: root >> >> 14/06/03 10:30:21 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(root) >> >> 14/06/03 10:30:21 INFO HttpServer: Starting HTTP Server >> >> Welcome to >> >> ____ __ >> >> / __/__ ___ _____/ /__ >> >> _\ \/ _ \/ _ `/ __/ '_/ >> >> /___/ .__/\_,_/_/ /_/\_\ version 1.0.0 >> >> /_/ >> >> >> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java >> 1.7.0_45) >> >> Type in expressions to have them evaluated. >> >> Type :help for more information. >> >> 14/06/03 10:30:28 INFO SecurityManager: Changing view acls to: root >> >> 14/06/03 10:30:28 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(root) >> >> 14/06/03 10:30:29 INFO Slf4jLogger: Slf4jLogger started >> >> 14/06/03 10:30:29 INFO Remoting: Starting remoting >> >> 14/06/03 10:30:29 INFO Remoting: Remoting started; listening on addresses >> :[akka.tcp://sp...@sandbox.hortonworks.com:58176] >> >> 14/06/03 10:30:29 INFO Remoting: Remoting now listens on addresses: >> [akka.tcp://sp...@sandbox.hortonworks.com:58176] >> >> 14/06/03 10:30:29 INFO SparkEnv: Registering MapOutputTracker >> >> 14/06/03 10:30:29 INFO SparkEnv: Registering BlockManagerMaster >> >> 14/06/03 10:30:29 INFO DiskBlockManager: Created local directory at >> /tmp/spark-local-20140603103029-75e9 >> >> 14/06/03 10:30:29 INFO MemoryStore: MemoryStore started with capacity >> 294.9 MB. >> >> 14/06/03 10:30:29 INFO ConnectionManager: Bound socket to port 33645 with >> id = ConnectionManagerId(sandbox.hortonworks.com,33645) >> >> 14/06/03 10:30:29 INFO BlockManagerMaster: Trying to register BlockManager >> >> 14/06/03 10:30:29 INFO BlockManagerInfo: Registering block manager >> sandbox.hortonworks.com:33645 with 294.9 MB RAM >> >> 14/06/03 10:30:29 INFO BlockManagerMaster: Registered BlockManager >> >> 14/06/03 10:30:29 INFO HttpServer: Starting HTTP Server >> >> 14/06/03 10:30:29 INFO HttpBroadcast: Broadcast server started at >> http://10.0.2.15:43286 >> >> 14/06/03 10:30:29 INFO HttpFileServer: HTTP File server directory is >> /tmp/spark-fc0d099f-16ca-48e3-8c49-6f809bf67449 >> >> 14/06/03 10:30:29 INFO HttpServer: Starting HTTP Server >> >> 14/06/03 10:30:30 INFO SparkUI: Started SparkUI at >> http://sandbox.hortonworks.com:4040 >> >> 14/06/03 10:30:30 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> >> 14/06/03 10:30:31 INFO Executor: Using REPL class URI: >> http://10.0.2.15:34588 >> >> 14/06/03 10:30:31 INFO SparkILoop: Created spark context.. >> >> Spark context available as sc. >> >> >> scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) >> >> 14/06/03 10:30:53 INFO deprecation: mapred.input.dir.recursive is >> deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive >> >> 14/06/03 10:30:53 INFO deprecation: mapred.max.split.size is deprecated. >> Instead, use mapreduce.input.fileinputformat.split.maxsize >> >> 14/06/03 10:30:53 INFO deprecation: mapred.min.split.size is deprecated. >> Instead, use mapreduce.input.fileinputformat.split.minsize >> >> 14/06/03 10:30:53 INFO deprecation: mapred.min.split.size.per.rack is >> deprecated. Instead, use >> mapreduce.input.fileinputformat.split.minsize.per.rack >> >> 14/06/03 10:30:53 INFO deprecation: mapred.min.split.size.per.node is >> deprecated. Instead, use >> mapreduce.input.fileinputformat.split.minsize.per.node >> >> 14/06/03 10:30:53 INFO deprecation: mapred.reduce.tasks is deprecated. >> Instead, use mapreduce.job.reduces >> >> 14/06/03 10:30:53 INFO deprecation: >> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use >> mapreduce.reduce.speculative >> >> java.lang.RuntimeException: >> org.apache.hadoop.hive.ql.metadata.HiveException: >> java.lang.ClassNotFoundException: >> org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator >> >> at >> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:286) >> >> at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:166) >> >> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:12) >> >> at $iwC$$iwC$$iwC.<init>(<console>:17) >> >> at $iwC$$iwC.<init>(<console>:19) >> >> at $iwC.<init>(<console>:21) >> >> at <init>(<console>:23) >> >> at .<init>(<console>:27) >> >> at .<clinit>(<console>) >> >> at .<init>(<console>:7) >> >> at .<clinit>(<console>) >> >> at $print(<console>) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788) >> >> at >> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056) >> >> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614) >> >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645) >> >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609) >> >> at >> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796) >> >> at >> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841) >> >> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753) >> >> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601) >> >> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608) >> >> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611) >> >> at >> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936) >> >> at >> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884) >> >> at >> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884) >> >> at >> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >> >> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884) >> >> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982) >> >> at org.apache.spark.repl.Main$.main(Main.scala:31) >> >> at org.apache.spark.repl.Main.main(Main.scala) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292) >> >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) >> >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: >> java.lang.ClassNotFoundException: >> org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator >> >> at >> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:368) >> >> at >> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278) >> >> ... 41 more >> >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator >> >> at >> scala.tools.nsc.interpreter.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:83) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> >> at java.lang.Class.forName0(Native Method) >> >> at java.lang.Class.forName(Class.java:270) >> >> at >> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:361) >> >> ... 42 more >> >> -- >> >> Lars Selsaas >> >> Data Engineer >> >> Think Big Analytics <http://thinkbiganalytics.com> >> >> lars.sels...@thinkbiganalytics.com >> >> 650-537-5321 >> >> > -- Lars Selsaas Data Engineer Think Big Analytics <http://thinkbiganalytics.com> lars.sels...@thinkbiganalytics.com 650-537-5321
