Ok what was wrong was that the spark-env did not contain the HADOOP_CONF_DIR properly set to /etc/hadoop/conf/ With that fixed, this issue is gone, but I can't seem to get Spark SQL 1.4.0 with Hive working on CDH 5.3 or 5.4 : Using this command line : IPYTHON=1 /.../spark-1.4.0-bin-hadoop2.4/bin/pyspark --master yarn-client --driver-class-path `hadoop classpath`
I end up with this issue : : java.lang.NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class; at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<init>(ScalaNumberDeserializersModule.scala:49) at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<clinit>(ScalaNumberDeserializersModule.scala) at com.fasterxml.jackson.module.scala.deser.ScalaNumberDeserializersModule$class.$init$(ScalaNumberDeserializersModule.scala:61) at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:19) at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:35) at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala) at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:78) at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87) at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:118) at org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:125) at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1255) at org.apache.spark.sql.DataFrame.head(DataFrame.scala:1189) at org.apache.spark.sql.DataFrame.take(DataFrame.scala:1248) at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) at py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:207) at java.lang.Thread.run(Thread.java:745) This seems to be related to this Jira Issue : https://issues.apache.org/jira/browse/SPARK-8332 This is a blocker for me to deploy a Spark dataframe based app on an existing cluster, any input regarding how to create a proper classpath would be great. Regards, Olivier. Le mer. 17 juin 2015 à 11:37, Olivier Girardot < o.girar...@lateral-thoughts.com> a écrit : > Hi everyone, > After copying the hive-site.xml from a CDH5 cluster, I can't seem to > connect to the hive metastore using spark-shell, here's a part of the stack > trace I get : > > 15/06/17 04:41:57 ERROR TSaslTransport: SASL negotiation failure > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)] > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) > at > org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) > > The user has a non-expired ticket, I can execute hadoop fs -ls, all in all > I should have access to this. > I am stuck with this issue on Spark 1.4.0, did not try a version before... > > Any guess regarding what might be wrong ? > > Regards, > > Olivier. >
