ClassNotFoundException generally means that jars are missing from your class path.
You probably need to link the spark jar to $HIVE_HOME/lib https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive On Tue, Nov 29, 2016 at 2:03 AM, Joaquin Alzola <joaquin.alz...@lebara.com> wrote: > Hi Guys > > > > No matter what I do that when I execute “select count(*) from employee” I > get the following output on the logs: > > It is quiet funny because if I put hive.execution.engine=mr the output is > correct. If I put hive.execution.engine=spark then I get the bellow errors. > > If I do the search directly through spark-shell it work great. > > +-------+ > > | _c0| > > +-------+ > > |1005635| > > +-------+ > > So there has to be a problem from hive to spark. > > > > Seems as the RPC(??) connection is not setup …. Can somebody guide me on > what to look for. > > spark.master=spark://172.16.173.31:7077 > > hive.execution.engine=spark > > spark.executor.extraClassPath /mnt/spark/lib/spark-1.6.2- > yarn-shuffle.jar:/mnt/hive/lib/hive-exec-2.0.1.jar > > > > Hive2.0.1à Spark 1.6.2 –> Hadoop – 2.6.5 à Scala 2.10 > > > > 2016-11-29T00:35:11,099 WARN [RPC-Handler-2]: rpc.RpcDispatcher > (RpcDispatcher.java:handleError(142)) - Received error > message:io.netty.handler.codec.DecoderException: > java.lang.NoClassDefFoundError: > org/apache/hive/spark/client/Job > > at io.netty.handler.codec.ByteToMessageDecoder.callDecode( > ByteToMessageDecoder.java:358) > > at io.netty.handler.codec.ByteToMessageDecoder.channelRead( > ByteToMessageDecoder.java:230) > > at io.netty.handler.codec.ByteToMessageCodec.channelRead( > ByteToMessageCodec.java:103) > > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > > at io.netty.channel.ChannelInboundHandlerAdapter.channelRead( > ChannelInboundHandlerAdapter.java:86) > > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > > at io.netty.channel.DefaultChannelPipeline.fireChannelRead( > DefaultChannelPipeline.java:846) > > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read( > AbstractNioByteChannel.java:131) > > at io.netty.channel.nio.NioEventLoop.processSelectedKey( > NioEventLoop.java:511) > > at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( > NioEventLoop.java:468) > > at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > NioEventLoop.java:382) > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > run(SingleThreadEventExecutor.java:111) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.NoClassDefFoundError: org/apache/hive/spark/client/ > Job > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > > at java.security.SecureClassLoader.defineClass( > SecureClassLoader.java:142) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:411) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:348) > > at org.apache.hive.com.esotericsoftware.kryo.util. > DefaultClassResolver.readName(DefaultClassResolver.java:154) > > at org.apache.hive.com.esotericsoftware.kryo.util. > DefaultClassResolver.readClass(DefaultClassResolver.java:133) > > at org.apache.hive.com.esotericsoftware.kryo.Kryo. > readClass(Kryo.java:670) > > at org.apache.hive.com.esotericsoftware.kryo. > serializers.ObjectField.read(ObjectField.java:118) > > at org.apache.hive.com.esotericsoftware.kryo. > serializers.FieldSerializer.read(FieldSerializer.java:551) > > at org.apache.hive.com.esotericsoftware.kryo.Kryo. > readClassAndObject(Kryo.java:790) > > at org.apache.hive.spark.client.rpc.KryoMessageCodec.decode( > KryoMessageCodec.java:97) > > at io.netty.handler.codec.ByteToMessageCodec$1.decode( > ByteToMessageCodec.java:42) > > at io.netty.handler.codec.ByteToMessageDecoder.callDecode( > ByteToMessageDecoder.java:327) > > ... 15 more > > Caused by: java.lang.ClassNotFoundException: org.apache.hive.spark.client. > Job > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > ... 39 more > This email is confidential and may be subject to privilege. If you are not > the intended recipient, please do not copy or disclose its content but > contact the sender immediately upon receipt. >
