After I changed the dependency to the following:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>1.2.1</version>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
I got the following error. Any idea on this? Thanks.
===============Caused by: java.lang.IncompatibleClassChangeError: Implementing
class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at
org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
at
org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
at
org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
... 6 moreFrom: [email protected]
To: [email protected]
CC: [email protected]; [email protected]
Subject: RE: spark 1.1.0 save data to hdfs failed
Date: Thu, 22 Jan 2015 17:05:26 -0800
Thanks. But after I replace the maven dependence from
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.5.0-cdh5.2.0</version>
<scope>provided</scope> <exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId> </exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId> </exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId> </exclusion>
</exclusions> </dependency>
to <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>1.0.4</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
the warning message is still shown up in the namenode log. Is there any other
thing I need to do?
Thanks.
Ey-Chih Chow
> From: [email protected]
> Date: Thu, 22 Jan 2015 22:34:22 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: [email protected]
> CC: [email protected]; [email protected]
>
> It means your client app is using Hadoop 2.x and your HDFS is Hadoop 1.x.
>
> On Thu, Jan 22, 2015 at 10:32 PM, ey-chih chow <[email protected]> wrote:
> > I looked into the namenode log and found this message:
> >
> > 2015-01-22 22:18:39,441 WARN org.apache.hadoop.ipc.Server: Incorrect header
> > or version mismatch from 10.33.140.233:53776 got version 9 expected version
> > 4
> >
> > What should I do to fix this?
> >
> > Thanks.
> >
> > Ey-Chih
> >
> > ________________________________
> > From: [email protected]
> > To: [email protected]
> > CC: [email protected]
> > Subject: RE: spark 1.1.0 save data to hdfs failed
> > Date: Wed, 21 Jan 2015 23:12:56 -0800
> >
> > The hdfs release should be hadoop 1.0.4.
> >
> > Ey-Chih Chow
> >
> > ________________________________
> > Date: Wed, 21 Jan 2015 16:56:25 -0800
> > Subject: Re: spark 1.1.0 save data to hdfs failed
> > From: [email protected]
> > To: [email protected]
> > CC: [email protected]
> >
> > What hdfs release are you using ?
> >
> > Can you check namenode log around time of error below to see if there is
> > some clue ?
> >
> > Cheers
> >
> > On Wed, Jan 21, 2015 at 4:51 PM, ey-chih chow <[email protected]> wrote:
> >
> > Hi,
> >
> > I used the following fragment of a scala program to save data to hdfs:
> >
> > contextAwareEvents
> > .map(e => (new AvroKey(e), null))
> > .saveAsNewAPIHadoopFile("hdfs://" + masterHostname + ":9000/ETL/output/"
> > + dateDir,
> > classOf[AvroKey[GenericRecord]],
> > classOf[NullWritable],
> > classOf[AvroKeyOutputFormat[GenericRecord]],
> > job.getConfiguration)
> >
> > But it failed with the following error messages. Is there any people who
> > can help? Thanks.
> >
> > Ey-Chih Chow
> >
> > =============================================
> >
> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> > Caused by: java.io.IOException: Failed on local exception:
> > java.io.EOFException; Host Details : local host is:
> > "ip-10-33-140-157/10.33.140.157"; destination host is:
> > "ec2-54-203-58-2.us-west-2.compute.amazonaws.com":9000;
> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1415)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> > at
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> > at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> > at
> > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> > at
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> > at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1925)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1079)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1075)
> > at
> > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1075)
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> > at
> > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:900)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:101)
> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> > ... 6 more
> > Caused by: java.io.EOFException
> > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > at
> > org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950)
> >
> > =======================================================
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-save-data-to-hdfs-failed-tp21305.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >
