"Found class org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected" is the classic error meaning "you compiled against Hadoop 1, but are running against Hadoop 2"
I think you need to override the hadoop-client artifact that Spark depends on to be a Hadoop 2.x version. On Tue, Jun 3, 2014 at 5:23 PM, toivoa <toivo....@gmail.com> wrote: > Hi > > Set up project under Eclipse using Maven: > > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-core_2.10</artifactId> > <version>1.0.0</version> > </dependency> > > Simple example fails: > > def main(args: Array[String]): Unit = { > > val conf = new SparkConf() > .setMaster("local") > .setAppName("CountingSheep") > .set("spark.executor.memory", "1g") > > val sc = new SparkContext(conf) > > val indir = "src/main/resources/testdata" > val files = sc.wholeTextFiles(indir, 10) > for( pair <- files) > println(pair._1 + " = " + pair._2) > > > 14/06/03 19:20:34 ERROR executor.Executor: Exception in task ID 0 > java.lang.RuntimeException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:164) > at > org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.<init>(CombineFileRecordReader.java:126) > at > org.apache.spark.input.WholeTextFileInputFormat.createRecordReader(WholeTextFileInputFormat.scala:44) > at > org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:111) > at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:99) > at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:61) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:229) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) > at org.apache.spark.scheduler.Task.run(Task.scala:51) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:155) > ... 13 more > Caused by: java.lang.IncompatibleClassChangeError: Found class > org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected > at > org.apache.spark.input.WholeTextFileRecordReader.<init>(WholeTextFileRecordReader.scala:40) > ... 18 more > > Any idea? > > thanks > toivo > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/wholeTextFiles-java-lang-IncompatibleClassChangeError-Found-class-org-apache-hadoop-mapreduce-TaskAtd-tp6818.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.