We are hitting the same issue on Spark 1.6.1 with tungsten enabled, kryo enabled & sort based shuffle.
Did you find a resolution? On Sat, Apr 9, 2016 at 6:31 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Not much. > > So no chance of different snappy version ? > > On Fri, Apr 8, 2016 at 1:26 PM, Nicolas Tilmans <ntilm...@gmail.com> > wrote: > >> Hi Ted, >> >> The Spark version is 1.6.1, a nearly identical set of operations does >> fine on smaller datasets. It's just a few joins then a groupBy and a count >> in pyspark.sql on a Spark DataFrame. >> >> Any ideas? >> >> Nicolas >> >> On Fri, Apr 8, 2016 at 1:13 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Did you encounter similar error on a smaller dataset ? >>> >>> Which release of Spark are you using ? >>> >>> Is it possible you have an incompatible snappy version somewhere in your >>> classpath ? >>> >>> Thanks >>> >>> On Fri, Apr 8, 2016 at 12:36 PM, entee <ntilm...@gmail.com> wrote: >>> >>>> I'm trying to do a relatively large join (0.5TB shuffle read/write) and >>>> just >>>> calling a count (or show) on the dataframe fails to complete, getting >>>> to the >>>> last task before failing: >>>> >>>> Py4JJavaError: An error occurred while calling o133.count. >>>> : org.apache.spark.SparkException: Job aborted due to stage failure: >>>> Task 5 >>>> in stage 11.0 failed 4 times, most recent failure: Lost task 5.3 in >>>> stage >>>> 11.0 (TID 7836, xxxx.com): java.io.IOException: failed to uncompress >>>> the >>>> chunk: PARSING_ERROR(2) >>>> >>>> (full stack trace below) >>>> >>>> I'm not sure why this would happen, any ideas about whether our >>>> configuration is off or how to fix this? >>>> >>>> Nicolas >>>> >>>> >>>> >>>> Py4JJavaError: An error occurred while calling o133.count. >>>> : org.apache.spark.SparkException: Job aborted due to stage failure: >>>> Task 5 >>>> in stage 11.0 failed 4 times, most recent failure: Lost task 5.3 in >>>> stage >>>> 11.0 (TID 7836, xxxx.com): java.io.IOException: failed to uncompress >>>> the >>>> chunk: PARSING_ERROR(2) >>>> at >>>> >>>> org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:361) >>>> at >>>> org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158) >>>> at >>>> org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142) >>>> at >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246) >>>> at >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286) >>>> at >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345) >>>> at java.io.DataInputStream.read(DataInputStream.java:149) >>>> at >>>> org.spark-project.guava.io.ByteStreams.read(ByteStreams.java:899) >>>> at >>>> org.spark-project.guava.io.ByteStreams.readFully(ByteStreams.java:733) >>>> at >>>> >>>> org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:119) >>>> at >>>> >>>> org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:102) >>>> at scala.collection.Iterator$$anon$12.next(Iterator.scala:397) >>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:370) >>>> at >>>> >>>> org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) >>>> at >>>> >>>> org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) >>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:370) >>>> at >>>> >>>> org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:168) >>>> at >>>> org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:90) >>>> at >>>> org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64) >>>> at >>>> >>>> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728) >>>> at >>>> >>>> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728) >>>> at >>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>>> at >>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>>> at >>>> >>>> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88) >>>> at >>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>>> at >>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>>> at >>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>>> at >>>> >>>> org.apache.spark.rdd.CoalescedRDD$$anonfun$compute$1.apply(CoalescedRDD.scala:96) >>>> at >>>> >>>> org.apache.spark.rdd.CoalescedRDD$$anonfun$compute$1.apply(CoalescedRDD.scala:95) >>>> at >>>> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:396) >>>> at >>>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369) >>>> at >>>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369) >>>> at >>>> >>>> org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:163) >>>> at >>>> >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) >>>> at >>>> >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:89) >>>> at >>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> Driver stacktrace: >>>> at >>>> org.apache.spark.scheduler.DAGScheduler.org >>>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) >>>> at >>>> >>>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >>>> at >>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) >>>> at scala.Option.foreach(Option.scala:257) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599) >>>> at >>>> >>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588) >>>> at >>>> org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) >>>> at >>>> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620) >>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832) >>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845) >>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858) >>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929) >>>> at >>>> org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927) >>>> at >>>> >>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) >>>> at >>>> >>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) >>>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) >>>> at org.apache.spark.rdd.RDD.collect(RDD.scala:926) >>>> at >>>> >>>> org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:166) >>>> at >>>> >>>> org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:174) >>>> at >>>> >>>> org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1499) >>>> at >>>> >>>> org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1499) >>>> at >>>> >>>> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56) >>>> at >>>> org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:2086) >>>> at >>>> org.apache.spark.sql.DataFrame.org >>>> $apache$spark$sql$DataFrame$$execute$1(DataFrame.scala:1498) >>>> at >>>> org.apache.spark.sql.DataFrame.org >>>> $apache$spark$sql$DataFrame$$collect(DataFrame.scala:1505) >>>> at >>>> >>>> org.apache.spark.sql.DataFrame$$anonfun$count$1.apply(DataFrame.scala:1515) >>>> at >>>> >>>> org.apache.spark.sql.DataFrame$$anonfun$count$1.apply(DataFrame.scala:1514) >>>> at >>>> org.apache.spark.sql.DataFrame.withCallback(DataFrame.scala:2099) >>>> at org.apache.spark.sql.DataFrame.count(DataFrame.scala:1514) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> at >>>> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:497) >>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) >>>> at >>>> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) >>>> at py4j.Gateway.invoke(Gateway.java:259) >>>> at >>>> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) >>>> at py4j.commands.CallCommand.execute(CallCommand.java:79) >>>> at py4j.GatewayConnection.run(GatewayConnection.java:209) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.io.IOException: failed to uncompress the chunk: >>>> PARSING_ERROR(2) >>>> at >>>> >>>> org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:361) >>>> at >>>> org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158) >>>> at >>>> org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142) >>>> at >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246) >>>> at >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286) >>>> at >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345) >>>> at java.io.DataInputStream.read(DataInputStream.java:149) >>>> at >>>> org.spark-project.guava.io.ByteStreams.read(ByteStreams.java:899) >>>> at >>>> org.spark-project.guava.io.ByteStreams.readFully(ByteStreams.java:733) >>>> at >>>> >>>> org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:119) >>>> at >>>> >>>> org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:102) >>>> at scala.collection.Iterator$$anon$12.next(Iterator.scala:397) >>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:370) >>>> at >>>> >>>> org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) >>>> at >>>> >>>> org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) >>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:370) >>>> at >>>> >>>> org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:168) >>>> at >>>> org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:90) >>>> at >>>> org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64) >>>> at >>>> >>>> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728) >>>> at >>>> >>>> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728) >>>> at >>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>>> at >>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>>> at >>>> >>>> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88) >>>> at >>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>>> at >>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>>> at >>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>>> at >>>> >>>> org.apache.spark.rdd.CoalescedRDD$$anonfun$compute$1.apply(CoalescedRDD.scala:96) >>>> at >>>> >>>> org.apache.spark.rdd.CoalescedRDD$$anonfun$compute$1.apply(CoalescedRDD.scala:95) >>>> at >>>> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:396) >>>> at >>>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369) >>>> at >>>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369) >>>> at >>>> >>>> org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:163) >>>> at >>>> >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) >>>> at >>>> >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:89) >>>> at >>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> ... 1 more >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/DataFrame-job-fails-on-parsing-error-help-tp26724.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >> >
