Is this deterministically reproducible? Can you try this on the latest master branch?
Would be great to turn debug logging and and dump the generated code. Also would be great to dump the array size at your line 314 in UnsafeRow (and whatever master branch's appropriate line is). On Fri, Jul 31, 2015 at 1:31 AM, james <yiaz...@gmail.com> wrote: > Another error： > 15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send > map output locations for shuffle 3 to bignode1:40443 > 15/07/31 16:15:28 INFO spark.MapOutputTrackerMaster: Size of output > statuses > for shuffle 3 is 583 bytes > 15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send > map output locations for shuffle 3 to bignode1:40474 > 15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send > map output locations for shuffle 3 to bignode2:34052 > 15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send > map output locations for shuffle 3 to bignode3:46929 > 15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send > map output locations for shuffle 3 to bignode3:50890 > 15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send > map output locations for shuffle 3 to bignode2:47795 > 15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send > map output locations for shuffle 3 to bignode4:35120 > 15/07/31 16:15:28 INFO scheduler.TaskSetManager: Finished task 32.0 in > stage > 151.0 (TID 1203) in 155 ms on bignode3 (1/50) > 15/07/31 16:15:28 INFO scheduler.TaskSetManager: Finished task 35.0 in > stage > 151.0 (TID 1204) in 157 ms on bignode2 (2/50) > 15/07/31 16:15:28 INFO scheduler.TaskSetManager: Finished task 8.0 in stage > 151.0 (TID 1196) in 168 ms on bignode3 (3/50) > 15/07/31 16:15:28 WARN scheduler.TaskSetManager: Lost task 46.0 in stage > 151.0 (TID 1184, bignode1): java.lang.NegativeArraySizeException > at > > org.apache.spark.sql.catalyst.expressions.UnsafeRow.getBinary(UnsafeRow.java:314) > at > > org.apache.spark.sql.catalyst.expressions.UnsafeRow.getUTF8String(UnsafeRow.java:297) > at SC$SpecificProjection.apply(Unknown Source) > at > > org.apache.spark.sql.catalyst.expressions.FromUnsafeProjection.apply(Projection.scala:152) > at > > org.apache.spark.sql.catalyst.expressions.FromUnsafeProjection.apply(Projection.scala:140) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at scala.collection.Iterator$$anon$10.next(Iterator.scala:312) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at > > org.apache.spark.shuffle.unsafe.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:148) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:71) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Came-across-Spark-SQL-hang-Error-issue-with-Spark-1-5-Tungsten-feature-tp13537p13538.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
