I've had multiple jobs crash due to "java.io.IOException: unexpected exception type"; I've been running the 1.1 branch for some time and am now running the 1.1 release binaries. Note that I only use PySpark. I haven't kept detailed notes or the tracebacks around since there are other problems that have caused my greater grief (namely "key not found" errors).
For me the exception seems to occur non-deterministically, which is a bit interesting since the error message shows that the same stage has failed multiple times. Are you able to consistently re-produce the bug across multiple invocations at the same place? On Fri, Sep 26, 2014 at 6:11 AM, Arun Ahuja <[email protected]> wrote: > Has anyone else seen this erorr in task deserialization? The task is > processing a small amount of data and doesn't seem to have much data > hanging to the closure? I've only seen this with Spark 1.1 > > Job aborted due to stage failure: Task 975 in stage 8.0 failed 4 times, most > recent failure: Lost task 975.3 in stage 8.0 (TID 24777, host.com): > java.io.IOException: unexpected exception type > > java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1025) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:744) > >
