Look in the worker logs and see whats going on.
Thanks
Best Regards
On Tue, Jul 14, 2015 at 4:02 PM, Arthur Chan <arthur.hk.c...@gmail.com>
wrote:
> Hi,
>
> I use Spark 1.4. When saving the model to HDFS, I got error?
>
> Please help!
> Regards
>
>
>
> my scala command:
> sc.makeRDD(model.clusterCenters,10).saveAsObjectFile("/tmp/tweets/model")
>
> The error log:
>
> 15/07/14 18:27:40 INFO SequenceFileRDDFunctions: Saving as sequence file
> of type (NullWritable,BytesWritable)
>
> 15/07/14 18:27:40 INFO SparkContext: Starting job: saveAsObjectFile at
> <console>:45
>
> 15/07/14 18:27:40 INFO DAGScheduler: Got job 110 (saveAsObjectFile at
> <console>:45) with 10 output partitions (allowLocal=false)
>
> 15/07/14 18:27:40 INFO DAGScheduler: Final stage: ResultStage
> 174(saveAsObjectFile at <console>:45)
>
> 15/07/14 18:27:40 INFO DAGScheduler: Parents of final stage: List()
>
> 15/07/14 18:27:40 INFO DAGScheduler: Missing parents: List()
>
> 15/07/14 18:27:40 INFO DAGScheduler: Submitting ResultStage 174
> (MapPartitionsRDD[258] at saveAsObjectFile at <console>:45), which has no
> missing parents
>
> 15/07/14 18:27:40 INFO MemoryStore: ensureFreeSpace(135360) called with
> curMem=14724380, maxMem=280248975
>
> 15/07/14 18:27:40 INFO MemoryStore: Block broadcast_256 stored as values
> in memory (estimated size 132.2 KB, free 253.1 MB)
>
> 15/07/14 18:27:40 INFO MemoryStore: ensureFreeSpace(46231) called with
> curMem=14859740, maxMem=280248975
>
> 15/07/14 18:27:40 INFO MemoryStore: Block broadcast_256_piece0 stored as
> bytes in memory (estimated size 45.1 KB, free 253.1 MB)
>
> 15/07/14 18:27:40 INFO BlockManagerInfo: Added broadcast_256_piece0 in
> memory on localhost:52681 (size: 45.1 KB, free: 263.1 MB)
>
> 15/07/14 18:27:40 INFO SparkContext: Created broadcast 256 from broadcast
> at DAGScheduler.scala:874
>
> 15/07/14 18:27:40 INFO DAGScheduler: Submitting 10 missing tasks from
> ResultStage 174 (MapPartitionsRDD[258] at saveAsObjectFile at <console>:45)
>
> 15/07/14 18:27:40 INFO TaskSchedulerImpl: Adding task set 174.0 with 10
> tasks
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 0.0 in stage 174.0
> (TID 4513, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 1.0 in stage 174.0
> (TID 4514, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 2.0 in stage 174.0
> (TID 4515, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 3.0 in stage 174.0
> (TID 4516, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 4.0 in stage 174.0
> (TID 4517, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 5.0 in stage 174.0
> (TID 4518, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 6.0 in stage 174.0
> (TID 4519, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 7.0 in stage 174.0
> (TID 4520, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 8.0 in stage 174.0
> (TID 4521, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 9.0 in stage 174.0
> (TID 4522, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO Executor: Running task 0.0 in stage 174.0 (TID 4513)
>
> 15/07/14 18:27:40 INFO Executor: Running task 1.0 in stage 174.0 (TID 4514)
>
> 15/07/14 18:27:40 INFO Executor: Running task 2.0 in stage 174.0 (TID 4515)
>
> 15/07/14 18:27:40 INFO Executor: Running task 3.0 in stage 174.0 (TID 4516)
>
> 15/07/14 18:27:40 INFO Executor: Running task 4.0 in stage 174.0 (TID 4517)
>
> 15/07/14 18:27:40 INFO Executor: Running task 5.0 in stage 174.0 (TID 4518)
>
> 15/07/14 18:27:40 INFO Executor: Running task 6.0 in stage 174.0 (TID 4519)
>
> 15/07/14 18:27:40 INFO Executor: Running task 7.0 in stage 174.0 (TID 4520)
>
> 15/07/14 18:27:40 INFO Executor: Running task 8.0 in stage 174.0 (TID 4521)
>
> 15/07/14 18:27:40 ERROR Executor: Exception in task 1.0 in stage 174.0
> (TID 4514)
>
> java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/07/14 18:27:40 ERROR Executor: Exception in task 3.0 in stage 174.0
> (TID 4516)
>
> java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/07/14 18:27:40 ERROR Executor: Exception in task 5.0 in stage 174.0
> (TID 4518)
>
> java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/07/14 18:27:40 INFO Executor: Running task 9.0 in stage 174.0 (TID 4522)
>
> 15/07/14 18:27:40 WARN TaskSetManager: Lost task 1.0 in stage 174.0 (TID
> 4514, localhost): java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> 15/07/14 18:27:40 ERROR TaskSetManager: Task 1 in stage 174.0 failed 1
> times; aborting job
>
> 15/07/14 18:27:40 INFO TaskSetManager: Lost task 3.0 in stage 174.0 (TID
> 4516) on executor localhost: java.lang.IllegalStateException (unread block
> data) [duplicate 1]
>
> 15/07/14 18:27:40 INFO TaskSchedulerImpl: Cancelling stage 174
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 6.0 in
> stage 174.0 (TID 4519)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 0.0 in
> stage 174.0 (TID 4513)
>
> 15/07/14 18:27:40 INFO TaskSchedulerImpl: Stage 174 was cancelled
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 7.0 in
> stage 174.0 (TID 4520)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 4.0 in
> stage 174.0 (TID 4517)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 8.0 in
> stage 174.0 (TID 4521)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 9.0 in
> stage 174.0 (TID 4522)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 2.0 in
> stage 174.0 (TID 4515)
>
> 15/07/14 18:27:40 INFO DAGScheduler: ResultStage 174 (saveAsObjectFile at
> <console>:45) failed in 0.005 s
>
> 15/07/14 18:27:40 INFO DAGScheduler: Job 110 failed: saveAsObjectFile at
> <console>:45, took 0.016422 s
>
> 15/07/14 18:27:40 ERROR Executor: Exception in task 8.0 in stage 174.0
> (TID 4521)
>
> java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Lost task 5.0 in stage 174.0 (TID
> 4518) on executor localhost: java.lang.IllegalStateException (unread block
> data) [duplicate 2]
>
> 15/07/14 18:27:40 INFO TaskSetManager: Lost task 8.0 in stage 174.0 (TID
> 4521) on executor localhost: java.lang.IllegalStateException (unread block
> data) [duplicate 3]
>
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
> in stage 174.0 failed 1 times, most recent failure: Lost task 1.0 in stage
> 174.0 (TID 4514, localhost): java.lang.IllegalStateException: unread block
> data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> Driver stacktrace:
>
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1266)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1257)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1256)
>
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1256)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
>
> at scala.Option.foreach(Option.scala:257)
>
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
>
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1450)
>
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1411)
>
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>
