Hi all I have written a job that reads data from HBASE and writes to HDFS (fairly simple). While running the job, I noticed that a few of the tasks failed with the following error. Quick googling on the error suggests that its an unexplained error and is perhaps intermittent. What I am curious to know is why didn't Spark retry writing file to HDFS? It just shows it as failed job in Spark UI.
Error: java.io.IOException: All datanodes x.x.x.x:yyyy are bad. Aborting... org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1128) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486) Thanks, Aniket
