Hi, I am facing issue with writing RDD[X] to HDFS file path. X is a simple case class with variable time as primitive long.
When I run the driver program with - master as spark://<name_node>:7077 I get this - Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2744) at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1032) at org.apache.hadoop.io.Text.readString(Text.java:473) at org.apache.hadoop.io.Text.readString(Text.java:464) at org.apache.hadoop.io.WritableUtils.readEnum(WritableUtils.java:415) at org.apache.hadoop.mapreduce.TaskID.readFields(TaskID.java:223) at org.apache.hadoop.mapreduce.TaskAttemptID.readFields(TaskAttemptID.java:139) at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285) at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77) at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:45) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1205) at org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:41) Full stack trace - https://gist.github.com/imaxxs/650efd1cb367783897e3d37c1103512b The same code works fine with I run the driver program in local mode with master as - local[4] Can anyone please provide some pointers to debug this issue? I have tried many options - writing to the local file, writing to the HDFS file, writing in different formats e.t.c. Thanks for the help in advance, Mahendra about.me/mahendrakutare <https://about.me/mahendrakutare?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Only those who will risk going too far can possibly find out how far one can go.
