Re: EOFException when I list all files in hdfs directory

2014-07-25 Thread Bertrand Dechoux
Well, anyone can open an account on apache jira and post a new ticket/enhancement/issue/bug... Bertrand Dechoux On Fri, Jul 25, 2014 at 4:07 PM, Sparky wrote: > Thanks for the suggestion. I can confirm that my problem is I have files > with zero bytes. It's a known bug and is marked as a hig

Re: EOFException when I list all files in hdfs directory

2014-07-25 Thread Sparky
Thanks for the suggestion. I can confirm that my problem is I have files with zero bytes. It's a known bug and is marked as a high priority: https://issues.apache.org/jira/browse/SPARK-1960 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/EOFException-when

Re: EOFException when I list all files in hdfs directory

2014-07-25 Thread Akhil Das
Try without the * val avroRdd = sc.newAPIHadoopFile("hdfs://:8020//", classOf[AvroSequenceFileInputFormat[AvroKey[GenericRecord],NullWritable]], classOf[AvroKey[GenericRecord]], classOf[NullWritable]) avroRdd.collect() Thanks Best Regards On Fri, Jul 25, 2014 at 7:22 PM, Sparky wrote: > I'm

Re: EOFException when I list all files in hdfs directory

2014-07-25 Thread Sparky
I'm pretty sure my problem is related to this unresolved bug regarding files with size zero: https://issues.apache.org/jira/browse/SPARK-1960 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/EOFException-when-I-list-all-files-in-hdfs-directory-tp10648p10649.