If you underlying filesystem is HDFS, you need to use HDFS APIs. A google search brought up this link which appears reasonable.
http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample If you want to use java.io APIs, you have to make sure your filesystem is accessible from all nodes in your cluster. You did not mention what errors you get with your code. They may mean something. On Mon, Sep 15, 2014 at 9:51 AM, rapelly kartheek <kartheek.m...@gmail.com> wrote: > Can you please direct me to the right way of doing this. > > On Mon, Sep 15, 2014 at 10:18 PM, rapelly kartheek < > kartheek.m...@gmail.com> wrote: > >> I came across these APIs in one the scala tutorials over the net. >> >> On Mon, Sep 15, 2014 at 10:14 PM, Mohit Jaggi <mohitja...@gmail.com> >> wrote: >> >>> But the above APIs are not for HDFS. >>> >>> On Mon, Sep 15, 2014 at 9:40 AM, rapelly kartheek < >>> kartheek.m...@gmail.com> wrote: >>> >>>> Yes. I have HDFS. My cluster has 5 nodes. When I run the above >>>> commands, I see that the file gets created in the master node. But, there >>>> wont be any data written to it. >>>> >>>> >>>> On Mon, Sep 15, 2014 at 10:06 PM, Mohit Jaggi <mohitja...@gmail.com> >>>> wrote: >>>> >>>>> Is this code running in an executor? You need to make sure the file is >>>>> accessible on ALL executors. One way to do that is to use a distributed >>>>> filesystem like HDFS or GlusterFS. >>>>> >>>>> On Mon, Sep 15, 2014 at 8:51 AM, rapelly kartheek < >>>>> kartheek.m...@gmail.com> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> I am trying to perform some read/write file operations in spark. >>>>>> Somehow I am neither able to write to a file nor read. >>>>>> >>>>>> import java.io._ >>>>>> >>>>>> val writer = new PrintWriter(new File("test.txt" )) >>>>>> >>>>>> writer.write("Hello Scala") >>>>>> >>>>>> >>>>>> Can someone please tell me how to perform file I/O in spark. >>>>>> >>>>>> >>>>> >>>> >>> >> >
