Fwd: Saving large textfile

2016-04-24 Thread Simon Hafner
2016-04-24 13:38 GMT+02:00 Stefan Falk : > sc.parallelize(cfile.toString() > .split("\n"), 1) Try `sc.textFile(pathToFile)` instead. >java.io.IOException: Broken pipe >at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >at sun.nio.ch.SocketDispatcher.write(SocketD

Saving large textfile

2016-04-24 Thread Stefan Falk
I try to save a large text file of a approx. 5GB sc.parallelize(cfile.toString() .split("\n"), 1) .saveAsTextFile(new Path(path+".cs", "data").toUri.toString) but I keep getting java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method)