Hi all,
The subject line says it all, ClusterDumper is writing to local file system
instead of HDFS.
After looking at the source
From the ClusterDumper class
if (this.outputFile == null) {
shouldClose = false;
writer = new OutputStreamWriter(System.out);
} else {
shouldClose = true;
if (outputFile.getName().startsWith("s3n://")) {
Path p = outputPath;
FileSystem fs = FileSystem.get(p.toUri(), conf);
writer = new OutputStreamWriter(fs.create(p), Charsets.UTF_8);
} else {
writer = Files.newWriter(this.outputFile, Charsets.UTF_8);
}
}
From the Files class
public static BufferedWriter newWriter(File file, Charset charset)
throws FileNotFoundException {
return new BufferedWriter(
new OutputStreamWriter(new FileOutputStream(file), charset));
}
So a few questions on the above.
1. Am I correct in saying if the outputFile starts with "s3n://" it writes to
the HDFS other wise it writes to the local FS?
2. If the above is true then what is the meaning of a URI starting with s3n://
3. Is there a way to force it to write to the HDFS even if the URI doesn't
start with s3n:// or am I going to have to modify ClusterDumper class myself?
