Hey Jay :)
I tried "s3n" which uses the Jets3tNativeFileSystemStore, and the double
slash went away.
As far as I can see, it does look like a bug in hadoop-common; I'll file a
ticket for it.
Hope you are doing well, by the way!
PS:
Jets3tNativeFileSystemStore's implementation of pathToKey is:
======
private static String pathToKey(Path path) {
if (path.toUri().getScheme() != null &&
path.toUri().getPath().isEmpty()) {
// allow uris without trailing slash after bucket to refer to root,
// like s3n://mybucket
return "";
}
if (!path.isAbsolute()) {
throw new IllegalArgumentException("Path must be absolute: " + path);
}
String ret = path.toUri().getPath().substring(1); // remove initial
slash
if (ret.endsWith("/") && (ret.indexOf("/") != ret.length() - 1)) {
ret = ret.substring(0, ret.length() -1);
}
return ret;
}
======
whereas Jets3tFileSystemStore uses:
======
private String pathToKey(Path path) {
if (!path.isAbsolute()) {
throw new IllegalArgumentException("Path must be absolute: " + path);
}
return path.toUri().getPath();
}
======
ᐧ
On Tue, Dec 23, 2014 at 1:07 PM, Jay Vyas <jayunit100.apa...@gmail.com>
wrote:
> Hi enno. Might be worthwhile to cross post this on dev@hadoop...
> Obviously a simple spark way to test this would be to change the uri to
> write to hdfs:// or maybe you could do file:// , and confirm that the extra
> slash goes away.
>
> - if it's indeed a jets3t issue we should add a new unit test for this if
> the hcfs tests are passing for jets3tfilesystem, yet this error still
> exists.
>
> - To learn how to run HCFS tests against any FileSystem , see the wiki
> page : https://wiki.apache.org/hadoop/HCFS/Progress (see the July 14th
> entry on that page).
>
> - Is there another S3FileSystem implementation for AbstractFileSystem or
> is jets3t the only one? That would be a easy way to test this. And also a
> good workaround.
>
> I'm wondering, also why jets3tfilesystem is the AbstractFileSystem used by
> so many - is that the standard impl for storing using AbstractFileSystem
> interface?
>
> On Dec 23, 2014, at 6:06 AM, Enno Shioji <eshi...@gmail.com> wrote:
>
> Is anybody experiencing this? It looks like a bug in JetS3t to me, but
> thought I'd sanity check before filing an issue.
>
>
> ================
> I'm writing to S3 using ReceiverInputDStream#saveAsTextFiles with a S3 URL
> ("s3://fake-test/1234").
>
> The code does write to S3, but with double forward slashes (e.g.
> "s3://fake-test//1234/-1419334280000/".
>
> I did a debug and it seem like the culprit is
> Jets3tFileSystemStore#pathToKey(path), which returns "/fake-test/1234/..."
> for the input "s3://fake-test/1234/...". when it should hack off the first
> forward slash. However, I couldn't find any bug report for JetS3t for this.
>
> Am I missing something, or is this likely a JetS3t bug?
> ================
>
>
> ᐧ
>
>
