Hello,
in the Commons VFS project we have a Provider for HDFS. This
Provider is unit-tested with the help of the excelent MiniDFSCluster
component.
A while back I enabled automated testing on Windows (cause it worked
for me), but I did not realize it only works with cygwin (namely
ls.exe) in the Path. This was some 1.2 version and it used the ls
command to read the owner.
We currently switch the dependency to 2.4.0 and had the hope that it better
(i.e.
without binary dependencies) works in this scenario - as we want to run
the maven tests without pre-installing a tool chain on Windows.
However it got actually worse, it warns about a missing winutils.exe
and it requires the hadoop.dll. The first location where it fails is
testing writeability for the name node. And unlike all the comments in
FileUtil/NativeIO it does not cleanly fall back to java portable
methods.
I think the access/access0 method can easily be fixed, but before I work on
this patch, does it make sense to go in this direction or will I have to fix
some other, more complicated platform dependencies. My goal would be to start
MiniDFSCluster inside surefire from maven without the need for complicated
external toolchain setup.
Is actually anybody caring about making this possible?
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Time elapsed:
2.291 sec <<< ERROR!
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at
org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:879)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:975)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:856)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640)
at
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95)
Greetings
Bernd
PS - in case you want to follow the VFS progress:
* https://issues.apache.org/jira/browse/VFS-530 - tracks 2.4 migration
* https://issues.apache.org/jira/browse/VFS-529 - the broken windows tests
* https://issues.apache.org/jira/browse/VFS-526 - failed enable of win tests
