Dear all,
I wrote a plugin codes for Hadoop, which calls the interfaces
in Cpp-built .so library. The plugin codes are written in java,
so I prepared a JNI class to encapsulate the C interfaces.
The java codes can be executed successfully when I compiled
it and run it standalone. However, it does not work when I embedded
in Hadoop. The exception shown out is (found in Hadoop logs):
------------ screen dump ---------------------
# grep myClass logs/* -r
logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
...
--------------------------------------------------------
It seems the library can not be loaded in Hadoop. My codes
(myClass.java) is like:
--------------- myClass.java ------------------
public class myClass
{
public static final Log LOG =
LogFactory.getLog("org.apache.hadoop.mapred.myClass");
public myClass() {
try {
//System.setProperty("java.library.path",
"/usr/local/lib");
/* The above line does not work, so I have to
do something
* like following line.
*/
addDir(new String("/usr/local/lib"));
System.loadLibrary("myclass");
}
catch(UnsatisfiedLinkError e) {
LOG.info( "Cannot load library:\n " +
e.toString() );
}
catch(IOException ioe) {
LOG.info( "IO error:\n " +
ioe.toString() );
}
}
/* Since the System.setProperty() does not work, I have to add the
following
* function to force the path is added in java.library.path
*/
public static void addDir(String s) throws IOException {
try {
Field field =
ClassLoader.class.getDeclaredField("usr_paths");
field.setAccessible(true);
String[] paths = (String[])field.get(null);
for (int i = 0; i < paths.length; i++) {
if (s.equals(paths[i])) {
return;
}
}
String[] tmp = new String[paths.length+1];
System.arraycopy(paths,0,tmp,0,paths.length);
tmp[paths.length] = s;
field.set(null,tmp);
} catch (IllegalAccessException e) {
throw new IOException("Failed to get
permissions to set library path");
} catch (NoSuchFieldException e) {
throw new IOException("Failed to get field
handle to set library path");
}
}
public native int myClassfsMount(String subsys);
public native int myClassfsUmount(String subsys);
}
--------------------------------------------------------
I don't know what missed in my codes and am wondering whether there are any
rules in Hadoop I should obey if I want to achieve my target.
FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
services from a daemon. I would better if this design did not cause the fail in
my codes.
Any comments?
Thanks in advance,
Ian
