The solution mentioned by Timo works well with a standalone Flink cluster but might not work with a YARN or Mesos cluster. An alternative is to have your Java library contain the native library within itself, and to extract it to a temporary directory before calling `System.loadLibrary(...)`. Note that you lose the advantages of using the native OS's packaging system (e.g. security patches, dependency management). The TensorFlow Java library demonstrates the technique:
https://github.com/tensorflow/tensorflow/blob/v1.2.1/tensorflow/java/src/main/java/org/tensorflow/NativeLibrary.java -Eron On Tue, Jul 18, 2017 at 8:02 AM, Timo Walther <twal...@apache.org> wrote: > Hi Mike, > > do you run Flink locally or in a cluster? You have to make sure that VM > argument -Djava.library.path is set for all Flink JVMs. Job Manager and > Task Managers might run in separate JVMs. Make also sure that the library > is accessible from all node. I don't know what happens if the file is > accessed by multiple processes/threads at the same time. It might also > important where you put the static { ... } loading. It should be in the > Function, because these classes get deserialized on the TaskManager. > > I hope this helps. > > Timo > > > Am 17.07.17 um 21:30 schrieb Mike Accola: > > I am new Flink user just trying to learn a little bit. I am trying to >> incorporate an existing C++ library into a new Flink application. I am >> seeing some strange behavior when trying to link in the native (C++) >> library using java via JNI. >> I am running this on Linux (RHEL6) >> I can run my application once without error. Sometimes it will run >> successfully a 2nd or 3rd time. However, eventually on a subsequent run, >> I get an exception about the the native library not being found: >> java.lang.UnsatisfiedLinkError: no dummy2native in java.library.path >> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) >> at java.lang.Runtime.loadLibrary0(Runtime.java:870) >> at java.lang.System.loadLibrary(System.java:1122) >> at com.att.flink.tdata.spss.TinyLoader.loadNative(Dummy2.java: >> 10) >> For debugging purposes for now, my native library does not have any >> external references. It really contains 1 method that essentially does >> nothing. >> The behavior seems to indicate that there is some kind of cleanup being >> done that "unloads" the native library. I suspect this is somehow related >> to Flink's implementation of its library cache manager, but I have not >> been able to prove this yet. >> A few more details: >> - I have a c++ library libdummy2native.so that contains a method that >> can >> be invoked via JNI. >> - I have a jar containing a class, called Dummy2. The Dummy2 constructor >> will invoke the JNI method. >> - The libdummy2native.so library is invoked with System.loadLibrary() like >> this: >> static {System.loadLibrary("dummy2native"); } >> - In my simple Flink application, I have extended the ProcessFunction >> class. Within this class, I have overriden processElement method that >> declares a Dummy2 object. >> - The Dummy2 class can be called and invoked without error when used in a >> standalone java program. >> Any thoughts or ideas on what to try next would be appreciated. >> Initially, >> I'd be happy to be able to just explain this behavior. I will worry about >> fixing it afterwards. >> Thanks. >> >> >> >> >> >
