Thanks Steve and Kant. Apologies for late reply as I was out for vacation.
Got it working. For other users:
def loadResources() {
System.loadLibrary("foolib")
val MyInstance = new MyClass
val retstr = MyInstance.foo("mystring") // method trying to invoke
}
val conf = new
SparkConf().setAppName("SimpleApplication").set("SPARK_LIBRARY_PATH",
"/lib/location")
val sc = new SparkContext(conf)
sc.parallelize(1 to 10, 2).mapPartitions ( iter => {
MySimpleApp.loadResources()
iter
}).count
Regards,
Vineet
On Sun, Nov 27, 2016 at 2:15 PM, Steve Loughran <[email protected]>
wrote:
>
> On 27 Nov 2016, at 02:55, kant kodali <[email protected]> wrote:
>
> I would say instead of LD_LIBRARY_PATH you might want to use java.library.
> path
>
> in the following way
>
> java -Djava.library.path=/path/to/my/library or pass java.library.path
> along with spark-submit
>
>
> This is only going to set up paths on the submitting system; to load JNI
> code in the executors, the binary needs to be sent to far end and then put
> on the Java load path there.
>
> Copy the relevant binary to somewhere on the PATH of the destination
> machine. Do that and you shouldn't have to worry about other JVM options,
> (though it's been a few years since I did any JNI).
>
> One trick: write a simple main() object/entry point which calls the JNI
> method, and doesn't attempt to use any spark libraries; have it log any
> exception and return an error code if the call failed. This will let you
> use it as a link test after deployment: if you can't run that class then
> things are broken, before you go near spark
>
>
> On Sat, Nov 26, 2016 at 6:44 PM, Gmail <[email protected]> wrote:
>
>> Maybe you've already checked these out. Some basic questions that come to
>> my mind are:
>> 1) is this library "foolib" or "foo-C-library" available on the worker
>> node?
>> 2) if yes, is it accessible by the user/program (rwx)?
>>
>> Thanks,
>> Vasu.
>>
>> On Nov 26, 2016, at 5:08 PM, kant kodali <[email protected]> wrote:
>>
>> If it is working for standalone program I would think you can apply the
>> same settings across all the spark worker and client machines and give
>> that a try. Lets start with that.
>>
>> On Sat, Nov 26, 2016 at 11:59 AM, vineet chadha <[email protected]>
>> wrote:
>>
>>> Just subscribed to Spark User. So, forwarding message again.
>>>
>>> On Sat, Nov 26, 2016 at 11:50 AM, vineet chadha <[email protected]>
>>> wrote:
>>>
>>>> Thanks Kant. Can you give me a sample program which allows me to call
>>>> jni from executor task ? I have jni working in standalone program in
>>>> scala/java.
>>>>
>>>> Regards,
>>>> Vineet
>>>>
>>>> On Sat, Nov 26, 2016 at 11:43 AM, kant kodali <[email protected]>
>>>> wrote:
>>>>
>>>>> Yes this is a Java JNI question. Nothing to do with Spark really.
>>>>>
>>>>> java.lang.UnsatisfiedLinkError typically would mean the way you
>>>>> setup LD_LIBRARY_PATH is wrong unless you tell us that it is working
>>>>> for other cases but not this one.
>>>>>
>>>>> On Sat, Nov 26, 2016 at 11:23 AM, Reynold Xin <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> That's just standard JNI and has nothing to do with Spark, does it?
>>>>>>
>>>>>>
>>>>>> On Sat, Nov 26, 2016 at 11:19 AM, vineet chadha <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Thanks Reynold for quick reply.
>>>>>>>
>>>>>>> I have tried following:
>>>>>>>
>>>>>>> class MySimpleApp {
>>>>>>> // ---Native methods
>>>>>>> @native def fooMethod (foo: String): String
>>>>>>> }
>>>>>>>
>>>>>>> object MySimpleApp {
>>>>>>> val flag = false
>>>>>>> def loadResources() {
>>>>>>> System.loadLibrary("foo-C-library")
>>>>>>> val flag = true
>>>>>>> }
>>>>>>> def main() {
>>>>>>> sc.parallelize(1 to 10).mapPartitions ( iter => {
>>>>>>> if(flag == false){
>>>>>>> MySimpleApp.loadResources()
>>>>>>> val SimpleInstance = new MySimpleApp
>>>>>>> }
>>>>>>> SimpleInstance.fooMethod ("fooString")
>>>>>>> iter
>>>>>>> })
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> I don't see way to invoke fooMethod which is implemented in
>>>>>>> foo-C-library. Is I am missing something ? If possible, can you point
>>>>>>> me to
>>>>>>> existing implementation which i can refer to.
>>>>>>>
>>>>>>> Thanks again.
>>>>>>>
>>>>>>> ~
>>>>>>>
>>>>>>> On Fri, Nov 25, 2016 at 3:32 PM, Reynold Xin <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> bcc dev@ and add user@
>>>>>>>>
>>>>>>>>
>>>>>>>> This is more a user@ list question rather than a dev@ list
>>>>>>>> question. You can do something like this:
>>>>>>>>
>>>>>>>> object MySimpleApp {
>>>>>>>> def loadResources(): Unit = // define some idempotent way to load
>>>>>>>> resources, e.g. with a flag or lazy val
>>>>>>>>
>>>>>>>> def main() = {
>>>>>>>> ...
>>>>>>>>
>>>>>>>> sc.parallelize(1 to 10).mapPartitions { iter =>
>>>>>>>> MySimpleApp.loadResources()
>>>>>>>>
>>>>>>>> // do whatever you want with the iterator
>>>>>>>> }
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Nov 25, 2016 at 2:33 PM, vineet chadha <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am trying to invoke C library from the Spark Stack using JNI
>>>>>>>>> interface (here is sample application code)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> class SimpleApp {
>>>>>>>>> // ---Native methods
>>>>>>>>> @native def foo (Top: String): String
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> object SimpleApp {
>>>>>>>>> def main(args: Array[String]) {
>>>>>>>>>
>>>>>>>>> val conf = new SparkConf().setAppName("Simple
>>>>>>>>> Application").set("SPARK_LIBRARY_PATH", "lib")
>>>>>>>>> val sc = new SparkContext(conf)
>>>>>>>>> System.loadLibrary("foolib")
>>>>>>>>> //instantiate the class
>>>>>>>>> val SimpleAppInstance = new SimpleApp
>>>>>>>>> //String passing - Working
>>>>>>>>> val ret = SimpleAppInstance.foo("fooString")
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Above code work fines.
>>>>>>>>>
>>>>>>>>> I have setup LD_LIBRARY_PATH and spark.executor.extraClassPath,
>>>>>>>>> spark.executor.extraLibraryPath at worker node
>>>>>>>>>
>>>>>>>>> How can i invoke JNI library from worker node ? Where should i
>>>>>>>>> load it in executor ?
>>>>>>>>> Calling System.loadLibrary("foolib") inside the work node gives
>>>>>>>>> me following error :
>>>>>>>>>
>>>>>>>>> Exception in thread "main" java.lang.UnsatisfiedLinkError:
>>>>>>>>>
>>>>>>>>> Any help would be really appreciated.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
