Have you tried setting the configuration `spark.executor.extraLibraryPath` to point to a location where your .so's are available? (Not sure if non-local files, such as HDFS, are supported)
On Mon, Mar 14, 2016 at 2:12 PM, Tristan Nixon <st...@memeticlabs.org> wrote: > What build system are you using to compile your code? > If you use a dependency management system like maven or sbt, then you should > be able to instruct it to build a single jar that contains all the other > dependencies, including third-party jars and .so’s. I am a maven user myself, > and I use the shade plugin for this: > https://maven.apache.org/plugins/maven-shade-plugin/ > > However, if you are using SBT or another dependency manager, someone else on > this list may be able to give you help on that. > > If you’re not using a dependency manager - well, you should be. Trying to > manage this manually is a pain that you do not want to get in the way of your > project. There are perfectly good tools to do this for you; use them. > >> On Mar 14, 2016, at 3:56 PM, prateek arora <prateek.arora...@gmail.com> >> wrote: >> >> Hi >> >> Thanks for the information . >> >> but my problem is that if i want to write spark application which depend on >> third party libraries like opencv then whats is the best approach to >> distribute all .so and jar file of opencv in all cluster ? >> >> Regards >> Prateek >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-distribute-dependent-files-so-jar-across-spark-worker-nodes-tp26464p26489.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
