Ah I see. This explains the issues I had with submitting streaming jobs that package JDBC drivers. Is there a second in the guide/docs about classloader considerations with Flink?
On Thu, Dec 10, 2015 at 11:53 PM, Stephan Ewen <se...@apache.org> wrote: > Flink's classloading is different from Hadoop's. > > In Hadoop, the entire JVM is started with all classes (including the user > jar) in the classpath already. In Flink, jars are added dymanically, to > running JVMs with custom class loaders. That way, running worker/master > processes can accept new jars without restarts. Important for low-latency, > shells, etc > > Flink itself respects these classloaders whenever dynamically looking up a > class. It may be that Closure is written such that it can only dynamically > instantiate what is the original classpath. > > > > On Fri, Dec 11, 2015 at 1:31 AM, Nick Dimiduk <ndimi...@apache.org> wrote: > > > As far as the jvm is concerned, clojure is just another library. You > should > > be able to package it up like any other dependency and submit the job. > > That's always how it worked in Hadoop/MR anyway... > > > > On Thu, Dec 10, 2015 at 3:22 PM, Matthias J. Sax <mj...@apache.org> > wrote: > > > > > Thanks for this idea. > > > > > > I extended my pom to include clojure-1.5.1.jar in my program jar. > > > However, the problem is still there... I did some research on the > > > Internet, and it seems I need to mess around with Clojure's class > > > loading strategy... > > > > > > -Matthias > > > > > > On 12/10/2015 06:47 PM, Nick Dimiduk wrote: > > > > I think Mattias's project is using maven though -- there's a pom in > the > > > > project that doesn't look generated. If you want to do it from lein, > > > maybe > > > > my old lein-hadoop [0] plugin can help? > > > > > > > > [0]: https://github.com/ndimiduk/lein-hadoop > > > > > > > > On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <rmetz...@apache.org > > > > > wrote: > > > > > > > >> I had the same though as Nick. Maybe Leiningen allows to somehow > > build a > > > >> fat-jar containing the clojure standard library. > > > >> > > > >> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <ndimi...@apache.org> > > > wrote: > > > >> > > > >>> What happens when you follow the packaging examples provided in the > > > flink > > > >>> quick start archetypes? These have the maven-foo required to > package > > an > > > >>> uberjar suitable for flink submission. Can you try adding that step > > to > > > >> your > > > >>> pom.xml? > > > >>> > > > >>> On Thursday, December 10, 2015, Stephan Ewen <se...@apache.org> > > wrote: > > > >>> > > > >>>> This is a problem in Java. > > > >>>> I think you cannot dynamically modify the initial system class > > loader. > > > >>>> > > > >>>> What most apps do is check for the thread context class loader > when > > > >>>> dynamically loading classes. We can check and make sure that one > is > > > >> set, > > > >>>> but if Closure does not respect that, we have a problem. > > > >>>> Then Closure is not built for dynamic class loading. > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax < > mj...@apache.org > > > >>>> <javascript:;>> wrote: > > > >>>> > > > >>>>> Would it make sense (if possible?) for Flink to add the user jar > > > >>>>> dynamically to it's own classpath so Clojure can find it? Or > > somehow > > > >>>>> modify Clojure's class loader? > > > >>>>> > > > >>>>> The jars in lib are added to the classpath at startup. This makes > > it > > > >>>>> practically impossible to execute a Flink program that is written > > in > > > >>>>> Clojure right now... > > > >>>>> > > > >>>>> > > > >>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote: > > > >>>>>> Clojure is not considering the user-jar when trying to load the > > > >>> class. > > > >>>>>> > > > >>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <mj...@apache.org > > > >>>> <javascript:;>> wrote: > > > >>>>>>> > > > >>>>>>> Hi Squirrels, > > > >>>>>>> > > > >>>>>>> I was playing with a Flink Clojure WordCount example today. > > > >>>>>>> > > https://github.com/mjsax/flink-external/tree/master/flink-clojure > > > >>>>>>> > > > >>>>>>> After building the project with "mvn package" I tried to submit > > it > > > >>> to > > > >>>> a > > > >>>>>>> local cluster. Before I started the cluster, I manually copied > > > >>>>>>> "clojure-1.5.1.jar" into Flink's lib folder. > > > >>>>>>> > > > >>>>>>>> cp > ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar > > > >>> lib/ > > > >>>>>>>> bin/start-local.sh > > > >>>>>>> > > > >>>>>>> However, when submitting the jar, I get an exception: > > > >>>>>>> > > > >>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount > > > >>>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>> ------------------------------------------------------------ > > > >>>>>>>> The program finished with the following exception: > > > >>>>>>>> > > > >>>>>>>> org.apache.flink.client.program.ProgramInvocationException: > The > > > >>>>> program's entry point class 'org.apache.flink.clojure.WordCount' > > > >> threw > > > >>> an > > > >>>>> error during initialization. > > > >>>>>>>> at > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585) > > > >>>>>>>> at > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195) > > > >>>>>>>> at > > > >>>>> > > > >> > org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784) > > > >>>>>>>> at > org.apache.flink.client.CliFrontend.run(CliFrontend.java:288) > > > >>>>>>>> at > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050) > > > >>>>>>>> at > > > >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103) > > > >>>>>>>> Caused by: java.lang.ExceptionInInitializerError > > > >>>>>>>> at java.lang.Class.forName0(Native Method) > > > >>>>>>>> at java.lang.Class.forName(Class.java:278) > > > >>>>>>>> at > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578) > > > >>>>>>>> ... 5 more > > > >>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate > > > >>>>> org/apache/flink/clojure/WordCount__init.class or > > > >>>>> org/apache/flink/clojure/WordCount.clj on classpath: > > > >>>>>>>> at clojure.lang.RT.load(RT.java:443) > > > >>>>>>>> at clojure.lang.RT.load(RT.java:411) > > > >>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530) > > > >>>>>>>> at clojure.core$load.doInvoke(core.clj:5529) > > > >>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408) > > > >>>>>>>> at clojure.lang.Var.invoke(Var.java:415) > > > >>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source) > > > >>>>>>>> ... 8 more > > > >>>>>>> > > > >>>>>>> I am not sure why the class is not found. It is contained in > the > > > >> jar > > > >>>>>>> file. I can fix this error by copying the user jar > > > >>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder. > > > >>>>>>> > > > >>>>>>> It seems, that Flink is not looking into the user-jar when > > loading > > > >>>> this > > > >>>>>>> class. Can anybody explain why? > > > >>>>>>> > > > >>>>>>> Thx. > > > >>>>>>> > > > >>>>>>> -Matthias > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > > > > >
