Hi Till,
Sorry, I understand that this was confusing.
The JarWithMain contains a class called Launcher with main method that does
something like:
Class.getForName("classFromUserJar").newInstance().launch()
So it doesnt have any "static" dependency to the UserJar. JarWithMain has a
lot of API classes that UserJar depends on.
And you are right, I put the jars in a place where it is accessible from
every node. (We have a distributed fs mounted under the same path on each
machine)
I hope I'm clearer now, but let me know if you need some more info (or you
can drop me a line on skype)
Gyula
Till Rohrmann <trohrm...@apache.org> ezt írta (időpont: 2016. nov. 15., K,
14:55):
> Hi Gyula,
>
> did I understand it correct that JarWithMain depends on UserJar because
> the former will execute classes from the latter and UserJar depends on
> JarWithMain because it contains classes depending on class from
> JarWithMain? This sounds like a cyclic dependency to me.
>
> The command line help for -C says the following: “Adds a URL to each user
> code classloader on all nodes in the cluster. The paths must specify a
> protocol (e.g. file://) and be accessible on all nodes (e.g. by means of a
> NFS share). You can use this option multiple times for specifying more than
> one URL. The protocol must be supported by the {@link
> java.net.URLClassLoader}.”
>
> Thus, I assume that you’ve placed the jar whose classpath you specify via
> -C somewhere where it is accessible from all nodes including the one
> running the CliFrontend, right? Otherwise running such a job cannot work
> because -C won’t distribute the jars in the cluster.
>
> I tried to reproduce your issue by setting up a multi module where my main
> module depends on the user module via dynamic class loading and the user
> module depends on the main module by a class dependency. Specifying a
> classpath which is accessible from all nodes allows to run the job via
> flink/run
> -C main -c Job util and flink/run -C util -c Job main where Job is
> residing in main.
>
> Thus, I would need some more information about the actual setup to be able
> to reproduce your problem.
>
> Cheers,
> Till
>
>
> On Tue, Nov 15, 2016 at 10:06 AM, Gyula Fóra <gyula.f...@gmail.com> wrote:
>
> Hi Scott,
>
> Thanks, I am familiar with the ways you suggested. Unfortunately packaging
> everything together is not really an option in our case, we specifically
> want to avoid having to do this as many people will set up their own builds
> and they will inevitable fail to include everything necessary with the
> correct setup.
>
> Including it in the lib folder would mean I have to copy/remove jars all
> the time when deploying new jobs if I don't want to have dependency clashes
> and also it doesn't seem to be a clean solution for something so simple.
>
> I thought the -C option would actually do what I am looking for in an
> elegant way that's why I am trying to understand what happened.
>
> Cheers,
> Gyula
>
> Scott Kidder <kidder.sc...@gmail.com> ezt írta (időpont: 2016. nov. 14.,
> H, 18:49):
>
> Hi Gyula,
>
> I've typically added external library dependencies to my own application
> JAR as shaded-dependencies. This ensures that all dependencies are included
> with my application while being distributed to Flink Job Manager & Task
> Manager instances.
>
> Another approach is to place these external JARs in the 'lib'
> sub-directory of your Flink installation. Keep in mind that the external
> JARs must be installed on every Flink node where your application is
> expected to run. This works well for dependencies that are large in size or
> used by multiple Flink applications in your cluster (avoid duplication of
> dependencies).
>
> Best,
>
> --Scott Kidder
>
> On Mon, Nov 14, 2016 at 7:59 AM, Gyula Fóra <gyula.f...@gmail.com> wrote:
>
> Hi,
>
> I have been trying to use the -C flag to add external jars with user code
> and I have observed some strange behaviour.
>
> What I am trying to do is the following:
> I have 2 jars, JarWithMain.jar contains the main class and UserJar.jar
> contains some classes that the main method will eventually execute and also
> depends on classes from JarWithMain.
>
> Running this works:
> flink run .... -C UserJar.jar -c MainMethod JarWithMain.jar args...
>
> Running this leads to no class def found errors in the StreamTask
> initialization where it reads the functions from the config:
> flink run .... -C JarWithMain.jar -c MainMethod UserJar.jar args...
>
> Did I miss something?
>
> Cheers,
> Gyula
>
>
>
>
