Hi Gyula,
did I understand it correct that JarWithMain depends on UserJar because the
former will execute classes from the latter and UserJar depends on
JarWithMain because it contains classes depending on class from
JarWithMain? This sounds like a cyclic dependency to me.
The command line help for -C says the following: “Adds a URL to each user
code classloader on all nodes in the cluster. The paths must specify a
protocol (e.g. file://) and be accessible on all nodes (e.g. by means of a
NFS share). You can use this option multiple times for specifying more than
one URL. The protocol must be supported by the {@link
java.net.URLClassLoader}.”
Thus, I assume that you’ve placed the jar whose classpath you specify via
-C somewhere where it is accessible from all nodes including the one
running the CliFrontend, right? Otherwise running such a job cannot work
because -C won’t distribute the jars in the cluster.
I tried to reproduce your issue by setting up a multi module where my main
module depends on the user module via dynamic class loading and the user
module depends on the main module by a class dependency. Specifying a
classpath which is accessible from all nodes allows to run the job via
flink/run
-C main -c Job util and flink/run -C util -c Job main where Job is residing
in main.
Thus, I would need some more information about the actual setup to be able
to reproduce your problem.
Cheers,
Till
On Tue, Nov 15, 2016 at 10:06 AM, Gyula Fóra <gyula.f...@gmail.com> wrote:
> Hi Scott,
>
> Thanks, I am familiar with the ways you suggested. Unfortunately packaging
> everything together is not really an option in our case, we specifically
> want to avoid having to do this as many people will set up their own builds
> and they will inevitable fail to include everything necessary with the
> correct setup.
>
> Including it in the lib folder would mean I have to copy/remove jars all
> the time when deploying new jobs if I don't want to have dependency clashes
> and also it doesn't seem to be a clean solution for something so simple.
>
> I thought the -C option would actually do what I am looking for in an
> elegant way that's why I am trying to understand what happened.
>
> Cheers,
> Gyula
>
> Scott Kidder <kidder.sc...@gmail.com> ezt írta (időpont: 2016. nov. 14.,
> H, 18:49):
>
>> Hi Gyula,
>>
>> I've typically added external library dependencies to my own application
>> JAR as shaded-dependencies. This ensures that all dependencies are included
>> with my application while being distributed to Flink Job Manager & Task
>> Manager instances.
>>
>> Another approach is to place these external JARs in the 'lib'
>> sub-directory of your Flink installation. Keep in mind that the external
>> JARs must be installed on every Flink node where your application is
>> expected to run. This works well for dependencies that are large in size or
>> used by multiple Flink applications in your cluster (avoid duplication of
>> dependencies).
>>
>> Best,
>>
>> --Scott Kidder
>>
>> On Mon, Nov 14, 2016 at 7:59 AM, Gyula Fóra <gyula.f...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have been trying to use the -C flag to add external jars with user code
>> and I have observed some strange behaviour.
>>
>> What I am trying to do is the following:
>> I have 2 jars, JarWithMain.jar contains the main class and UserJar.jar
>> contains some classes that the main method will eventually execute and also
>> depends on classes from JarWithMain.
>>
>> Running this works:
>> flink run .... -C UserJar.jar -c MainMethod JarWithMain.jar args...
>>
>> Running this leads to no class def found errors in the StreamTask
>> initialization where it reads the functions from the config:
>> flink run .... -C JarWithMain.jar -c MainMethod UserJar.jar args...
>>
>> Did I miss something?
>>
>> Cheers,
>> Gyula
>>
>>
>>
