A couple of reasons I 've done that
- it's listed as an option here :
https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/docker.html#advanced-customization
under optional libraries
- I have over 200 jobs running that rely on the same core functionality
provided by the jar in question and it seems somewhat wasteful to do 200
uploads of 80M+ of the same code to the cluster
- I was having issues shipping jar content when it was packaged as a part
of my "fat" jar application. I was providing a URL to the jar
(jar:file:xxxx!/xxx) but flink could not find it there.


On Fri, Nov 20, 2020 at 12:29 PM Arvid Heise <ar...@ververica.com> wrote:

> Oh no, please never put user code (with included libraries) into flink's
> classpath. It's not supported exactly for classloader reasons. Why did you
> think that this would be a good approach? Is your jar too big?
>
> Maybe a different deployment mode would be more appropriate? [1]
>
> Alternatively, if you want to go the hacky route, you could also try to
> shade your dependencies.
>
> [1] https://flink.apache.org/news/2020/07/14/application-mode.html
>
> On Fri, Nov 20, 2020 at 9:18 PM Alexander Bagerman <bager...@gmail.com>
> wrote:
>
>> I just switched from providing my jar while creating a Remote environment
>> to providing this jar on flink's classpath. It used to work just fine when
>> the jar was shipped to Fllink with the job graph. Now when jar is available
>> to flink on the startup the same job that used to run is failing with
>> exception I provided. I suspect that it might be class loader issue but am
>> not sure
>>
>> On Fri, Nov 20, 2020 at 12:10 PM Arvid Heise <ar...@ververica.com> wrote:
>>
>>> The easiest solution for all non-trivial issues like this is to start
>>> the application locally in a local executor, so you can debug in your IDE.
>>>
>>> Additionally, double-check that you have no lambdas/anonymous classes
>>> that reference outer classes with ObjectMapper. ObjectMapper should also be
>>> static as it's fully immutable, so you can also check that.
>>>
>>> On Fri, Nov 20, 2020 at 8:55 PM Alexander Bagerman <bager...@gmail.com>
>>> wrote:
>>>
>>>> Thanks, Arvid,
>>>> That is what I thought too. I went through all the instances where it
>>>> might 've been a member variable and made sure that it's declared as
>>>> transient :-( Is there anything else I can check?
>>>> Alex
>>>>
>>>> On Fri, Nov 20, 2020 at 11:50 AM Arvid Heise <ar...@ververica.com>
>>>> wrote:
>>>>
>>>>> Are you using ObjectMapper as a non-transient field? If so, please
>>>>> make it transient and initialize in open() of a Rich*Function.
>>>>>
>>>>> On Fri, Nov 20, 2020 at 7:56 PM Alexander Bagerman <bager...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I added my custom jar (that includes dependencies on Jackson) to
>>>>>> Flink classpath. It seems to be loaded just fine. But when the job 
>>>>>> starts I
>>>>>> am getting an exception below. I am sure how to interpret the exception
>>>>>> though and would appreciate it if somebody gives me advice on it.
>>>>>> Thanks
>>>>>> Alex
>>>>>>
>>>>>> 2020-11-20 18:34:35,643 WARN org.apache.flink.runtime.taskmanager.
>>>>>> Task [] - Source: Custom Source -> Flat Map -> Flat Map -> Sink:
>>>>>> Unnamed (1/1) (dcbf799dadba5d4b7e7f5af15919a4b6) switched from
>>>>>> RUNNING to FAILED.
>>>>>> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
>>>>>> instantiate user function.
>>>>>>     at org.apache.flink.streaming.api.graph.StreamConfig
>>>>>> .getStreamOperatorFactory(StreamConfig.java:275) ~[flink-dist_2.11-
>>>>>> 1.11.2.jar:1.11.2]
>>>>>>     at org.apache.flink.streaming.runtime.tasks.OperatorChain
>>>>>> .createChainedOperator(OperatorChain.java:471) ~[flink-dist_2.11-1.11
>>>>>> .2.jar:1.11.2]
>>>>>>     at org.apache.flink.streaming.runtime.tasks.OperatorChain
>>>>>> .createOutputCollector(OperatorChain.java:393) ~[flink-dist_2.11-1.11
>>>>>> .2.jar:1.11.2]
>>>>>>     at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(
>>>>>> OperatorChain.java:155) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
>>>>>>     at org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>>> .beforeInvoke(StreamTask.java:459) ~[flink-dist_2.11-1.11.2.jar:1.11.
>>>>>> 2]
>>>>>>     at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(
>>>>>> StreamTask.java:528) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
>>>>>>     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
>>>>>> [flink-dist_2.11-1.11.2.jar:1.11.2]
>>>>>>     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
>>>>>> [flink-dist_2.11-1.11.2.jar:1.11.2]
>>>>>>     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
>>>>>> Caused by: java.lang.ClassCastException: cannot assign instance of
>>>>>> java.util.concurrent.ConcurrentHashMap to field
>>>>>> com.fasterxml.jackson.databind.deser.DeserializerCache._cachedDeserializers
>>>>>> of type com.fasterxml.jackson.databind.util.LRUMap in instance of
>>>>>> com.fasterxml.jackson.databind.deser.DeserializerCache
>>>>>>     at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(
>>>>>> ObjectStreamClass.java:2301) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass
>>>>>> .java:1431) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>>>>>> .java:2372) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream
>>>>>> .java:2290) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>>>>>> .java:2148) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:
>>>>>> 1647) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>>>>>> .java:2366) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream
>>>>>> .java:2290) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>>>>>> .java:2148) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:
>>>>>> 1647) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>>>>>> .java:2366) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream
>>>>>> .java:2290) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>>>>>> .java:2148) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:
>>>>>> 1647) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>>>>>> .java:2366) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream
>>>>>> .java:2290) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>>>>>> .java:2148) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:
>>>>>> 1647) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>>>>>> .java:2366) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream
>>>>>> .java:2290) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>>>>>> .java:2148) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:
>>>>>> 1647) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>>>>>> .java:2366) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream
>>>>>> .java:2290) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>>>>>> .java:2148) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:
>>>>>> 1647) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>>>>>> .java:2366) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream
>>>>>> .java:2290) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>>>>>> .java:2148) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:
>>>>>> 1647) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:
>>>>>> 483) ~[?:1.8.0_265]
>>>>>>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:
>>>>>> 441) ~[?:1.8.0_265]
>>>>>>     at org.apache.flink.util.InstantiationUtil.deserializeObject(
>>>>>> InstantiationUtil.java:576) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
>>>>>>     at org.apache.flink.util.InstantiationUtil.deserializeObject(
>>>>>> InstantiationUtil.java:562) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
>>>>>>     at org.apache.flink.util.InstantiationUtil.deserializeObject(
>>>>>> InstantiationUtil.java:550) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
>>>>>>     at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(
>>>>>> InstantiationUtil.java:511) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
>>>>>>     at org.apache.flink.streaming.api.graph.StreamConfig
>>>>>> .getStreamOperatorFactory(StreamConfig.java:260) ~[flink-dist_2.11-
>>>>>> 1.11.2.jar:1.11.2]
>>>>>>     ... 8 more
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Arvid Heise | Senior Java Developer
>>>>>
>>>>> <https://www.ververica.com/>
>>>>>
>>>>> Follow us @VervericaData
>>>>>
>>>>> --
>>>>>
>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>>> Conference
>>>>>
>>>>> Stream Processing | Event Driven | Real Time
>>>>>
>>>>> --
>>>>>
>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>>
>>>>> --
>>>>> Ververica GmbH
>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
>>>>> Ji (Toni) Cheng
>>>>>
>>>>
>>>
>>> --
>>>
>>> Arvid Heise | Senior Java Developer
>>>
>>> <https://www.ververica.com/>
>>>
>>> Follow us @VervericaData
>>>
>>> --
>>>
>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>> Conference
>>>
>>> Stream Processing | Event Driven | Real Time
>>>
>>> --
>>>
>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>
>>> --
>>> Ververica GmbH
>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>> (Toni) Cheng
>>>
>>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <https://www.ververica.com/>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>

Reply via email to