Hi Sandeep, please have a look at [1], you should add most Flink dependencies as provided - exceptions are connectors (or in general stuff that is not in flink/lib/ or flink/plugins).
[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/project-configuration.html#setting-up-a-project-basic-dependencies On Tue, Mar 23, 2021 at 5:28 AM Sandeep khanzode <sand...@shiftright.ai> wrote: > Hi Arvid, > > I copy the JAR to the usrlib folder. This works in the Cloud EKS cluster. > I wanted to set this up for my testing purposes. > > Below is the Dockerfile: > > FROM apache/flink:1.12.1-java11 > RUN mv /opt/flink/opt/flink-queryable-state-runtime_2.12-1.12.1.jar > /opt/flink/lib/flink-queryable-state-runtime_2.12-1.12.1.jar > ADD myJar.jar /opt/flink/usrlib/myJar.jar > > > … But, in my process, this is a Fat JAR created by the Maven Shade Plugin. > Are you saying that all Flink classes should not be part of the user JAR? > How does that work? Do we set the scope of the dependencies to compile (or, > not runtime) for Flink Jars? Do we have any samples/examples that shows > this? Would be really helpful. > > > On 22-Mar-2021, at 8:00 PM, Arvid Heise <ar...@apache.org> wrote: > > Hi Sandeep, > > The first error definitively indicates a classloading issue, which may > also be the cause for the second error. > > Can you describe where you put your jar inside the docker image and which > execution mode you are using? As a general rule, the jar is not supposed to > go into flink/lib. > Also make sure to never shade non-connector classes of Flink into your > jar. A typical user jar should be ~1MB. > > On Fri, Mar 19, 2021 at 8:58 PM Sandeep khanzode <sand...@shiftright.ai> > wrote: > >> Hello, >> >> I have a fat JAR compiled using the Man Shade plugin and everything >> works correctly when I deploy it on a standalone local cluster i.e. one >> job and one task manager node. >> >> But I installed Minikube and the same JAR file packaged into a docker >> image fails with weird serialization errors: >> >> Caused by: java.lang.ClassCastException: cannot assign instance of >> java.lang.invoke.SerializedLambda to field >> org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.keySelector >> of type org.apache.flink.api.java.functions.KeySelector in instance of >> org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner >> >> >> … or in certain cases, if I comment out everything except the Kafka >> Source, then ... >> Caused by: java.lang.NoClassDefFoundError: Could not initialize class >> org.apache.kafka.common.requests.MetadataRequest$Builder >> >> >> Is there anything I am missing with the Minikube setup? I initially tried >> with the steps for the Job Application cluster on the website, but I was >> unable to get the /usrlib mounted from the hostpath. >> >> >> So, I created a simple docker image from ... >> >> apache/flink:1.12.1-java11 >> >> >> But I have not had any success getting the same job to run here. Please >> let me know if there are well-known steps or issues that I can check. >> >> Thanks, >> Sandeep >> > >
