Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

There is no (accurate) reference of included dependencies for the flink-shaded-hadoop uber jars. The contained NOTICE file is a good starting point, but for the time being we're using a generalized version that we apply to all hadoop versions (so some things may be missing). I believe for hado

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Hi Gary, No, I am running a YARN session (which I start with: flink-yarn-session --slots 4 --taskManagerMemory 16GB --jobManagerMemory 3GB --detached) and submit jobs through the REST interface. Thank you for the tips - I will probably shade it on my side. Is there an official location that the ub

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Hi Austin, Are you running your job detached in a per-job cluster? In that case inverted class loading does not work. This is because we add the user jar to the system class path, and there is no dynamic class loading involved at the moment [1]. You can try the YARN session mode, or – as Chesnay

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Thanks Gary, I will try to look into why the child-first strategy seems to have failed for this dependency. Best, Austin On Wed, Feb 27, 2019 at 12:25 PM Gary Yao wrote: > Hi, > > Actually Flink's inverted class loading feature was designed to mitigate > problems with different versions of lib

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Hi, Actually Flink's inverted class loading feature was designed to mitigate problems with different versions of libraries that are not compatible with each other [1]. You may want to debug why it does not work for you. You can also try to use the Hadoop free Flink distribution, and export the HA

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Short-term I'd try relocating the okio/okhttp dependencies in your jar. I'm not too keen on adding more relocations to the hadoop jar; I can't gauge the possible side-effects. On 27.02.2019 14:54, Austin Cawley-Edwards wrote: Following up to add more info, I am building my app with maven based

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Following up to add more info, I am building my app with maven based on the sample Flink pom.xml My shade plugin config is: org.apache.maven.plugins maven-shade-plugin 3.0.0 package shade

okio and okhttp not shaded in the Flink Uber Jar on EMR

Hi, I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue) , I found that both okio and okhttp exist in the F