Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Flink 1.11 will support Hadoop 3. EMR 6 requires Hadoop 3, that's why Flink was not included anymore. Amazon will add Flink back to EMR 6.0 soon. On Thu, May 14, 2020 at 7:11 PM aj wrote: > Hi Yang, > > I am able to resolve the issue by removing Hadoop dependency as you > mentioned. > > 1. Remov

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi Yang, I am able to resolve the issue by removing Hadoop dependency as you mentioned. 1. Removed hadoop-common dependency and org.apache.flink flink-streaming-java_${scala.binary.version} ${flink.version} org.apache.flink flink-hadoop-f

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hello Yang, I have attached my pom file and I did not see that I am using any Hadoop dependency. Can you please help me. On Wed, May 6, 2020 at 1:22 PM Yang Wang wrote: > Hi aj, > > From the logs you have provided, the hadoop version is still 2.4.1. > Could you check the user jar(i.e. events-pr

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi aj, >From the logs you have provided, the hadoop version is still 2.4.1. Could you check the user jar(i.e. events-processor-1.0-SNAPSHOT.jar) have some hadoop classes? If it is, you need to exclude the hadoop dependency. Best, Yang aj 于2020年5月6日周三 下午3:38写道： > Hello, > > Please help me upgr

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hello, Please help me upgrade to 1.10 in AWS EMR. On Fri, May 1, 2020 at 4:05 PM aj wrote: > Hi Yang, > > I am attaching the logs for your reference, please help me what i am doing > wrong. > > Thanks, > Anuj > > On Wed, Apr 29, 2020 at 9:06 AM Yang Wang wrote: > >> Hi Anuj, >> >> I think the

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi Yang, I am attaching the logs for your reference, please help me what i am doing wrong. Thanks, Anuj On Wed, Apr 29, 2020 at 9:06 AM Yang Wang wrote: > Hi Anuj, > > I think the exception you come across still because the hadoop version > is 2.4.1. I have checked the hadoop code, the code li

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi Anuj, I think the exception you come across still because the hadoop version is 2.4.1. I have checked the hadoop code, the code line are exactly same. For 2.8.1, i also have checked the ruleParse. It could work. /** * A pattern for parsing a auth_to_local rule. */ private static final Patter

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hello Yang, My Hadoop version is Hadoop 3.2.1-amzn-0 and I have put in flink/lib. flink-shaded-hadoop-2-uber-2.8.3-10.0.jar then I am getting below error : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_15879

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi Anuj, It seems that you are using hadoop version 2.4.1. I think "L" could not be supported in this version. Could you upgrade your hadoop version to 2.8 and have a try? If your YARN cluster version is 2.8+, then you could directly remove the flink-shaded-hadoop in your lib directory. Otherwise,

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi Robert, attached the full application log file. Thanks, Anuj Container: container_1585301409143_0044_01_01 on ip-172-25-2-209.ap-south-1.compute.internal_8041 =

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hey Anuj, can you post the "header" of the jobmanager log, I'm interested in seeing the classpath of your jobmanager. Most likely, there's a mixup in your dependency versions in your classpath. On Tue, Apr 7, 2020 at 8:08 AM aj wrote: > Hello All, > > I am running Flink on AWS EMR, as currentl

upgrade flink from 1.9.1 to 1.10.0 on EMR

Hello All, I am running Flink on AWS EMR, as currently the latest version available on EMR is 1.9.1 but I want to upgrade to 1.10.0. I tried to manually replace lib jars by downloading the 1.10.0 version but this is not working. I am getting the following exception when trying to submit a job on y