Flink 1.11 will support Hadoop 3. EMR 6 requires Hadoop 3, that's why Flink
was not included anymore. Amazon will add Flink back to EMR 6.0 soon.
On Thu, May 14, 2020 at 7:11 PM aj <ajainje...@gmail.com> wrote:
> Hi Yang,
>
> I am able to resolve the issue by removing Hadoop dependency as you
> mentioned.
>
> 1. Removed hadoop-common dependency and
>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
> <version>${flink.version}</version>
> <exclusions>
> <exclusion>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-hadoop-fs</artifactId>
> </exclusion>
> </exclusions>
>
> </dependency>
>
>
> 2. After the above changes, I am able to submit job on yarn but facing
> issues with s3 plugin that I resolved by putting *
> flink-s3-fs-hadoop-1.9.0.jar *i n the plugins/s3-fs-hadoop directory.
>
> Thanks for your support.
>
> Any update when will flink.10 officially supported in EMR. Even in new
> EMR version(emr 6.0) flink has been removed.
>
>
>
> On Sat, May 9, 2020 at 1:36 PM aj <ajainje...@gmail.com> wrote:
>
>> Hello Yang,
>>
>> I have attached my pom file and I did not see that I am using any Hadoop
>> dependency. Can you please help me.
>>
>> On Wed, May 6, 2020 at 1:22 PM Yang Wang <danrtsey...@gmail.com> wrote:
>>
>>> Hi aj,
>>>
>>> From the logs you have provided, the hadoop version is still 2.4.1.
>>> Could you check the user jar(i.e. events-processor-1.0-SNAPSHOT.jar)
>>> have some
>>> hadoop classes? If it is, you need to exclude the hadoop dependency.
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> aj <ajainje...@gmail.com> 于2020年5月6日周三 下午3:38写道：
>>>
>>>> Hello,
>>>>
>>>> Please help me upgrade to 1.10 in AWS EMR.
>>>>
>>>> On Fri, May 1, 2020 at 4:05 PM aj <ajainje...@gmail.com> wrote:
>>>>
>>>>> Hi Yang,
>>>>>
>>>>> I am attaching the logs for your reference, please help me what i am
>>>>> doing wrong.
>>>>>
>>>>> Thanks,
>>>>> Anuj
>>>>>
>>>>> On Wed, Apr 29, 2020 at 9:06 AM Yang Wang <danrtsey...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Anuj,
>>>>>>
>>>>>> I think the exception you come across still because the hadoop version
>>>>>> is 2.4.1. I have checked the hadoop code, the code line are exactly
>>>>>> same.
>>>>>> For 2.8.1, i also have checked the ruleParse. It could work.
>>>>>>
>>>>>> /**
>>>>>> * A pattern for parsing a auth_to_local rule.
>>>>>> */
>>>>>> private static final Pattern ruleParser =
>>>>>>
>>>>>> Pattern.compile("\\s*((DEFAULT)|(RULE:\\[(\\d*):([^\\]]*)](\\(([^)]*)\\))?"+
>>>>>> "(s/([^/]*)/([^/]*)/(g)?)?))/?(L)?");
>>>>>>
>>>>>>
>>>>>> Could you share the jobmanager logs so that i could check the
>>>>>> classpath and hadoop version?
>>>>>>
>>>>>> Best,
>>>>>> Yang
>>>>>>
>>>>>> aj <ajainje...@gmail.com> 于2020年4月28日周二 上午1:01写道：
>>>>>>
>>>>>>> Hello Yang,
>>>>>>> My Hadoop version is Hadoop 3.2.1-amzn-0
>>>>>>> and I have put in flink/lib.
>>>>>>> flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
>>>>>>>
>>>>>>> then I am getting below error :
>>>>>>>
>>>>>>> SLF4J: Class path contains multiple SLF4J bindings.
>>>>>>> SLF4J: Found binding in
>>>>>>> [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1587983834922_0002/filecache/10/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>>> SLF4J: Found binding in
>>>>>>> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>>>>> explanation.
>>>>>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>> Invalid rule: /L
>>>>>>> RULE:[2:$1@$0](.*@)s/@.*///L
>>>>>>> DEFAULT
>>>>>>> at
>>>>>>> org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:321)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:386)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:75)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)
>>>>>>> at
>>>>>>> org.apache.flink.yarn.entrypoint.YarnEntrypointUtils.logYarnEnvironmentInformation(YarnEntrypointUtils.java:136)
>>>>>>> at
>>>>>>> org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:109)
>>>>>>>
>>>>>>>
>>>>>>> if I remove the flink-shaded-hadoop-2-uber-2.8.3-10.0.jar from
>>>>>>> lib then i get below error:
>>>>>>>
>>>>>>> 2020-04-27 16:59:37,293 INFO
>>>>>>> org.apache.flink.client.cli.CliFrontend -
>>>>>>> Classpath:
>>>>>>> /usr/lib/flink/lib/flink-table-blink_2.11-1.10.0.jar:/usr/lib/flink/lib/flink-table_2.11-1.10.0.jar:/usr/lib/flink/lib/log4j-1.2.17.jar:/usr/lib/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/lib/flink/lib/flink-dist_2.11-1.10.0.jar::/etc/hadoop/conf:/etc/hadoop/conf
>>>>>>> 2020-04-27 16:59:37,293 INFO
>>>>>>> org.apache.flink.client.cli.CliFrontend -
>>>>>>> --------------------------------------------------------------------------------
>>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: jobmanager.heap.size, 1024m
>>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: taskmanager.memory.process.size, 1568m
>>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: taskmanager.numberOfTaskSlots, 1
>>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: parallelism.default, 1
>>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: env.yarn.conf.dir, /etc/hadoop/conf
>>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
>>>>>>> 2020-04-27 16:59:37,301 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: jobmanager.execution.failover-strategy, region
>>>>>>> 2020-04-27 16:59:37,301 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: classloader.resolve-order, parent-first
>>>>>>> 2020-04-27 16:59:37,301 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: s3.access-key, AKIA52DD5QA5FC7HPKXG
>>>>>>> 2020-04-27 16:59:37,301 INFO
>>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>>> configuration property: s3.secret-key, ******
>>>>>>> 2020-04-27 16:59:37,305 WARN
>>>>>>> org.apache.flink.client.cli.CliFrontend - Could
>>>>>>> not
>>>>>>> load CLI class org.apache.flink.yarn.cli.FlinkYarnSessionCli.
>>>>>>> java.lang.NoClassDefFoundError:
>>>>>>> org/apache/hadoop/yarn/exceptions/YarnException
>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>> at java.lang.Class.forName(Class.java:264)
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1076)
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1030)
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:957)
>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> org.apache.hadoop.yarn.exceptions.YarnException
>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>>>>>>> at
>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>>>>>>> ... 5 more
>>>>>>> 2020-04-27 16:59:37,406 INFO org.apache.flink.core.fs.FileSystem
>>>>>>> - Hadoop is not in the classpath/dependencies.
>>>>>>> The
>>>>>>> extended set of supported File Systems via Hadoop is not available.
>>>>>>> 2020-04-27 16:59:37,458 INFO
>>>>>>> org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot
>>>>>>> create Hadoop Security Module because Hadoop cannot be found in the
>>>>>>> Classpath.
>>>>>>> 2020-04-27 16:59:37,476 INFO
>>>>>>> org.apache.flink.runtime.security.modules.JaasModule - Jaas
>>>>>>> file
>>>>>>> will be created as /tmp/jaas-7054453135321774613.conf.
>>>>>>> 2020-04-27 16:59:37,480 INFO
>>>>>>> org.apache.flink.runtime.security.SecurityUtils - Cannot
>>>>>>> install HadoopSecurityContext because Hadoop cannot be found in the
>>>>>>> Classpath.
>>>>>>> 2020-04-27 16:59:37,481 INFO
>>>>>>> org.apache.flink.client.cli.CliFrontend - Running
>>>>>>> 'run' command.
>>>>>>> 2020-04-27 16:59:37,488 INFO
>>>>>>> org.apache.flink.client.cli.CliFrontend -
>>>>>>> Building
>>>>>>> program from JAR file
>>>>>>> 2020-04-27 16:59:37,488 ERROR
>>>>>>> org.apache.flink.client.cli.CliFrontend - Invalid
>>>>>>> command line arguments.
>>>>>>> org.apache.flink.client.cli.CliArgsException: Could not build the
>>>>>>> program from JAR file.
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:203)
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
>>>>>>> Caused by: java.io.FileNotFoundException: JAR file does not exist:
>>>>>>> -ynm
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.getJarFile(CliFrontend.java:719)
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:695)
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:200)
>>>>>>> ... 4 more
>>>>>>>
>>>>>>>
>>>>>>> Can you please help.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Anuj
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 13, 2020 at 7:43 AM Yang Wang <danrtsey...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Anuj,
>>>>>>>>
>>>>>>>> It seems that you are using hadoop version 2.4.1. I think "L" could
>>>>>>>> not be supported in
>>>>>>>> this version. Could you upgrade your hadoop version to 2.8 and have
>>>>>>>> a try? If your
>>>>>>>> YARN cluster version is 2.8+, then you could directly remove the
>>>>>>>> flink-shaded-hadoop
>>>>>>>> in your lib directory. Otherwise, you need to download the
>>>>>>>> flink-shaded-hadoop with
>>>>>>>> version 2.8 here[1].
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]. https://flink.apache.org/downloads.html#additional-components
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Yang
>>>>>>>>
>>>>>>>> aj <ajainje...@gmail.com> 于2020年4月11日周六 上午4:21写道：
>>>>>>>>
>>>>>>>>> Hi Robert,
>>>>>>>>> attached the full application log file.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Anuj
>>>>>>>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks & Regards,
>>>>>>> Anuj Jain
>>>>>>> Mob. : +91- 8588817877
>>>>>>> Skype : anuj.jain07
>>>>>>> <http://www.oracle.com/>
>>>>>>>
>>>>>>>
>>>>>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anuj Jain
>>>>> Mob. : +91- 8588817877
>>>>> Skype : anuj.jain07
>>>>> <http://www.oracle.com/>
>>>>>
>>>>>
>>>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anuj Jain
>>>> Mob. : +91- 8588817877
>>>> Skype : anuj.jain07
>>>> <http://www.oracle.com/>
>>>>
>>>>
>>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>>
>>>
>>
>> --
>> Thanks & Regards,
>> Anuj Jain
>> Mob. : +91- 8588817877
>> Skype : anuj.jain07
>> <http://www.oracle.com/>
>>
>>
>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>
>
>
> --
> Thanks & Regards,
> Anuj Jain
> Mob. : +91- 8588817877
> Skype : anuj.jain07
> <http://www.oracle.com/>
>
>
> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>
