Hi Yang,
I am able to resolve the issue by removing Hadoop dependency as you
mentioned.
1. Removed hadoop-common dependency and
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hadoop-fs</artifactId>
</exclusion>
</exclusions>
</dependency>
2. After the above changes, I am able to submit job on yarn but facing
issues with s3 plugin that I resolved by putting *
flink-s3-fs-hadoop-1.9.0.jar *i n the plugins/s3-fs-hadoop directory.
Thanks for your support.
Any update when will flink.10 officially supported in EMR. Even in new EMR
version(emr 6.0) flink has been removed.
On Sat, May 9, 2020 at 1:36 PM aj <ajainje...@gmail.com> wrote:
> Hello Yang,
>
> I have attached my pom file and I did not see that I am using any Hadoop
> dependency. Can you please help me.
>
> On Wed, May 6, 2020 at 1:22 PM Yang Wang <danrtsey...@gmail.com> wrote:
>
>> Hi aj,
>>
>> From the logs you have provided, the hadoop version is still 2.4.1.
>> Could you check the user jar(i.e. events-processor-1.0-SNAPSHOT.jar) have
>> some
>> hadoop classes? If it is, you need to exclude the hadoop dependency.
>>
>>
>> Best,
>> Yang
>>
>> aj <ajainje...@gmail.com> 于2020年5月6日周三 下午3:38写道：
>>
>>> Hello,
>>>
>>> Please help me upgrade to 1.10 in AWS EMR.
>>>
>>> On Fri, May 1, 2020 at 4:05 PM aj <ajainje...@gmail.com> wrote:
>>>
>>>> Hi Yang,
>>>>
>>>> I am attaching the logs for your reference, please help me what i am
>>>> doing wrong.
>>>>
>>>> Thanks,
>>>> Anuj
>>>>
>>>> On Wed, Apr 29, 2020 at 9:06 AM Yang Wang <danrtsey...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Anuj,
>>>>>
>>>>> I think the exception you come across still because the hadoop version
>>>>> is 2.4.1. I have checked the hadoop code, the code line are exactly
>>>>> same.
>>>>> For 2.8.1, i also have checked the ruleParse. It could work.
>>>>>
>>>>> /**
>>>>> * A pattern for parsing a auth_to_local rule.
>>>>> */
>>>>> private static final Pattern ruleParser =
>>>>>
>>>>> Pattern.compile("\\s*((DEFAULT)|(RULE:\\[(\\d*):([^\\]]*)](\\(([^)]*)\\))?"+
>>>>> "(s/([^/]*)/([^/]*)/(g)?)?))/?(L)?");
>>>>>
>>>>>
>>>>> Could you share the jobmanager logs so that i could check the
>>>>> classpath and hadoop version?
>>>>>
>>>>> Best,
>>>>> Yang
>>>>>
>>>>> aj <ajainje...@gmail.com> 于2020年4月28日周二 上午1:01写道：
>>>>>
>>>>>> Hello Yang,
>>>>>> My Hadoop version is Hadoop 3.2.1-amzn-0
>>>>>> and I have put in flink/lib.
>>>>>> flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
>>>>>>
>>>>>> then I am getting below error :
>>>>>>
>>>>>> SLF4J: Class path contains multiple SLF4J bindings.
>>>>>> SLF4J: Found binding in
>>>>>> [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1587983834922_0002/filecache/10/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>> SLF4J: Found binding in
>>>>>> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>>>> explanation.
>>>>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>> Invalid rule: /L
>>>>>> RULE:[2:$1@$0](.*@)s/@.*///L
>>>>>> DEFAULT
>>>>>> at
>>>>>> org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:321)
>>>>>> at
>>>>>> org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:386)
>>>>>> at
>>>>>> org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:75)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)
>>>>>> at
>>>>>> org.apache.flink.yarn.entrypoint.YarnEntrypointUtils.logYarnEnvironmentInformation(YarnEntrypointUtils.java:136)
>>>>>> at
>>>>>> org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:109)
>>>>>>
>>>>>>
>>>>>> if I remove the flink-shaded-hadoop-2-uber-2.8.3-10.0.jar from lib
>>>>>> then i get below error:
>>>>>>
>>>>>> 2020-04-27 16:59:37,293 INFO org.apache.flink.client.cli.CliFrontend
>>>>>> - Classpath:
>>>>>> /usr/lib/flink/lib/flink-table-blink_2.11-1.10.0.jar:/usr/lib/flink/lib/flink-table_2.11-1.10.0.jar:/usr/lib/flink/lib/log4j-1.2.17.jar:/usr/lib/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/lib/flink/lib/flink-dist_2.11-1.10.0.jar::/etc/hadoop/conf:/etc/hadoop/conf
>>>>>> 2020-04-27 16:59:37,293 INFO org.apache.flink.client.cli.CliFrontend
>>>>>> -
>>>>>> --------------------------------------------------------------------------------
>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: jobmanager.heap.size, 1024m
>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: taskmanager.memory.process.size, 1568m
>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: taskmanager.numberOfTaskSlots, 1
>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: parallelism.default, 1
>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: env.yarn.conf.dir, /etc/hadoop/conf
>>>>>> 2020-04-27 16:59:37,300 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
>>>>>> 2020-04-27 16:59:37,301 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: jobmanager.execution.failover-strategy, region
>>>>>> 2020-04-27 16:59:37,301 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: classloader.resolve-order, parent-first
>>>>>> 2020-04-27 16:59:37,301 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: s3.access-key, AKIA52DD5QA5FC7HPKXG
>>>>>> 2020-04-27 16:59:37,301 INFO
>>>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>>>> configuration property: s3.secret-key, ******
>>>>>> 2020-04-27 16:59:37,305 WARN org.apache.flink.client.cli.CliFrontend
>>>>>> - Could not load CLI class
>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.
>>>>>> java.lang.NoClassDefFoundError:
>>>>>> org/apache/hadoop/yarn/exceptions/YarnException
>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>> at java.lang.Class.forName(Class.java:264)
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1076)
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1030)
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:957)
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.hadoop.yarn.exceptions.YarnException
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>>>>>> at
>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>>>>>> ... 5 more
>>>>>> 2020-04-27 16:59:37,406 INFO org.apache.flink.core.fs.FileSystem
>>>>>> - Hadoop is not in the classpath/dependencies. The
>>>>>> extended set of supported File Systems via Hadoop is not available.
>>>>>> 2020-04-27 16:59:37,458 INFO
>>>>>> org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot
>>>>>> create Hadoop Security Module because Hadoop cannot be found in the
>>>>>> Classpath.
>>>>>> 2020-04-27 16:59:37,476 INFO
>>>>>> org.apache.flink.runtime.security.modules.JaasModule - Jaas
>>>>>> file
>>>>>> will be created as /tmp/jaas-7054453135321774613.conf.
>>>>>> 2020-04-27 16:59:37,480 INFO
>>>>>> org.apache.flink.runtime.security.SecurityUtils - Cannot
>>>>>> install HadoopSecurityContext because Hadoop cannot be found in the
>>>>>> Classpath.
>>>>>> 2020-04-27 16:59:37,481 INFO org.apache.flink.client.cli.CliFrontend
>>>>>> - Running 'run' command.
>>>>>> 2020-04-27 16:59:37,488 INFO org.apache.flink.client.cli.CliFrontend
>>>>>> - Building program from JAR file
>>>>>> 2020-04-27 16:59:37,488 ERROR org.apache.flink.client.cli.CliFrontend
>>>>>> - Invalid command line arguments.
>>>>>> org.apache.flink.client.cli.CliArgsException: Could not build the
>>>>>> program from JAR file.
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:203)
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
>>>>>> at
>>>>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
>>>>>> Caused by: java.io.FileNotFoundException: JAR file does not exist:
>>>>>> -ynm
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.getJarFile(CliFrontend.java:719)
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:695)
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:200)
>>>>>> ... 4 more
>>>>>>
>>>>>>
>>>>>> Can you please help.
>>>>>>
>>>>>> Thanks,
>>>>>> Anuj
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 13, 2020 at 7:43 AM Yang Wang <danrtsey...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Anuj,
>>>>>>>
>>>>>>> It seems that you are using hadoop version 2.4.1. I think "L" could
>>>>>>> not be supported in
>>>>>>> this version. Could you upgrade your hadoop version to 2.8 and have
>>>>>>> a try? If your
>>>>>>> YARN cluster version is 2.8+, then you could directly remove the
>>>>>>> flink-shaded-hadoop
>>>>>>> in your lib directory. Otherwise, you need to download the
>>>>>>> flink-shaded-hadoop with
>>>>>>> version 2.8 here[1].
>>>>>>>
>>>>>>>
>>>>>>> [1]. https://flink.apache.org/downloads.html#additional-components
>>>>>>>
>>>>>>> Best,
>>>>>>> Yang
>>>>>>>
>>>>>>> aj <ajainje...@gmail.com> 于2020年4月11日周六 上午4:21写道：
>>>>>>>
>>>>>>>> Hi Robert,
>>>>>>>> attached the full application log file.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Anuj
>>>>>>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Anuj Jain
>>>>>> Mob. : +91- 8588817877
>>>>>> Skype : anuj.jain07
>>>>>> <http://www.oracle.com/>
>>>>>>
>>>>>>
>>>>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anuj Jain
>>>> Mob. : +91- 8588817877
>>>> Skype : anuj.jain07
>>>> <http://www.oracle.com/>
>>>>
>>>>
>>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anuj Jain
>>> Mob. : +91- 8588817877
>>> Skype : anuj.jain07
>>> <http://www.oracle.com/>
>>>
>>>
>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>
>>
>
> --
> Thanks & Regards,
> Anuj Jain
> Mob. : +91- 8588817877
> Skype : anuj.jain07
> <http://www.oracle.com/>
>
>
> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>
--
Thanks & Regards,
Anuj Jain
Mob. : +91- 8588817877
Skype : anuj.jain07
<http://www.oracle.com/>
<http://www.cse.iitm.ac.in/%7Eanujjain/>
