I know this is really basic but have you verified that you're Flink lib folder contains log4j-1.2.17.jar? I imagine that's fine given the yarn-session.sh approach is working fine. What version of EMR are you running? What version of Flink?
-Jamie On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <[email protected]> wrote: > I’m trying to submit a stand-alone Flink job to a YARN cluster running on > EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container > for the Job Manager, it fails. The error message from the container is > below. The command I’m using is: > > $ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount > ./flink-1.0.3/examples/streaming/WordCount.jar > > I have tried adding log4j and slf4j libraries to the classpath using -C > and that doesn’t help. > > This does not happen on other YARN clusters I have that are not EMR nodes. > And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" to > create a Flink cluster in the YARN cluster and then use “flink run …” to > submit the job to the Flink cluster. > > Does anyone out there know how I could fix this? > > Thanks in advance for any help you can give. > > Error message in the jobmanager.err file: > > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > > SLF4J: Defaulting to no-operation (NOP) logger implementation > > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/log4j/Level > > at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:278) > > at > org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) > > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95) > > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) > > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) > > at org.apache.hadoop.security.Groups.<init>(Groups.java:79) > > at org.apache.hadoop.security.Groups.<init>(Groups.java:74) > > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303) > > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283) > > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) > > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790) > > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760) > > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633) > > at > org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64) > > at > org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36) > > at > org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala) > > Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > ... 18 more > > > *Bruce Hanson* > > Software Engineer > > HERE Predictive Analytics > > > > *HERE Seattle* > > 701 Pike St, Suite 2000, Seattle, WA 98101 > > *47° 36' 41" N. 122° 19' 57" W > <http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street>* > > > > <http://360.here.com/> <https://twitter.com/here> > <https://www.facebook.com/here> <https://linkedin.com/company/heremaps> > <https://www.instagram.com/here> > > > > -- Jamie Grier data Artisans, Director of Applications Engineering @jamiegrier <https://twitter.com/jamiegrier> [email protected]
