Hi Bruce, I just spun up an EMR cluster and tried this out. Hadoop 2.7.2 and Flink 1.0.3. I ran the exact same command as you and everything works just fine.
Please verify one thing, though. In your command you do not specify the path to the Flink executable, which means it's just getting picked up from your PATH. Is it possible you're running a flink-1.0.3 example program but using a different version of Flink? Try the following: ./flink-1.0.3/bin/flink run -m yarn-cluster -yn 1 -ynm test1:WordCount ./flink-1.0.3/examples/streaming/WordCount.jar -Jamie On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <bruce.han...@here.com> wrote: > I’m trying to submit a stand-alone Flink job to a YARN cluster running on > EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container > for the Job Manager, it fails. The error message from the container is > below. The command I’m using is: > > $ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount > ./flink-1.0.3/examples/streaming/WordCount.jar > > I have tried adding log4j and slf4j libraries to the classpath using -C > and that doesn’t help. > > This does not happen on other YARN clusters I have that are not EMR nodes. > And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" to > create a Flink cluster in the YARN cluster and then use “flink run …” to > submit the job to the Flink cluster. > > Does anyone out there know how I could fix this? > > Thanks in advance for any help you can give. > > Error message in the jobmanager.err file: > > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > > SLF4J: Defaulting to no-operation (NOP) logger implementation > > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/log4j/Level > > at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:278) > > at > org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) > > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95) > > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) > > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) > > at org.apache.hadoop.security.Groups.<init>(Groups.java:79) > > at org.apache.hadoop.security.Groups.<init>(Groups.java:74) > > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303) > > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283) > > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) > > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790) > > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760) > > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633) > > at > org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64) > > at > org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36) > > at > org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala) > > Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > ... 18 more > > > *Bruce Hanson* > > Software Engineer > > HERE Predictive Analytics > > > > *HERE Seattle* > > 701 Pike St, Suite 2000, Seattle, WA 98101 > > *47° 36' 41" N. 122° 19' 57" W > <http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street>* > > > > <http://360.here.com/> <https://twitter.com/here> > <https://www.facebook.com/here> <https://linkedin.com/company/heremaps> > <https://www.instagram.com/here> > > > > -- Jamie Grier data Artisans, Director of Applications Engineering @jamiegrier <https://twitter.com/jamiegrier> ja...@data-artisans.com
