Can you try the same thing without -yt, but a yarn-session? – Ufuk
On Thu, Mar 17, 2016 at 12:29 PM, Stefano Baghino <stefano.bagh...@radicalbit.io> wrote: > I have another interesting test result on this matter, running again an > individual job on a YARN cluster. > > When running bin/flink run -m yarn-cluster -yn 1 > examples/batch/WordCount.jar the job fails with an error in the Job Manager > > LogType:jobmanager.err > Log Upload Time:Thu Mar 17 07:05:32 -0400 2016 > LogLength:1221 > Log Contents: > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > Exception in thread "main" java.lang.ExceptionInInitializerError > at > org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64) > at org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36) > at org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala) > Caused by: org.apache.commons.logging.LogConfigurationException: > User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' > cannot be found or is not useable. > at > org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:804) > at > org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541) > at > org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292) > at > org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269) > at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657) > at > org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:84) > ... 3 more > End of LogType:jobmanager.err > > But if I just add -yt lib (the Flink lib folder already contains SLF4J and > Log4J) the libraries are shipped and the job goes on. > > > On Wed, Mar 16, 2016 at 9:15 PM, Ufuk Celebi <u...@apache.org> wrote: >> >> Sounds like a bug. I've only tried this with yarn-sessions. Could it >> be that the behaviour is different between the two variants? In any >> case, I sounds a lot like a bug in Flink. Robert can probably chime in >> here with details. >> >> On Wed, Mar 16, 2016 at 6:25 PM, Andrea Sella >> <andrea.se...@radicalbit.io> wrote: >> > Hi, >> > >> > After few tests I am able to write and read on Alluxio. >> > I am using Flink 1.0.0 and in my case external libraries are not loaded >> > from >> > lib folder to classpath, it loads only flink-dist_2.11-1.0.0.jar. I need >> > to >> > specify the folder with -yt parameter to load the others. >> > >> > If I run `/bin/flink run -m yarn-cluster -yn 4 -yjm 2048 -ytm 4096 >> > some.jar >> > --input alluxio://somepath` it will throws an exception related to a >> > missing >> > library ie. Class alluxio.hadoop.FileSystem not found >> > If I run `./bin/flink run -m yarn-cluster -yt lib/ -yn 4 -yjm 2048 -ytm >> > 4096 >> > some.jar --input alluxio://somepath` with -yt params it will work fine. >> > >> > is it a bug? >> > >> > Cheers, >> > Andrea >> > >> > >> > >> > >> > 2016-03-14 15:00 GMT+01:00 Andrea Sella <andrea.se...@radicalbit.io>: >> >> >> >> Hi Robert, >> >> >> >> Ok, thank you. >> >> >> >> 2016-03-14 11:13 GMT+01:00 Robert Metzger <rmetz...@apache.org>: >> >>> >> >>> Hi Andrea, >> >>> >> >>> You don't have to manually replicate any operations on the slaves. All >> >>> files in the lib/ folder are transferred to all containers >> >>> (Jobmanagers and >> >>> TaskManagers). >> >>> >> >>> >> >>> On Sat, Mar 12, 2016 at 3:25 PM, Andrea Sella >> >>> <andrea.se...@radicalbit.io> wrote: >> >>>> >> >>>> Hi Ufuk, >> >>>> >> >>>> I'm trying to execute the WordCount batch example with input and >> >>>> output >> >>>> on Alluxio, i followed Running Flink on Alluxio and added the library >> >>>> to lib >> >>>> folder. Have I to replicate this operation on the slaves or YARN >> >>>> manage that >> >>>> and I must have the library just where I launch the job? >> >>>> >> >>>> Thanks, >> >>>> Andrea >> >>>> >> >>>> 2016-03-11 19:23 GMT+01:00 Ufuk Celebi <u...@apache.org>: >> >>>>> >> >>>>> Everything in the lib folder should be added to the classpath. Can >> >>>>> you >> >>>>> check the YARN client logs that the files are uploaded? Furthermore, >> >>>>> you can check the classpath of the JVM in the YARN logs of the >> >>>>> JobManager/TaskManager processes. >> >>>>> >> >>>>> – Ufuk >> >>>>> >> >>>>> >> >>>>> On Fri, Mar 11, 2016 at 5:33 PM, Andrea Sella >> >>>>> <andrea.se...@radicalbit.io> wrote: >> >>>>> > Hi, >> >>>>> > >> >>>>> > There is a way to add external dependencies to Flink Job, running >> >>>>> > on >> >>>>> > YARN, >> >>>>> > not using HADOOP_CLASSPATH? >> >>>>> > I am looking for a similar idea to standalone mode using lib >> >>>>> > folder. >> >>>>> > >> >>>>> > BR, >> >>>>> > Andrea >> >>>> >> >>>> >> >>> >> >> >> > > > > > > -- > BR, > Stefano Baghino > > Software Engineer @ Radicalbit
