Sounds like a bug. I've only tried this with yarn-sessions. Could it be that the behaviour is different between the two variants? In any case, I sounds a lot like a bug in Flink. Robert can probably chime in here with details.
On Wed, Mar 16, 2016 at 6:25 PM, Andrea Sella <andrea.se...@radicalbit.io> wrote: > Hi, > > After few tests I am able to write and read on Alluxio. > I am using Flink 1.0.0 and in my case external libraries are not loaded from > lib folder to classpath, it loads only flink-dist_2.11-1.0.0.jar. I need to > specify the folder with -yt parameter to load the others. > > If I run `/bin/flink run -m yarn-cluster -yn 4 -yjm 2048 -ytm 4096 some.jar > --input alluxio://somepath` it will throws an exception related to a missing > library ie. Class alluxio.hadoop.FileSystem not found > If I run `./bin/flink run -m yarn-cluster -yt lib/ -yn 4 -yjm 2048 -ytm 4096 > some.jar --input alluxio://somepath` with -yt params it will work fine. > > is it a bug? > > Cheers, > Andrea > > > > > 2016-03-14 15:00 GMT+01:00 Andrea Sella <andrea.se...@radicalbit.io>: >> >> Hi Robert, >> >> Ok, thank you. >> >> 2016-03-14 11:13 GMT+01:00 Robert Metzger <rmetz...@apache.org>: >>> >>> Hi Andrea, >>> >>> You don't have to manually replicate any operations on the slaves. All >>> files in the lib/ folder are transferred to all containers (Jobmanagers and >>> TaskManagers). >>> >>> >>> On Sat, Mar 12, 2016 at 3:25 PM, Andrea Sella >>> <andrea.se...@radicalbit.io> wrote: >>>> >>>> Hi Ufuk, >>>> >>>> I'm trying to execute the WordCount batch example with input and output >>>> on Alluxio, i followed Running Flink on Alluxio and added the library to >>>> lib >>>> folder. Have I to replicate this operation on the slaves or YARN manage >>>> that >>>> and I must have the library just where I launch the job? >>>> >>>> Thanks, >>>> Andrea >>>> >>>> 2016-03-11 19:23 GMT+01:00 Ufuk Celebi <u...@apache.org>: >>>>> >>>>> Everything in the lib folder should be added to the classpath. Can you >>>>> check the YARN client logs that the files are uploaded? Furthermore, >>>>> you can check the classpath of the JVM in the YARN logs of the >>>>> JobManager/TaskManager processes. >>>>> >>>>> – Ufuk >>>>> >>>>> >>>>> On Fri, Mar 11, 2016 at 5:33 PM, Andrea Sella >>>>> <andrea.se...@radicalbit.io> wrote: >>>>> > Hi, >>>>> > >>>>> > There is a way to add external dependencies to Flink Job, running on >>>>> > YARN, >>>>> > not using HADOOP_CLASSPATH? >>>>> > I am looking for a similar idea to standalone mode using lib folder. >>>>> > >>>>> > BR, >>>>> > Andrea >>>> >>>> >>> >> >
