Hi, After few tests I am able to write and read on Alluxio. I am using Flink 1.0.0 and in my case external libraries are not loaded from lib folder to classpath, it loads only flink-dist_2.11-1.0.0.jar. I need to specify the folder with -yt parameter to load the others.
If I run `/bin/flink run -m yarn-cluster -yn 4 -yjm 2048 -ytm 4096 some.jar --input alluxio://somepath` it will throws an exception related to a missing library ie. Class alluxio.hadoop.FileSystem not found If I run `./bin/flink run -m yarn-cluster -yt lib/ -yn 4 -yjm 2048 -ytm 4096 some.jar --input alluxio://somepath` with -yt params it will work fine. is it a bug? Cheers, Andrea 2016-03-14 15:00 GMT+01:00 Andrea Sella <andrea.se...@radicalbit.io>: > Hi Robert, > > Ok, thank you. > > 2016-03-14 11:13 GMT+01:00 Robert Metzger <rmetz...@apache.org>: > >> Hi Andrea, >> >> You don't have to manually replicate any operations on the slaves. All >> files in the lib/ folder are transferred to all containers (Jobmanagers and >> TaskManagers). >> >> >> On Sat, Mar 12, 2016 at 3:25 PM, Andrea Sella <andrea.se...@radicalbit.io >> > wrote: >> >>> Hi Ufuk, >>> >>> I'm trying to execute the WordCount batch example with input and output >>> on Alluxio, i followed Running Flink on Alluxio >>> <http://www.alluxio.org/documentation/en/Running-Flink-on-Alluxio.html> and >>> added the library to lib folder. Have I to replicate this operation on the >>> slaves or YARN manage that and I must have the library just where I launch >>> the job? >>> >>> Thanks, >>> Andrea >>> >>> 2016-03-11 19:23 GMT+01:00 Ufuk Celebi <u...@apache.org>: >>> >>>> Everything in the lib folder should be added to the classpath. Can you >>>> check the YARN client logs that the files are uploaded? Furthermore, >>>> you can check the classpath of the JVM in the YARN logs of the >>>> JobManager/TaskManager processes. >>>> >>>> – Ufuk >>>> >>>> >>>> On Fri, Mar 11, 2016 at 5:33 PM, Andrea Sella >>>> <andrea.se...@radicalbit.io> wrote: >>>> > Hi, >>>> > >>>> > There is a way to add external dependencies to Flink Job, running on >>>> YARN, >>>> > not using HADOOP_CLASSPATH? >>>> > I am looking for a similar idea to standalone mode using lib folder. >>>> > >>>> > BR, >>>> > Andrea >>>> >>> >>> >> >
