Re: Flink and YARN ship folder

I have another interesting test result on this matter, running again an individual job on a YARN cluster. When running bin/flink run -m yarn-cluster -yn 1 examples/batch/WordCount.jar the job fails with an error in the Job Manager LogType:jobmanager.err Log Upload Time:Thu Mar 17 07:05:32 -0400 2

Re: Flink and YARN ship folder

Can you try the same thing without -yt, but a yarn-session? – Ufuk On Thu, Mar 17, 2016 at 12:29 PM, Stefano Baghino wrote: > I have another interesting test result on this matter, running again an > individual job on a YARN cluster. > > When running bin/flink run -m yarn-cluster -yn 1 > example

Re: Flink and YARN ship folder

Hi Ufuk, I've just run a long running session and in that mode the libraries are correctly shipped without the need to specify -t lib. If you can double check the issue I can open an issue on JIRA. Thanks for helping us. On Thu, Mar 17, 2016 at 2:16 PM, Ufuk Celebi wrote: > Can you try the sa

Re: Flink and YARN ship folder

Hi, After few tests I am able to write and read on Alluxio. I am using Flink 1.0.0 and in my case external libraries are not loaded from lib folder to classpath, it loads only flink-dist_2.11-1.0.0.jar. I need to specify the folder with -yt parameter to load the others. If I run `/bin/flink run -

Re: Flink and YARN ship folder

Sounds like a bug. I've only tried this with yarn-sessions. Could it be that the behaviour is different between the two variants? In any case, I sounds a lot like a bug in Flink. Robert can probably chime in here with details. On Wed, Mar 16, 2016 at 6:25 PM, Andrea Sella wrote: > Hi, > > After f

Re: Flink and YARN ship folder

Thank you very much! Yes, please go ahead an open an issue for this. Thanks for reporting this! On Fri, Mar 18, 2016 at 5:08 PM, Stefano Baghino wrote: > Hi Ufuk, > > I've just run a long running session and in that mode the libraries are > correctly shipped without the need to specify -t lib. >

Re: Flink and YARN ship folder

Hi Robert, Ok, thank you. 2016-03-14 11:13 GMT+01:00 Robert Metzger : > Hi Andrea, > > You don't have to manually replicate any operations on the slaves. All > files in the lib/ folder are transferred to all containers (Jobmanagers and > TaskManagers). > > > On Sat, Mar 12, 2016 at 3:25 PM, Andr

Re: Flink and YARN ship folder

Hi Andrea, You don't have to manually replicate any operations on the slaves. All files in the lib/ folder are transferred to all containers (Jobmanagers and TaskManagers). On Sat, Mar 12, 2016 at 3:25 PM, Andrea Sella wrote: > Hi Ufuk, > > I'm trying to execute the WordCount batch example wit

Re: Flink and YARN ship folder

Hi Ufuk, I'm trying to execute the WordCount batch example with input and output on Alluxio, i followed Running Flink on Alluxio and added the library to lib folder. Have I to replicate this operation on the slaves or YARN man

Re: Flink and YARN ship folder

Everything in the lib folder should be added to the classpath. Can you check the YARN client logs that the files are uploaded? Furthermore, you can check the classpath of the JVM in the YARN logs of the JobManager/TaskManager processes. – Ufuk On Fri, Mar 11, 2016 at 5:33 PM, Andrea Sella wrote

Flink and YARN ship folder

Hi, There is a way to add external dependencies to Flink Job, running on YARN, not using HADOOP_CLASSPATH? I am looking for a similar idea to standalone mode using lib folder. BR, Andrea