Re: problem on yarn cluster

2015-05-20 Thread Michele Bertoni
It worked! We used the 0.9-SNAPSHOT as you said, and it worked perfectly, also with larger data set we didn’t face any outofmemory problem thanks Il giorno 18/mag/2015, alle ore 21:19, Robert Metzger mailto:rmetz...@apache.org>> ha scritto: You are not getting the issue in your local environme

Re: problem on yarn cluster

2015-05-18 Thread Robert Metzger
You are not getting the issue in your local environment, because there everything is running in one JVM and the needed class is available there. In the distributed case, we have a special usercode classloader which can load classes from the user's jar. On Mon, May 18, 2015 at 9:16 PM, Michele Bert

Re: problem on yarn cluster

2015-05-18 Thread Michele Bertoni
Got it! thank you Best, Michele Il giorno 18/mag/2015, alle ore 19:29, Robert Metzger mailto:rmetz...@apache.org>> ha scritto: Hi, ./yarn-session.sh -n 3 -s 1 -jm 1024 -tm 4096 will start a YARN session with 4 containers (3 workers, 1 master). Once the session is running, you can submit a

Re: problem on yarn cluster

2015-05-18 Thread Michele Bertoni
Hi! thanks for answer! for 1 you are right! I am using collect method on custom object, tomorrow I will try moving back to snapshot. why is it working correctly in my local environment (that is also a milestone1)? on the other side i am not collecting large results: i am running on a test datas

Re: problem on yarn cluster

2015-05-18 Thread Robert Metzger
Hi, ./yarn-session.sh -n 3 -s 1 -jm 1024 -tm 4096 will start a YARN session with 4 containers (3 workers, 1 master). Once the session is running, you can submit as many jobs as you want to this session, using ./bin/flink run ./path/to/jar The YARN session will create a hidden file in conf/ whic

Re: problem on yarn cluster

2015-05-18 Thread Stephan Ewen
Hi Michele! It looks like there are quite a few things going wrong in your case. Let me see what I can deduce from the output you are showing me: 1) You seem to run into a bug that exists in the 0.9-milestone-1 and has been fixed in the master: As far as I can tell, you call "collect()" on a da

problem on yarn cluster

2015-05-18 Thread Michele Bertoni
Hi, I have a problem running my app on a Yarn cluster I developed it in my computer and everything is working fine then we setup the environment on Amazon EMR reading data from HDFS not S3 we run it with these command ./yarn-session.sh -n 3 -s 1 -jm 1024 -tm 4096 ./flink run -m yarn-cluste