Re: Problem to show logs in task managers

Hi Till, Sorry for the delay, you were right, I was not restarting the yarn cluster… Many thanks for your help! Ana On 11 Jan 2016, at 14:39, Till Rohrmann mailto:trohrm...@apache.org>> wrote: You have to restart the yarn cluster to let your changes take effect. You can do that via HADOOP_HO

Re: Problem to show logs in task managers

You have to restart the yarn cluster to let your changes take effect. You can do that via HADOOP_HOME/sbin/stop-yarn.sh; HADOOP_HOME/sbin/start-yarn.sh. The commands yarn-session.sh ... and bin/flink run -m yarn cluster start a new yarn application within the yarn cluster. Cheers, Till ​ On Mon,

Re: Problem to show logs in task managers

Hi Till, Thanks for your help. I have checked both in Yarn’s web interface and through command line and it seems that there are not occupied containers. Additionally, I have checked the configuration values in the web interface and even though I have changed the log.aggregation property in the

Re: Problem to show logs in task managers

Hi Ana, good to hear that you found the logging statements. You can check in Yarn’s web interface whether there are still occupied containers. Alternatively you can go to the different machines and run jps which lists you the running Java processes. If you see an ApplicationMaster or YarnTaskManag

Re: Problem to show logs in task managers

Hi Till, Thanks for that! I can see the "Logger in LineSplitter.flatMap” output if I retrieve the task manager logs manually (under /var/log/hadoop-yarn/containers/application_X/…). However that solution is not ideal when for instance I am using 32 machines for my mapReduce operations. I would

Re: Problem to show logs in task managers

You’re right that the log statements of the LineSplitter are in the logs of the cluster nodes, because that’s where the LineSplitter code is executed. In contrast, you create a TestClass on the client when you submit the program. Therefore, you see the logging statement “Logger in TestClass” on the

Re: Problem to show logs in task managers

Thanks for the tip Robert! It was a good idea to rule out other possible causes, but I am afraid that is not the problem. If we stick to the WordCountExample (for simplicity), the Exception is thrown if placed into the flatMap function. I am going to try to re-write my problem and all the setti

Re: Problem to show logs in task managers

Maybe the isConverged() method is never called? For making that sure, just throw a RuntimeException inside the method to see whats happening. On Wed, Jan 6, 2016 at 3:58 PM, Ana M. Martinez wrote: > Hi Till, > > I am afraid it does not work in any case. > > I am following the steps you indicate

Re: Problem to show logs in task managers

Hi Till, I am afraid it does not work in any case. I am following the steps you indicate on your websites (for yarn configuration and loggers with slf4j): 1) Enable log aggregation in yarn-site: https://ci.apache.org/projects/flink/flink-docs-release-0.7/yarn_setup.html#log-files 2) Include Lo

Re: Problem to show logs in task managers

I think the YARN application has to be finished in order for the logs to be accessible. Judging from you commands, you’re starting a long running YARN application running Flink with ./bin/yarn-session.sh -n 1 -tm 2048 -s 4. This cluster won’t be used though, because you’re executing your job with

Re: Problem to show logs in task managers

Hi Till, Sorry for the delay (Xmas break). I have activated log aggregation on flink-conf.yaml with yarn.log-aggregation-enable: true (as I can’t find a yarn-site.xml). But the command yarn logs -applicationId application_1451903796996_0008 gives me the following output: INFO client.RMProxy: C

Re: Problem to show logs in task managers

In which log file are you exactly looking for the logging statements? And on what machine? You have to look on the machines on which the yarn container were started. Alternatively if you have log aggregation activated, then you can simply retrieve the log files via yarn logs. Cheers, Till On Fri,

Re: Problem to show logs in task managers

Hi Till, Many thanks for your quick response. I have modified the WordCountExample to re-reproduce my problem in a simple example. I run the code below with the following command: ./bin/flink run -m yarn-cluster -yn 1 -ys 4 -yjm 1024 -ytm 1024 -c mypackage.WordCountExample ../flinklink.jar An

Re: Problem to show logs in task managers

Hi Ana, you can simply modify the `log4j.properties` file in the `conf` directory. It should be automatically included in the Yarn application. Concerning your logging problem, it might be that you have set the logging level too high. Could you share the code with us? Cheers, Till On Thu, Dec 1

Problem to show logs in task managers

Hi flink community, I am trying to show log messages using log4j. It works fine overall except for the messages I want to show in an inner class that implements org.apache.flink.api.common.aggregators.ConvergenceCriterion. I am very new to this, but it seems that I’m having problems to show the