I think the YARN application has to be finished in order for the logs to be
accessible.
Judging from you commands, you’re starting a long running YARN application
running Flink with ./bin/yarn-session.sh -n 1 -tm 2048 -s 4. This cluster
won’t be used though, because you’re executing your job with ./bin/flink
run -m yarn-cluster which will start another YARN application which is only
alive as long as the Flink job is executed. If you want to run your job on
the long running YARN application, then you simply have to omit -m
yarn-cluster.
Cheers,
Till
On Mon, Jan 4, 2016 at 12:36 PM, Ana M. Martinez <a...@cs.aau.dk> wrote:
> Hi Till,
>
> Sorry for the delay (Xmas break). I have activated log aggregation
> on flink-conf.yaml with yarn.log-aggregation-enable: true (as I can’t find
> a yarn-site.xml).
> But the command yarn logs -applicationId application_1451903796996_0008
> gives me the following output:
>
> INFO client.RMProxy: Connecting to ResourceManager at xxx
> /var/log/hadoop-yarn/apps/hadoop/logs/application_1451903796996_0008does
> not exist.
> Log aggregation has not completed or is not enabled
>
>
> I’ve tried to restart the Flink JobManager and TaskManagers as follows:
> ./bin/yarn-session.sh -n 1 -tm 2048 -s 4
> and then with a detached screen, run my application with ./bin/flink run
> -m yarn-cluster ...
>
> I am not sure if my problem is that I am not setting the
> log-aggregation-enable property well or I am not restarting the Flink
> JobManager and TaskManagers as I should… Any idea?
>
> Thanks,
> Ana
>
> On 18 Dec 2015, at 16:29, Till Rohrmann <trohrm...@apache.org> wrote:
>
> In which log file are you exactly looking for the logging statements? And
> on what machine? You have to look on the machines on which the yarn
> container were started. Alternatively if you have log aggregation
> activated, then you can simply retrieve the log files via yarn logs.
>
> Cheers,
> Till
>
> On Fri, Dec 18, 2015 at 3:49 PM, Ana M. Martinez <a...@cs.aau.dk> wrote:
>
>> Hi Till,
>>
>> Many thanks for your quick response.
>>
>> I have modified the WordCountExample to re-reproduce my problem in a
>> simple example.
>>
>> I run the code below with the following command:
>> ./bin/flink run -m yarn-cluster -yn 1 -ys 4 -yjm 1024 -ytm 1024 -c
>> mypackage.WordCountExample ../flinklink.jar
>>
>> And if I check the log file I see all logger messages except the one in
>> the flatMap function of the inner LineSplitter class, which is actually the
>> one I am most interested in.
>>
>> Is that an expected behaviour?
>>
>> Thanks,
>> Ana
>>
>> import org.apache.flink.api.common.functions.FlatMapFunction;
>> import org.apache.flink.api.java.DataSet;
>> import org.apache.flink.api.java.ExecutionEnvironment;
>> import org.apache.flink.api.java.tuple.Tuple2;
>> import org.apache.flink.util.Collector;
>> import org.slf4j.Logger;
>> import org.slf4j.LoggerFactory;
>>
>> import java.io.Serializable;
>> import java.util.ArrayList;
>> import java.util.List;
>>
>> public class WordCountExample {
>> static Logger logger = LoggerFactory.getLogger(WordCountExample.class);
>>
>> public static void main(String[] args) throws Exception {
>> final ExecutionEnvironment env =
>> ExecutionEnvironment.getExecutionEnvironment();
>>
>> logger.info("Entering application.");
>>
>> DataSet<String> text = env.fromElements(
>> "Who's there?",
>> "I think I hear them. Stand, ho! Who's there?");
>>
>> List<Integer> elements = new ArrayList<Integer>();
>> elements.add(0);
>>
>>
>> DataSet<TestClass> set = env.fromElements(new TestClass(elements));
>>
>> DataSet<Tuple2<String, Integer>> wordCounts = text
>> .flatMap(new LineSplitter())
>> .withBroadcastSet(set, "set")
>> .groupBy(0)
>> .sum(1);
>>
>> wordCounts.print();
>>
>>
>> }
>>
>> public static class LineSplitter implements FlatMapFunction<String,
>> Tuple2<String, Integer>> {
>>
>> static Logger loggerLineSplitter =
>> LoggerFactory.getLogger(LineSplitter.class);
>>
>> @Override
>> public void flatMap(String line, Collector<Tuple2<String, Integer>>
>> out) {
>> loggerLineSplitter.info("Logger in LineSplitter.flatMap");
>> for (String word : line.split(" ")) {
>> out.collect(new Tuple2<String, Integer>(word, 1));
>> }
>> }
>> }
>>
>> public static class TestClass implements Serializable {
>> private static final long serialVersionUID = -2932037991574118651L;
>>
>> static Logger loggerTestClass =
>> LoggerFactory.getLogger("WordCountExample.TestClass");
>>
>> List<Integer> integerList;
>> public TestClass(List<Integer> integerList){
>> this.integerList=integerList;
>> loggerTestClass.info("Logger in TestClass");
>> }
>>
>>
>> }
>> }
>>
>>
>>
>>
>> On 17 Dec 2015, at 16:08, Till Rohrmann <trohrm...@apache.org> wrote:
>>
>> Hi Ana,
>>
>> you can simply modify the `log4j.properties` file in the `conf`
>> directory. It should be automatically included in the Yarn application.
>>
>> Concerning your logging problem, it might be that you have set the
>> logging level too high. Could you share the code with us?
>>
>> Cheers,
>> Till
>>
>> On Thu, Dec 17, 2015 at 1:56 PM, Ana M. Martinez <a...@cs.aau.dk> wrote:
>>
>>> Hi flink community,
>>>
>>> I am trying to show log messages using log4j.
>>> It works fine overall except for the messages I want to show in an inner
>>> class that implements
>>> org.apache.flink.api.common.aggregators.ConvergenceCriterion.
>>> I am very new to this, but it seems that I’m having problems to show the
>>> messages included in the isConverged function, as it runs in the task
>>> managers?
>>> E.g. the log messages in the outer class (before map-reduce operations)
>>> are properly shown.
>>>
>>> I am also interested in providing my own log4j.properties file. I am
>>> using the ./bin/flink run -m yarn-cluster on Amazon clusters.
>>>
>>> Thanks,
>>> Ana
>>
>>
>>
>>
>
>
