How do you start the flink cluster? If it's a standalone cluster and you
don't use a shared directory, then you'll find the log of the taskmanager
on the machine on which the taskmanager runs. If you use YARN then you can
activate log aggregation to retrieve the log easily after the job has
finished.
Cheers,
Till
On Wed, Dec 9, 2015 at 2:30 PM, Filip Łęczycki <filipleczy...@gmail.com>
wrote:
> Hi Stephan, Till
>
> I am looking for the memory usage info in flink_<user>_jobmanager... log.
> As there are no other log files in /log dir I assumed this is the right
> place to look. I will try to setup a new clean instance and check whether
> it will work.
>
> Thank you very much for your assistance!
> Best regards,
> Filip Łęczycki
>
> Pozdrawiam,
> Filip Łęczycki
>
> 2015-12-09 11:13 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
>> Hi Filip!
>>
>> Someone else just used the memory logging with the exact described
>> settings - it worked.
>>
>> There is probably some mixup, you may be looking into the wrong log file,
>> or may setting the a value in a different config...
>>
>> Stephan
>>
>>
>> On Wed, Dec 9, 2015 at 10:58 AM, Till Rohrmann <trohrm...@apache.org>
>> wrote:
>>
>>> I assume you're looking in the taskmanager log file for the memory usage
>>> logging statements, right?
>>>
>>> Cheers,
>>> Till
>>>
>>> On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <filipleczy...@gmail.com
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> Thank you for your reply!
>>>>
>>>> I have made sure I restarted the TaskManager after changing config, but
>>>> it didn't resolve the issue.The config is loaded as I can see the following
>>>> line in the log:
>>>> 09:12:2015 00:00:21,894 DEBUG
>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>> configuration property: taskmanager.debug.memory.startLogThread, true
>>>>
>>>> I am running a job on local standalone flink instance and my
>>>> log4j.properties look like this:
>>>> log4j.rootLogger=DEBUG, file
>>>>
>>>> # Log all infos in the given file
>>>> log4j.appender.file=org.apache.log4j.FileAppender
>>>> log4j.appender.file.file=${log.file}
>>>> log4j.appender.file.append=false
>>>> log4j.appender.file.layout=org.apache.log4j.PatternLayout
>>>> log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS}
>>>> %-5p %-60c %x - %m%n
>>>>
>>>> # suppress the irrelevant (wrong) warnings from the netty channel
>>>> handler
>>>> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
>>>>
>>>> Maybe I can set some property in the job's code so that it would force
>>>> such verbose logging? I need those logs to run some flink performance
>>>> tests but maybe I can somehow extract the benchmark results without them,
>>>> do
>>>> you know any other way to monitor Flink Job's memory usage and GC time,
>>>> other than looking at web interface?
>>>>
>>>> Best regards,
>>>> Filip Łęczycki
>>>>
>>>> Pozdrawiam,
>>>> Filip Łęczycki
>>>>
>>>> 2015-12-08 20:48 GMT+01:00 Stephan Ewen <se...@apache.org>:
>>>>
>>>>> Hi!
>>>>>
>>>>> That is exactly the right way to do it. Logging has to be at least
>>>>> INFO and the parameter "taskmanager.debug.memory.startLogThread" set to
>>>>> true.
>>>>> The log output should be under
>>>>> "org.apache.flink.runtime.taskmanager.TaskManager".
>>>>>
>>>>> Do you see other outputs for that class in the log?
>>>>>
>>>>> Make sure you restarted the TaskManager processes after you changed
>>>>> the config file.
>>>>>
>>>>> Greetings,
>>>>> Stephan
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <
>>>>> filipleczy...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to enable logging of memory usage on flink 0.10.0 by
>>>>>> adding:
>>>>>>
>>>>>> taskmanager.debug.memory.startLogThread: true
>>>>>>
>>>>>> to config.yaml and setting log4j level to DEBUG however in the logs
>>>>>> after running the job I cannot see any info regarding memory usage.My
>>>>>> job lasted 30s so it should catch few intervals. Should I change
>>>>>> something else in the configuration?
>>>>>>
>>>>>>
>>>>>> Best regards/Pozdrawiam,
>>>>>>
>>>>>> Filip Łęczycki
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
