Hello Everyone, I'm able to get my Flink UI up and running (it was related to the session manager plugin on my local laptop) but I'm not seeing any taskmanager/jobmanager logs in my Flink application. I have attached some yarn application logs while it's running but am not able to figure out how to stop and get more logs. Could someone please help me figure this out? I'm running Flink 1.11 on the EMR 6.1 cluster.
On Tue, Oct 27, 2020 at 1:06 PM Diwakar Jha <diwakar.n...@gmail.com> wrote:
> Hi Robert,
> Could please correct me. I'm not able to stop the app. Also, i
> stopped flink job already.
>
> sh-4.2$ yarn app -stop application_1603649952937_0002
> 2020-10-27 20:04:25,543 INFO client.RMProxy: Connecting to ResourceManager
> at ip-10-0-55-50.ec2.internal/10.0.55.50:8032
> 2020-10-27 20:04:25,717 INFO client.AHSProxy: Connecting to Application
> History server at ip-10-0-55-50.ec2.internal/10.0.55.50:10200
> Exception in thread "main" java.lang.IllegalArgumentException: App admin
> client class name not specified for type Apache Flink
> at
> org.apache.hadoop.yarn.client.api.AppAdminClient.createAppAdminClient(AppAdminClient.java:76)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:597)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:126)
> sh-4.2$
>
> On Tue, Oct 27, 2020 at 9:34 AM Robert Metzger <rmetz...@apache.org>
> wrote:
>
>> Hi,
>> are you intentionally not posting this response to the mailing list?
>>
>> As you can see from the yarn logs, log aggregation only works for
>> finished applications ("End of LogType:prelaunch.out.This log file belongs
>> to a running container (container_1603649952937_0002_01_000002) and so may
>> not be complete.")
>>
>> Please stop the app, then provide the logs.
>>
>>
>> On Tue, Oct 27, 2020 at 5:11 PM Diwakar Jha <diwakar.n...@gmail.com>
>> wrote:
>>
>>> Hi Robert,
>>>
>>> Yes, i'm using Flink on EMR using YARN. Please find attached the yarn
>>> logs -applicationId. I also attached haddop-yarn-nodemanager logs.
>>> Also, I followed this link below which has the same problem :
>>> http://mail-archives.apache.org/mod_mbox/flink-user/202009.mbox/%3CCAGDv3o5WyJTrXs9Pg+Vy-b+LwgEE26iN54iqE0=f5t+m8vw...@mail.gmail.com%3E
>>>
>>> https://www.talkend.net/post/75078.html
>>> Based on this I changed the log4j.properties.
>>> Let me know what you think. Please also let me know if you need some
>>> specific logs. Appreciate your help.
>>>
>>> Best,
>>> Diwakar
>>>
>>> On Tue, Oct 27, 2020 at 12:26 AM Robert Metzger <rmetz...@apache.org>
>>> wrote:
>>>
>>>> Hey Diwakar,
>>>>
>>>> how are you deploying Flink on EMR? Are you using YARN?
>>>> If so, you could also use log aggregation to see all the logs at once
>>>> (from both JobManager and TaskManagers). (yarn logs -applicationId
>>>> <Application ID>)
>>>>
>>>> Could you post (or upload somewhere) all logs you have of one run? It
>>>> is much easier for us to debug something if we have the full logs (the logs
>>>> show for example the classpath that you are using, we would see how you are
>>>> deploying Flink, etc.)
>>>>
>>>> From the information available, my guess is that you have modified your
>>>> deployment in some way (use of a custom logging version, custom deployment
>>>> method, version mixup with jars from both Flink 1.8 and 1.11, ...).
>>>>
>>>> Best,
>>>> Robert
>>>>
>>>>
>>>> On Tue, Oct 27, 2020 at 12:41 AM Diwakar Jha <diwakar.n...@gmail.com>
>>>> wrote:
>>>>
>>>>> This is what I see on the WebUI.
>>>>>
>>>>> 23:19:24.263 [flink-akka.actor.default-dispatcher-1865] ERROR
>>>>> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
>>>>> - Failed to transfer file from TaskExecutor
>>>>> container_1603649952937_0002_01_000004.
>>>>> java.util.concurrent.CompletionException:
>>>>> org.apache.flink.util.FlinkException: The file LOG does not exist on the
>>>>> TaskExecutor. at
>>>>> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$25(
>>>>> TaskExecutor.java:1742 <http://taskexecutor.java:1742/>)
>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0] at
>>>>> java.util.concurrent.CompletableFuture$AsyncSupply.run
>>>>> <http://java.util.concurrent.completablefuture$asyncsupply.run/>(
>>>>> CompletableFuture.java:1604 <http://completablefuture.java:1604/>)
>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>> ThreadPoolExecutor.java:1149 <http://threadpoolexecutor.java:1149/>)
>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run
>>>>> <http://java.util.concurrent.threadpoolexecutor$worker.run/>(
>>>>> ThreadPoolExecutor.java:624 <http://threadpoolexecutor.java:624/>)
>>>>> ~[?:1.8.0_252] at java.lang.Thread.run <http://java.lang.thread.run/>(
>>>>> Thread.java:748 <http://thread.java:748/>) ~[?:1.8.0_252] Caused by:
>>>>> org.apache.flink.util.FlinkException: The file LOG does not exist on the
>>>>> TaskExecutor. ... 5 more 23:19:24.275
>>>>> [flink-akka.actor.default-dispatcher-1865] ERROR
>>>>> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
>>>>> - Unhandled exception. org.apache.flink.util.FlinkException: The file LOG
>>>>> does not exist on the TaskExecutor. at
>>>>> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$25(
>>>>> TaskExecutor.java:1742 <http://taskexecutor.java:1742/>)
>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0] at
>>>>> java.util.concurrent.CompletableFuture$AsyncSupply.run
>>>>> <http://java.util.concurrent.completablefuture$asyncsupply.run/>(
>>>>> CompletableFuture.java:1604 <http://completablefuture.java:1604/>)
>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>> ThreadPoolExecutor.java:1149 <http://threadpoolexecutor.java:1149/>)
>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run
>>>>> <http://java.util.concurrent.threadpoolexecutor$worker.run/>(
>>>>> ThreadPoolExecutor.java:624 <http://threadpoolexecutor.java:624/>)
>>>>> ~[?:1.8.0_252] at java.lang.Thread.run <http://java.lang.thread.run/>(
>>>>> Thread.java:748 <http://thread.java:748/>) ~[?:1.8.0_252]
>>>>>
>>>>> Appreciate if anyone has any pointer for this.
>>>>>
>>>>> On Mon, Oct 26, 2020 at 10:45 AM Chesnay Schepler <ches...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Flink 1.11 uses slf4j 1.7.15; the easiest way to check the log files
>>>>>> is usually via the WebUI.
>>>>>>
>>>>>> On 10/26/2020 5:30 PM, Diwakar Jha wrote:
>>>>>>
>>>>>> I think my problem is with Sl4j library. I'm using sl4j 1.7 with
>>>>>> Flink 1.11. If that's correct then i appreciate if someone can point me
>>>>>> to
>>>>>> the exact Slf4j library that i should use with Flink 1.11
>>>>>>
>>>>>> Flink = 1.11.x;
>>>>>> Slf4j = 1.7;
>>>>>>
>>>>>>
>>>>>> On Sun, Oct 25, 2020 at 8:00 PM Diwakar Jha <diwakar.n...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for checking my configurations. Could you also point me where
>>>>>>> I can see the log files? Just to give more details. I'm trying to access
>>>>>>> these logs in AWS cloudwatch.
>>>>>>>
>>>>>>> Best,
>>>>>>> Diwakar
>>>>>>>
>>>>>>> On Sun, Oct 25, 2020 at 2:16 PM Chesnay Schepler <ches...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> With Flink 1.11 reporters were refactored to plugins, and are now
>>>>>>>> accessible by default (so you no longer have to bother with copying
>>>>>>>> jars
>>>>>>>> around).
>>>>>>>>
>>>>>>>> Your configuration appears to be correct, so I suggest to take a
>>>>>>>> look at the log files.
>>>>>>>>
>>>>>>>> On 10/25/2020 9:52 PM, Diwakar Jha wrote:
>>>>>>>>
>>>>>>>> Hello Everyone,
>>>>>>>>
>>>>>>>> I'm new to flink and i'm trying to upgrade from flink 1.8 to flink
>>>>>>>> 1.11 on an emr cluster. after upgrading to flink1.11 One of the
>>>>>>>> differences
>>>>>>>> that i see is i don't get any metrics. I found out that flink 1.11
>>>>>>>> does not
>>>>>>>> have *org.apache.flink.metrics.statsd.StatsDReporterFactory* jar
>>>>>>>> in /usr/lib/flink/opt which was the case for flink 1.8. Could anyone
>>>>>>>> have
>>>>>>>> any pointer to locate
>>>>>>>> *org.apache.flink.metrics.statsd.StatsDReporterFactory* jar or how
>>>>>>>> to use metrics in flink.1.11?
>>>>>>>>
>>>>>>>> Things i tried :
>>>>>>>> a) the below setup
>>>>>>>>
>>>>>>>> metrics.reporters: stsdmetrics.reporter.stsd.factory.class:
>>>>>>>> org.apache.flink.metrics.statsd.StatsDReporterFactorymetrics.reporter.stsd.host:
>>>>>>>> localhostmetrics.reporter.stsd.port: 8125
>>>>>>>>
>>>>>>>> b) I tried downloading the statsd jar from
>>>>>>>> https://mvnrepository.com/artifact/org.apache.flink/flink-metrics-statsd
>>>>>>>> putting it inside plugins/statsd directory.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best,
>>>>>>>> Diwakar Jha.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best,
>>>>>>> Diwakar Jha.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best,
>>>>>> Diwakar Jha.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Best,
>>>>> Diwakar Jha.
>>>>>
>>>>
>>>
>>> --
>>> Best,
>>> Diwakar Jha.
>>>
>>
>
> --
> Best,
> Diwakar Jha.
>
yarn
Description: Binary data
