Hi,
I wanted to check if anyone can help me with the logs. I have sent several
emails but not getting any response.
I'm running Flink 1.11 on EMR 6.1. I don't see any logs though I get this
stdout error. I'm trying to upgrade Flink 1.8 to Flink 1.11
18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
- Failed to transfer file from TaskExecutor
container_1604033334508_0001_01_000004.
java.util.concurrent.CompletionException: org.apache.flink.util.
FlinkException: The file LOG does not exist on the TaskExecutor.
Thanks!
On Fri, Oct 30, 2020 at 9:04 AM Diwakar Jha <diwakar.n...@gmail.com> wrote:
> Hello,
>
> I see that in my class path (below) I have both log4j-1 and lo4j-api-2. is
> this because of which i'm not seeing any logs. If so, could someone suggest
> how to fix it?
>
> export
> CLASSPATH=":lib/flink-csv-1.11.0.jar:lib/flink-json-1.11.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table-blink_2.12-1.11.0.jar:lib/flink-table_2.12-1.11.0.jar:
> *lib/log4j-1.2-api-2.12.1.jar:lib/log4j-api-2.12.1.jar*
> :lib/log4j-core-2.12.1.jar:lib/
>
> export
> _FLINK_CLASSPATH=":lib/flink-csv-1.11.0.jar:lib/flink-json-1.11.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table-blink_2.12-1.11.0.jar:lib/flink-table_2.12-1.11.0.jar:
> *lib/log4j-1.2-api-2.12.1.jar:lib/log4j-api-2.12.1.jar*
> :lib/log4j-core-2.12.1.jar:lib/log4j-slf4j-impl-2.12.1.jar:flink-dist_2.12-1.11.0.jar:flink-conf.yaml:"
>
> thanks.
>
> On Thu, Oct 29, 2020 at 6:21 PM Diwakar Jha <diwakar.n...@gmail.com>
> wrote:
>
>> Hello Everyone,
>>
>> I'm able to get my Flink UI up and running (it was related to the session
>> manager plugin on my local laptop) but I'm not seeing any
>> taskmanager/jobmanager logs in my Flink application. I have attached some
>> yarn application logs while it's running but am not able to figure out how
>> to stop and get more logs. Could someone please help me figure this out?
>> I'm running Flink 1.11 on the EMR 6.1 cluster.
>>
>> On Tue, Oct 27, 2020 at 1:06 PM Diwakar Jha <diwakar.n...@gmail.com>
>> wrote:
>>
>>> Hi Robert,
>>> Could please correct me. I'm not able to stop the app. Also, i
>>> stopped flink job already.
>>>
>>> sh-4.2$ yarn app -stop application_1603649952937_0002
>>> 2020-10-27 20:04:25,543 INFO client.RMProxy: Connecting to
>>> ResourceManager at ip-10-0-55-50.ec2.internal/10.0.55.50:8032
>>> 2020-10-27 20:04:25,717 INFO client.AHSProxy: Connecting to Application
>>> History server at ip-10-0-55-50.ec2.internal/10.0.55.50:10200
>>> Exception in thread "main" java.lang.IllegalArgumentException: App admin
>>> client class name not specified for type Apache Flink
>>> at
>>> org.apache.hadoop.yarn.client.api.AppAdminClient.createAppAdminClient(AppAdminClient.java:76)
>>> at
>>> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:597)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>>> at
>>> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:126)
>>> sh-4.2$
>>>
>>> On Tue, Oct 27, 2020 at 9:34 AM Robert Metzger <rmetz...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>> are you intentionally not posting this response to the mailing list?
>>>>
>>>> As you can see from the yarn logs, log aggregation only works for
>>>> finished applications ("End of LogType:prelaunch.out.This log file belongs
>>>> to a running container (container_1603649952937_0002_01_000002) and so may
>>>> not be complete.")
>>>>
>>>> Please stop the app, then provide the logs.
>>>>
>>>>
>>>> On Tue, Oct 27, 2020 at 5:11 PM Diwakar Jha <diwakar.n...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Robert,
>>>>>
>>>>> Yes, i'm using Flink on EMR using YARN. Please find attached the yarn
>>>>> logs -applicationId. I also attached haddop-yarn-nodemanager logs.
>>>>> Also, I followed this link below which has the same problem :
>>>>> http://mail-archives.apache.org/mod_mbox/flink-user/202009.mbox/%3CCAGDv3o5WyJTrXs9Pg+Vy-b+LwgEE26iN54iqE0=f5t+m8vw...@mail.gmail.com%3E
>>>>>
>>>>> https://www.talkend.net/post/75078.html
>>>>> Based on this I changed the log4j.properties.
>>>>> Let me know what you think. Please also let me know if you need some
>>>>> specific logs. Appreciate your help.
>>>>>
>>>>> Best,
>>>>> Diwakar
>>>>>
>>>>> On Tue, Oct 27, 2020 at 12:26 AM Robert Metzger <rmetz...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hey Diwakar,
>>>>>>
>>>>>> how are you deploying Flink on EMR? Are you using YARN?
>>>>>> If so, you could also use log aggregation to see all the logs at once
>>>>>> (from both JobManager and TaskManagers). (yarn logs -applicationId
>>>>>> <Application ID>)
>>>>>>
>>>>>> Could you post (or upload somewhere) all logs you have of one run? It
>>>>>> is much easier for us to debug something if we have the full logs (the
>>>>>> logs
>>>>>> show for example the classpath that you are using, we would see how you
>>>>>> are
>>>>>> deploying Flink, etc.)
>>>>>>
>>>>>> From the information available, my guess is that you have modified
>>>>>> your deployment in some way (use of a custom logging version, custom
>>>>>> deployment method, version mixup with jars from both Flink 1.8 and 1.11,
>>>>>> ...).
>>>>>>
>>>>>> Best,
>>>>>> Robert
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 27, 2020 at 12:41 AM Diwakar Jha <diwakar.n...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> This is what I see on the WebUI.
>>>>>>>
>>>>>>> 23:19:24.263 [flink-akka.actor.default-dispatcher-1865] ERROR
>>>>>>> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
>>>>>>> - Failed to transfer file from TaskExecutor
>>>>>>> container_1603649952937_0002_01_000004.
>>>>>>> java.util.concurrent.CompletionException:
>>>>>>> org.apache.flink.util.FlinkException: The file LOG does not exist on the
>>>>>>> TaskExecutor. at
>>>>>>> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$25(
>>>>>>> TaskExecutor.java:1742 <http://taskexecutor.java:1742/>)
>>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0] at
>>>>>>> java.util.concurrent.CompletableFuture$AsyncSupply.run
>>>>>>> <http://java.util.concurrent.completablefuture$asyncsupply.run/>(
>>>>>>> CompletableFuture.java:1604 <http://completablefuture.java:1604/>)
>>>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1149 <http://threadpoolexecutor.java:1149/>)
>>>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run
>>>>>>> <http://java.util.concurrent.threadpoolexecutor$worker.run/>(
>>>>>>> ThreadPoolExecutor.java:624 <http://threadpoolexecutor.java:624/>)
>>>>>>> ~[?:1.8.0_252] at java.lang.Thread.run
>>>>>>> <http://java.lang.thread.run/>(Thread.java:748
>>>>>>> <http://thread.java:748/>) ~[?:1.8.0_252] Caused by:
>>>>>>> org.apache.flink.util.FlinkException: The file LOG does not exist on the
>>>>>>> TaskExecutor. ... 5 more 23:19:24.275
>>>>>>> [flink-akka.actor.default-dispatcher-1865] ERROR
>>>>>>> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
>>>>>>> - Unhandled exception. org.apache.flink.util.FlinkException: The file
>>>>>>> LOG
>>>>>>> does not exist on the TaskExecutor. at
>>>>>>> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$25(
>>>>>>> TaskExecutor.java:1742 <http://taskexecutor.java:1742/>)
>>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0] at
>>>>>>> java.util.concurrent.CompletableFuture$AsyncSupply.run
>>>>>>> <http://java.util.concurrent.completablefuture$asyncsupply.run/>(
>>>>>>> CompletableFuture.java:1604 <http://completablefuture.java:1604/>)
>>>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1149 <http://threadpoolexecutor.java:1149/>)
>>>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run
>>>>>>> <http://java.util.concurrent.threadpoolexecutor$worker.run/>(
>>>>>>> ThreadPoolExecutor.java:624 <http://threadpoolexecutor.java:624/>)
>>>>>>> ~[?:1.8.0_252] at java.lang.Thread.run
>>>>>>> <http://java.lang.thread.run/>(Thread.java:748
>>>>>>> <http://thread.java:748/>) ~[?:1.8.0_252]
>>>>>>>
>>>>>>> Appreciate if anyone has any pointer for this.
>>>>>>>
>>>>>>> On Mon, Oct 26, 2020 at 10:45 AM Chesnay Schepler <
>>>>>>> ches...@apache.org> wrote:
>>>>>>>
>>>>>>>> Flink 1.11 uses slf4j 1.7.15; the easiest way to check the log
>>>>>>>> files is usually via the WebUI.
>>>>>>>>
>>>>>>>> On 10/26/2020 5:30 PM, Diwakar Jha wrote:
>>>>>>>>
>>>>>>>> I think my problem is with Sl4j library. I'm using sl4j 1.7 with
>>>>>>>> Flink 1.11. If that's correct then i appreciate if someone can point
>>>>>>>> me to
>>>>>>>> the exact Slf4j library that i should use with Flink 1.11
>>>>>>>>
>>>>>>>> Flink = 1.11.x;
>>>>>>>> Slf4j = 1.7;
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Oct 25, 2020 at 8:00 PM Diwakar Jha <diwakar.n...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for checking my configurations. Could you also point me
>>>>>>>>> where I can see the log files? Just to give more details. I'm trying
>>>>>>>>> to
>>>>>>>>> access these logs in AWS cloudwatch.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Diwakar
>>>>>>>>>
>>>>>>>>> On Sun, Oct 25, 2020 at 2:16 PM Chesnay Schepler <
>>>>>>>>> ches...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> With Flink 1.11 reporters were refactored to plugins, and are now
>>>>>>>>>> accessible by default (so you no longer have to bother with copying
>>>>>>>>>> jars
>>>>>>>>>> around).
>>>>>>>>>>
>>>>>>>>>> Your configuration appears to be correct, so I suggest to take a
>>>>>>>>>> look at the log files.
>>>>>>>>>>
>>>>>>>>>> On 10/25/2020 9:52 PM, Diwakar Jha wrote:
>>>>>>>>>>
>>>>>>>>>> Hello Everyone,
>>>>>>>>>>
>>>>>>>>>> I'm new to flink and i'm trying to upgrade from flink 1.8 to
>>>>>>>>>> flink 1.11 on an emr cluster. after upgrading to flink1.11 One of the
>>>>>>>>>> differences that i see is i don't get any metrics. I found out that
>>>>>>>>>> flink
>>>>>>>>>> 1.11 does not have
>>>>>>>>>> *org.apache.flink.metrics.statsd.StatsDReporterFactory* jar in
>>>>>>>>>> /usr/lib/flink/opt which was the case for flink 1.8. Could anyone
>>>>>>>>>> have any
>>>>>>>>>> pointer to locate
>>>>>>>>>> *org.apache.flink.metrics.statsd.StatsDReporterFactory* jar or
>>>>>>>>>> how to use metrics in flink.1.11?
>>>>>>>>>>
>>>>>>>>>> Things i tried :
>>>>>>>>>> a) the below setup
>>>>>>>>>>
>>>>>>>>>> metrics.reporters: stsdmetrics.reporter.stsd.factory.class:
>>>>>>>>>> org.apache.flink.metrics.statsd.StatsDReporterFactorymetrics.reporter.stsd.host:
>>>>>>>>>> localhostmetrics.reporter.stsd.port: 8125
>>>>>>>>>>
>>>>>>>>>> b) I tried downloading the statsd jar from
>>>>>>>>>> https://mvnrepository.com/artifact/org.apache.flink/flink-metrics-statsd
>>>>>>>>>> putting it inside plugins/statsd directory.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best,
>>>>>>>>>> Diwakar Jha.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best,
>>>>>>>>> Diwakar Jha.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best,
>>>>>>>> Diwakar Jha.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best,
>>>>>>> Diwakar Jha.
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Best,
>>>>> Diwakar Jha.
>>>>>
>>>>
>>>
>>> --
>>> Best,
>>> Diwakar Jha.
>>>
>>
