Hello,
I see that in my class path (below) I have both log4j-1 and lo4j-api-2. is
this because of which i'm not seeing any logs. If so, could someone suggest
how to fix it?
export
CLASSPATH=":lib/flink-csv-1.11.0.jar:lib/flink-json-1.11.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table-blink_2.12-1.11.0.jar:lib/flink-table_2.12-1.11.0.jar:
*lib/log4j-1.2-api-2.12.1.jar:lib/log4j-api-2.12.1.jar*
:lib/log4j-core-2.12.1.jar:lib/
export
_FLINK_CLASSPATH=":lib/flink-csv-1.11.0.jar:lib/flink-json-1.11.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table-blink_2.12-1.11.0.jar:lib/flink-table_2.12-1.11.0.jar:
*lib/log4j-1.2-api-2.12.1.jar:lib/log4j-api-2.12.1.jar*
:lib/log4j-core-2.12.1.jar:lib/log4j-slf4j-impl-2.12.1.jar:flink-dist_2.12-1.11.0.jar:flink-conf.yaml:"
thanks.
On Thu, Oct 29, 2020 at 6:21 PM Diwakar Jha <diwakar.n...@gmail.com> wrote:
> Hello Everyone,
>
> I'm able to get my Flink UI up and running (it was related to the session
> manager plugin on my local laptop) but I'm not seeing any
> taskmanager/jobmanager logs in my Flink application. I have attached some
> yarn application logs while it's running but am not able to figure out how
> to stop and get more logs. Could someone please help me figure this out?
> I'm running Flink 1.11 on the EMR 6.1 cluster.
>
> On Tue, Oct 27, 2020 at 1:06 PM Diwakar Jha <diwakar.n...@gmail.com>
> wrote:
>
>> Hi Robert,
>> Could please correct me. I'm not able to stop the app. Also, i
>> stopped flink job already.
>>
>> sh-4.2$ yarn app -stop application_1603649952937_0002
>> 2020-10-27 20:04:25,543 INFO client.RMProxy: Connecting to
>> ResourceManager at ip-10-0-55-50.ec2.internal/10.0.55.50:8032
>> 2020-10-27 20:04:25,717 INFO client.AHSProxy: Connecting to Application
>> History server at ip-10-0-55-50.ec2.internal/10.0.55.50:10200
>> Exception in thread "main" java.lang.IllegalArgumentException: App admin
>> client class name not specified for type Apache Flink
>> at
>> org.apache.hadoop.yarn.client.api.AppAdminClient.createAppAdminClient(AppAdminClient.java:76)
>> at
>> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:597)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>> at
>> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:126)
>> sh-4.2$
>>
>> On Tue, Oct 27, 2020 at 9:34 AM Robert Metzger <rmetz...@apache.org>
>> wrote:
>>
>>> Hi,
>>> are you intentionally not posting this response to the mailing list?
>>>
>>> As you can see from the yarn logs, log aggregation only works for
>>> finished applications ("End of LogType:prelaunch.out.This log file belongs
>>> to a running container (container_1603649952937_0002_01_000002) and so may
>>> not be complete.")
>>>
>>> Please stop the app, then provide the logs.
>>>
>>>
>>> On Tue, Oct 27, 2020 at 5:11 PM Diwakar Jha <diwakar.n...@gmail.com>
>>> wrote:
>>>
>>>> Hi Robert,
>>>>
>>>> Yes, i'm using Flink on EMR using YARN. Please find attached the yarn
>>>> logs -applicationId. I also attached haddop-yarn-nodemanager logs.
>>>> Also, I followed this link below which has the same problem :
>>>> http://mail-archives.apache.org/mod_mbox/flink-user/202009.mbox/%3CCAGDv3o5WyJTrXs9Pg+Vy-b+LwgEE26iN54iqE0=f5t+m8vw...@mail.gmail.com%3E
>>>>
>>>> https://www.talkend.net/post/75078.html
>>>> Based on this I changed the log4j.properties.
>>>> Let me know what you think. Please also let me know if you need some
>>>> specific logs. Appreciate your help.
>>>>
>>>> Best,
>>>> Diwakar
>>>>
>>>> On Tue, Oct 27, 2020 at 12:26 AM Robert Metzger <rmetz...@apache.org>
>>>> wrote:
>>>>
>>>>> Hey Diwakar,
>>>>>
>>>>> how are you deploying Flink on EMR? Are you using YARN?
>>>>> If so, you could also use log aggregation to see all the logs at once
>>>>> (from both JobManager and TaskManagers). (yarn logs -applicationId
>>>>> <Application ID>)
>>>>>
>>>>> Could you post (or upload somewhere) all logs you have of one run? It
>>>>> is much easier for us to debug something if we have the full logs (the
>>>>> logs
>>>>> show for example the classpath that you are using, we would see how you
>>>>> are
>>>>> deploying Flink, etc.)
>>>>>
>>>>> From the information available, my guess is that you have modified
>>>>> your deployment in some way (use of a custom logging version, custom
>>>>> deployment method, version mixup with jars from both Flink 1.8 and 1.11,
>>>>> ...).
>>>>>
>>>>> Best,
>>>>> Robert
>>>>>
>>>>>
>>>>> On Tue, Oct 27, 2020 at 12:41 AM Diwakar Jha <diwakar.n...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> This is what I see on the WebUI.
>>>>>>
>>>>>> 23:19:24.263 [flink-akka.actor.default-dispatcher-1865] ERROR
>>>>>> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
>>>>>> - Failed to transfer file from TaskExecutor
>>>>>> container_1603649952937_0002_01_000004.
>>>>>> java.util.concurrent.CompletionException:
>>>>>> org.apache.flink.util.FlinkException: The file LOG does not exist on the
>>>>>> TaskExecutor. at
>>>>>> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$25(
>>>>>> TaskExecutor.java:1742 <http://taskexecutor.java:1742/>)
>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0] at
>>>>>> java.util.concurrent.CompletableFuture$AsyncSupply.run
>>>>>> <http://java.util.concurrent.completablefuture$asyncsupply.run/>(
>>>>>> CompletableFuture.java:1604 <http://completablefuture.java:1604/>)
>>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1149 <http://threadpoolexecutor.java:1149/>)
>>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run
>>>>>> <http://java.util.concurrent.threadpoolexecutor$worker.run/>(
>>>>>> ThreadPoolExecutor.java:624 <http://threadpoolexecutor.java:624/>)
>>>>>> ~[?:1.8.0_252] at java.lang.Thread.run <http://java.lang.thread.run/>
>>>>>> (Thread.java:748 <http://thread.java:748/>) ~[?:1.8.0_252] Caused
>>>>>> by: org.apache.flink.util.FlinkException: The file LOG does not exist on
>>>>>> the TaskExecutor. ... 5 more 23:19:24.275
>>>>>> [flink-akka.actor.default-dispatcher-1865] ERROR
>>>>>> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
>>>>>> - Unhandled exception. org.apache.flink.util.FlinkException: The file LOG
>>>>>> does not exist on the TaskExecutor. at
>>>>>> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$25(
>>>>>> TaskExecutor.java:1742 <http://taskexecutor.java:1742/>)
>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0] at
>>>>>> java.util.concurrent.CompletableFuture$AsyncSupply.run
>>>>>> <http://java.util.concurrent.completablefuture$asyncsupply.run/>(
>>>>>> CompletableFuture.java:1604 <http://completablefuture.java:1604/>)
>>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1149 <http://threadpoolexecutor.java:1149/>)
>>>>>> ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run
>>>>>> <http://java.util.concurrent.threadpoolexecutor$worker.run/>(
>>>>>> ThreadPoolExecutor.java:624 <http://threadpoolexecutor.java:624/>)
>>>>>> ~[?:1.8.0_252] at java.lang.Thread.run <http://java.lang.thread.run/>
>>>>>> (Thread.java:748 <http://thread.java:748/>) ~[?:1.8.0_252]
>>>>>>
>>>>>> Appreciate if anyone has any pointer for this.
>>>>>>
>>>>>> On Mon, Oct 26, 2020 at 10:45 AM Chesnay Schepler <ches...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Flink 1.11 uses slf4j 1.7.15; the easiest way to check the log files
>>>>>>> is usually via the WebUI.
>>>>>>>
>>>>>>> On 10/26/2020 5:30 PM, Diwakar Jha wrote:
>>>>>>>
>>>>>>> I think my problem is with Sl4j library. I'm using sl4j 1.7 with
>>>>>>> Flink 1.11. If that's correct then i appreciate if someone can point me
>>>>>>> to
>>>>>>> the exact Slf4j library that i should use with Flink 1.11
>>>>>>>
>>>>>>> Flink = 1.11.x;
>>>>>>> Slf4j = 1.7;
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Oct 25, 2020 at 8:00 PM Diwakar Jha <diwakar.n...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for checking my configurations. Could you also point me
>>>>>>>> where I can see the log files? Just to give more details. I'm trying to
>>>>>>>> access these logs in AWS cloudwatch.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Diwakar
>>>>>>>>
>>>>>>>> On Sun, Oct 25, 2020 at 2:16 PM Chesnay Schepler <
>>>>>>>> ches...@apache.org> wrote:
>>>>>>>>
>>>>>>>>> With Flink 1.11 reporters were refactored to plugins, and are now
>>>>>>>>> accessible by default (so you no longer have to bother with copying
>>>>>>>>> jars
>>>>>>>>> around).
>>>>>>>>>
>>>>>>>>> Your configuration appears to be correct, so I suggest to take a
>>>>>>>>> look at the log files.
>>>>>>>>>
>>>>>>>>> On 10/25/2020 9:52 PM, Diwakar Jha wrote:
>>>>>>>>>
>>>>>>>>> Hello Everyone,
>>>>>>>>>
>>>>>>>>> I'm new to flink and i'm trying to upgrade from flink 1.8 to flink
>>>>>>>>> 1.11 on an emr cluster. after upgrading to flink1.11 One of the
>>>>>>>>> differences
>>>>>>>>> that i see is i don't get any metrics. I found out that flink 1.11
>>>>>>>>> does not
>>>>>>>>> have *org.apache.flink.metrics.statsd.StatsDReporterFactory* jar
>>>>>>>>> in /usr/lib/flink/opt which was the case for flink 1.8. Could anyone
>>>>>>>>> have
>>>>>>>>> any pointer to locate
>>>>>>>>> *org.apache.flink.metrics.statsd.StatsDReporterFactory* jar or
>>>>>>>>> how to use metrics in flink.1.11?
>>>>>>>>>
>>>>>>>>> Things i tried :
>>>>>>>>> a) the below setup
>>>>>>>>>
>>>>>>>>> metrics.reporters: stsdmetrics.reporter.stsd.factory.class:
>>>>>>>>> org.apache.flink.metrics.statsd.StatsDReporterFactorymetrics.reporter.stsd.host:
>>>>>>>>> localhostmetrics.reporter.stsd.port: 8125
>>>>>>>>>
>>>>>>>>> b) I tried downloading the statsd jar from
>>>>>>>>> https://mvnrepository.com/artifact/org.apache.flink/flink-metrics-statsd
>>>>>>>>> putting it inside plugins/statsd directory.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best,
>>>>>>>>> Diwakar Jha.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best,
>>>>>>>> Diwakar Jha.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best,
>>>>>>> Diwakar Jha.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best,
>>>>>> Diwakar Jha.
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Best,
>>>> Diwakar Jha.
>>>>
>>>
>>
>> --
>> Best,
>> Diwakar Jha.
>>
>
