Re: Flink AskTimeoutException killing the jobs

2020-07-06 Thread M Singh
Thanks Xintong.  I will check the logs.   On Sunday, July 5, 2020, 09:29:31 PM EDT, Xintong Song wrote: As I already mentioned, I would suggest to look into the jobmanager logs and gc logs, see if there's any problem that prevent the process from handling the rpc messages timely. Th

Re: Flink AskTimeoutException killing the jobs

2020-07-05 Thread Xintong Song
As I already mentioned, > I would suggest to look into the jobmanager logs and gc logs, see if > there's any problem that prevent the process from handling the rpc messages > timely. > The Akka ask timeout does not seem to be the root problem to me. Thank you~ Xintong Song On Sat, Jul 4, 202

Re: Flink AskTimeoutException killing the jobs

2020-07-03 Thread M Singh
Hi Xintong/LakeShen: We have the following setting in flink-conf.yaml akka.ask.timeout: 180 s akka.tcp.timeout: 180 s But still see this exception.  Are there multiple akka.ask.timeout or additional settings required ? Thanks Mans On Friday, July 3, 2020, 01:08:05 AM EDT, Xintong Song

Re: Flink AskTimeoutException killing the jobs

2020-07-02 Thread Xintong Song
The configuration option you're looking for is `akka.ask.timeout`. However, I'm not sure increasing this configuration would help in your case. The error message shows that there is a timeout on a local message. It is wired a local message does not get replied within 10 sec. I would suggest to lo

Flink AskTimeoutException killing the jobs

2020-07-02 Thread M Singh
Hi: I am using Flink 1.10 on AWS EMR cluster. We are getting AskTimeoutExceptions which is causing the flink jobs to die.    Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/resourcemanager#-1602864959]] after [1 ms]. Message of type [org.apache.flink.run