As I already mentioned, > I would suggest to look into the jobmanager logs and gc logs, see if > there's any problem that prevent the process from handling the rpc messages > timely. >
The Akka ask timeout does not seem to be the root problem to me. Thank you~ Xintong Song On Sat, Jul 4, 2020 at 12:12 AM M Singh <mans2si...@yahoo.com> wrote: > Hi Xintong/LakeShen: > > We have the following setting in flink-conf.yaml > > akka.ask.timeout: 180 s > > akka.tcp.timeout: 180 s > > > But still see this exception. Are there multiple akka.ask.timeout or > additional settings required ? > > > Thanks > > Mans > > On Friday, July 3, 2020, 01:08:05 AM EDT, Xintong Song < > tonysong...@gmail.com> wrote: > > > The configuration option you're looking for is `akka.ask.timeout`. > > > However, I'm not sure increasing this configuration would help in your > case. The error message shows that there is a timeout on a local message. > It is wired a local message does not get replied within 10 sec. I would > suggest to look into the jobmanager logs and gc logs, see if there's any > problem that prevent the process from handling the rpc messages timely. > > > Thank you~ > > Xintong Song > > > > On Fri, Jul 3, 2020 at 3:51 AM M Singh <mans2si...@yahoo.com> wrote: > > Hi: > > I am using Flink 1.10 on AWS EMR cluster. > > We are getting AskTimeoutExceptions which is causing the flink jobs to > die. > > Caused by: akka.pattern.AskTimeoutException: Ask timed out on > [Actor[akka://flink/user/resourcemanager#-1602864959]] after [10000 ms]. > Message of type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. > A typical reason for `AskTimeoutException` is that the recipient actor > didn't send a reply. > at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635) > at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635) > at > akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648) > ... 9 more > > > Can you please let me know where can I set the timeout for this timeout ? > > I could not find this specific timeout in the flink doc - Apache Flink > 1.10 Documentation: Configuration > <https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html>. > > > Thanks > > Mans > >
