Re: akka.ask.timeout setting not honored

2018-09-04 Thread Greg Finch
Hi Gary, Turns out, the configuration warning you mentioned was the key. The akka.ask.timeout requires a duration unit, but the web.timeout setting is looking for a long. So the change I made earlier would not have applied since it couldn't read `300s`. Since making that change (`web.timeout: 3

Re: akka.ask.timeout setting not honored

2018-08-31 Thread Gary Yao
Hi Greg, Unfortunately the environment information [1] is not logged. Can you set the log level for all Flink packages to DEBUG? Do you install Flink yourself on EMR, or do you use the pre-installed one? Can you show us the command with which you start the cluster/submit the job? I do not know i

Re: akka.ask.timeout setting not honored

2018-08-31 Thread Greg Finch
Well ... that didn't take long. The next time I tried, I got the Akka timeout again. Attached are the logs from the last attempt. They're very similar to the other logs I sent. On Fri, Aug 31, 2018 at 2:04 PM Greg Finch wrote: > Thanks Gary. Attached is the jobmanager log. You are correct t

Re: akka.ask.timeout setting not honored

2018-08-31 Thread Greg Finch
Thanks Gary. Attached is the jobmanager log. You are correct that this is running on YARN. I changed web.timeout as you suggested - that seems to be working the few times I tested it. This problem comes and goes though - sometimes it starts before it times out. I'll keep the web.timeout settin

Re: akka.ask.timeout setting not honored

2018-08-31 Thread Gary Yao
Hi Greg, Can you describe the steps to reproduce the problem, or can you attach the full jobmanager logs? Because JobExecutionResultHandler appears in your log, I assume that you are starting a job cluster on YARN. Without seeing the complete logs, I cannot be sure what exactly happens. For now, y

akka.ask.timeout setting not honored

2018-08-31 Thread Greg Finch
I'm having a problem with akka timeout when starting my cluster. The error is "Ask timed out after 1 ms.". I have changed the akka.ask.timeout config setting to be 30 ms, but it still times out and fails after 10 seconds. I confirmed that the config is properly set by both checking the J