Well ... that didn't take long. The next time I tried, I got the Akka
timeout again. Attached are the logs from the last attempt. They're very
similar to the other logs I sent.
On Fri, Aug 31, 2018 at 2:04 PM Greg Finch <finchgreg...@gmail.com> wrote:
> Thanks Gary. Attached is the jobmanager log. You are correct that this
> is running on YARN. I changed web.timeout as you suggested - that seems to
> be working the few times I tested it. This problem comes and goes though -
> sometimes it starts before it times out. I'll keep the web.timeout setting
> and reply again if the problem comes up again. Thanks again for your quick
> response!
>
> On Fri, Aug 31, 2018 at 1:38 PM Gary Yao <g...@data-artisans.com> wrote:
>
>> Hi Greg,
>>
>> Can you describe the steps to reproduce the problem, or can you attach the
>> full jobmanager logs? Because JobExecutionResultHandler appears in your
>> log, I
>> assume that you are starting a job cluster on YARN. Without seeing the
>> complete logs, I cannot be sure what exactly happens. For now, you can try
>> setting the config option web.timeout to a higher value.
>>
>> Best,
>> Gary
>>
>> On Fri, Aug 31, 2018 at 8:01 PM, Greg Finch <finchgreg...@gmail.com>
>> wrote:
>>
>>> I'm having a problem with akka timeout when starting my cluster. The
>>> error is "Ask timed out after 10000 ms.". I have changed the
>>> akka.ask.timeout config setting to be 300000 ms, but it still times out and
>>> fails after 10 seconds. I confirmed that the config is properly set by
>>> both checking the Job Manager configuration tab (it shows 300000 ms) as
>>> well logging the output of AkkaUtils.getTimeout(configuration) which also
>>> shows 300000ms. It seems something is not honoring that configuration
>>> value.
>>>
>>> I did find a different thread that discussed the fact that the
>>> LocalStreamEnvironment will not honor this setting, but that is not my
>>> case. I am running on a cluster (AWS EMR) using the regular
>>> StreamExecutionEnvironment. This is Flink 1.5.2.
>>>
>>> Any ideas?
>>>
>>> ~~~~~
>>>
>>> 2018-08-31 17:37:55 INFO
>>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token
>>> for : ip-10-213-139-66.ec2.internal:8041
>>> 2018-08-31 17:37:55 INFO
>>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token
>>> for : ip-10-213-136-25.ec2.internal:8041
>>> 2018-08-31 17:38:34 ERROR
>>> o.a.flink.runtime.rest.handler.job.JobExecutionResultHandler -
>>> Implementation error: Unhandled exception.
>>> akka.pattern.AskTimeoutException: Ask timed out on
>>> [Actor[akka://flink/user/dispatcher#-219618710]] after [10000 ms].
>>> Sender[null] sent message of type
>>> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
>>> at
>>> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
>>> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
>>> at
>>> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
>>> at
>>> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>>> at
>>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
>>> at
>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
>>> at
>>> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
>>> at
>>> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
>>> at
>>> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
>>> at java.lang.Thread.run(Thread.java:748)
>>> 2018-08-31 17:38:41 INFO
>>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Waiting for
>>> application to be successfully unregistered.
>>> 2018-08-31 17:38:41 INFO
>>> o.a.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - Interrupted
>>> while waiting for queue
>>> java.lang.InterruptedException: null
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> at
>>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:323)
>>> 2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
>>> flink-akka.remote.default-remote-dispatcher-81 - Association with remote
>>> system [akka.tcp://flink@ip-10-213-142-102.ec2.internal:42027] has failed,
>>> address is now gated for [50] ms. Reason: [Disassociated]
>>>
>>>
>>>
>>
19:14:31,176 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
resource [file:logback.xml] at [file:logback.xml]
19:14:31,288 |-INFO in ch.qos.logback.core.joran.action.DefinePropertyAction -
About to instantiate property definer of type
[transform.utilities.logback.EnvironmentVariablePropertyDefiner]
19:14:31,296 |-INFO in ch.qos.logback.core.joran.action.DefinePropertyAction -
Popping property definer for property named [region] from the object stack
19:14:31,308 |-INFO in ch.qos.logback.core.joran.action.DefinePropertyAction -
About to instantiate property definer of type
[transform.utilities.logback.EnvironmentVariablePropertyDefiner]
19:14:31,309 |-INFO in ch.qos.logback.core.joran.action.DefinePropertyAction -
Popping property definer for property named [environment] from the object stack
19:14:31,309 |-INFO in
ch.qos.logback.core.joran.util.ConfigurationWatchListUtil@1e67a849 - Adding
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1535731101345_0009/filecache/21/packet-generator-async-assembly-1.0.jar!/logback.elasticsearch-urls.xml]
to configuration watch list.
19:14:31,309 |-INFO in
ch.qos.logback.core.joran.spi.ConfigurationWatchList@57d5872c - URL
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1535731101345_0009/filecache/21/packet-generator-async-assembly-1.0.jar!/logback.elasticsearch-urls.xml]
is not of type file
19:14:31,413 |-INFO in
ch.qos.logback.core.joran.util.ConfigurationWatchListUtil@1e67a849 - Adding
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1535731101345_0009/filecache/21/packet-generator-async-assembly-1.0.jar!/logback.aws.xml]
to configuration watch list.
19:14:31,413 |-INFO in
ch.qos.logback.core.joran.spi.ConfigurationWatchList@57d5872c - URL
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1535731101345_0009/filecache/21/packet-generator-async-assembly-1.0.jar!/logback.aws.xml]
is not of type file
19:14:31,415 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type
[com.internetitem.logback.elasticsearch.ElasticsearchAppender]
19:14:31,418 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [us01.prod.elasticsearch]
19:14:31,421 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.ElasticsearchProperties] for
[properties] property
19:14:31,422 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
19:14:31,422 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
19:14:31,423 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
19:14:31,427 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
19:14:31,428 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
19:14:31,428 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
19:14:31,429 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
19:14:31,429 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
19:14:31,432 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.HttpRequestHeaders] for
[headers] property
19:14:31,432 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.HttpRequestHeader] for [header]
property
19:14:31,459 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type
[ch.qos.logback.core.rolling.RollingFileAppender]
19:14:31,462 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [file]
19:14:31,466 |-INFO in
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@4c39bec8 - Will use zip
compression
19:14:31,470 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder]
for [encoder] property
19:14:31,472 |-WARN in ch.qos.logback.core.rolling.RollingFileAppender[file] -
Append mode is mandatory for RollingFileAppender. Defaulting to append=true.
19:14:31,472 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[file] -
Active log file name: logs/application.log
19:14:31,472 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[file] -
File property is set to [logs/application.log]
19:14:31,473 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
19:14:31,474 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [console]
19:14:31,474 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder]
for [encoder] property
19:14:31,475 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction -
Setting level of ROOT logger to INFO
19:14:31,475 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction -
Attaching appender named [file] to Logger[ROOT]
19:14:31,475 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction -
Attaching appender named [console] to Logger[ROOT]
19:14:31,478 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction -
Attaching appender named [us01.prod.elasticsearch] to Logger[ROOT]
19:14:31,479 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [ch.qos.logback] to WARN
19:14:31,479 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [org.apache.flink] to WARN
19:14:31,479 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem] to
WARN
19:14:31,479 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- End of configuration.
19:14:31,480 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@d21a74c -
Registering current configuration as safe fallback point
2018-08-31 19:14:31 WARN org.apache.flink.configuration.Configuration -
Config uses deprecated configuration key 'web.port' instead of proper key
'rest.port'
2018-08-31 19:14:32 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-08-31 19:14:32 WARN org.apache.flink.configuration.Configuration -
Configuration cannot evaluate value 300s as a long integer number
2018-08-31 19:14:32 WARN org.apache.flink.configuration.Configuration -
Configuration cannot evaluate value 300s as a long integer number
2018-08-31 19:14:32 WARN
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory
/tmp/flink-web-f57e66c2-0fe4-4d5c-8bd9-214997a4b831/flink-web-upload does not
exist, or has been deleted externally. Previously uploaded files are no longer
available.
2018-08-31 19:14:33 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting
to ResourceManager at ip-10-213-140-28.ec2.internal/10.213.140.28:8030
2018-08-31 19:14:33 INFO
o.a.h.yarn.client.api.impl.ContainerManagementProtocolProxy -
yarn.client.max-cached-nodemanagers-proxies : 0
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-31.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-225.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-137.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-221.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-254.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-102.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-172.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-66.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-125.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-177.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-107.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-42.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-198.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-208.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-124.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-42.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-186.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-5.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-81.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-5.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-245.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-221.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-223.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-175.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-66.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-213.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-25.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-186.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-36.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-5.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-139.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-145.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-165.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-253.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-182.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-254.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-99.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-236.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-11.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-202.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-51.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-83.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-201.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-189.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-126.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-201.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-58.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-214.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-14.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-127.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-188.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-96.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-2.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-141.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-157.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-149.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-3.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-67.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-10.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-75.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-91.ec2.internal:8041
2018-08-31 19:14:48 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-173.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-131.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-52.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-217.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-161.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-112.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-58.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-102.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-32.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-107.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-234.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-120.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-176.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-214.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-82.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-79.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-165.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-246.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-174.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-113.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-123.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-59.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-219.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-189.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-66.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-238.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-119.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-45.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-181.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-176.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-16.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-143.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-148.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-252.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-18.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-94.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-41.ec2.internal:8041
2018-08-31 19:14:49 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-16.ec2.internal:8041
2018-08-31 19:15:32 ERROR
o.a.flink.runtime.rest.handler.job.JobExecutionResultHandler - Implementation
error: Unhandled exception.
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher#-1157618104]] after [10000 ms].
Sender[null] sent message of type
"org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
at java.lang.Thread.run(Thread.java:748)
2018-08-31 19:15:33 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Waiting for application to be successfully unregistered.
2018-08-31 19:15:33 INFO
o.a.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - Interrupted while
waiting for queue
java.lang.InterruptedException: null
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:323)
