Thanks Gary. Attached is the jobmanager log. You are correct that this is
running on YARN. I changed web.timeout as you suggested - that seems to be
working the few times I tested it. This problem comes and goes though -
sometimes it starts before it times out. I'll keep the web.timeout setting
and reply again if the problem comes up again. Thanks again for your quick
response!
On Fri, Aug 31, 2018 at 1:38 PM Gary Yao <g...@data-artisans.com> wrote:
> Hi Greg,
>
> Can you describe the steps to reproduce the problem, or can you attach the
> full jobmanager logs? Because JobExecutionResultHandler appears in your
> log, I
> assume that you are starting a job cluster on YARN. Without seeing the
> complete logs, I cannot be sure what exactly happens. For now, you can try
> setting the config option web.timeout to a higher value.
>
> Best,
> Gary
>
> On Fri, Aug 31, 2018 at 8:01 PM, Greg Finch <finchgreg...@gmail.com>
> wrote:
>
>> I'm having a problem with akka timeout when starting my cluster. The
>> error is "Ask timed out after 10000 ms.". I have changed the
>> akka.ask.timeout config setting to be 300000 ms, but it still times out and
>> fails after 10 seconds. I confirmed that the config is properly set by
>> both checking the Job Manager configuration tab (it shows 300000 ms) as
>> well logging the output of AkkaUtils.getTimeout(configuration) which also
>> shows 300000ms. It seems something is not honoring that configuration
>> value.
>>
>> I did find a different thread that discussed the fact that the
>> LocalStreamEnvironment will not honor this setting, but that is not my
>> case. I am running on a cluster (AWS EMR) using the regular
>> StreamExecutionEnvironment. This is Flink 1.5.2.
>>
>> Any ideas?
>>
>> ~~~~~
>>
>> 2018-08-31 17:37:55 INFO
>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token
>> for : ip-10-213-139-66.ec2.internal:8041
>> 2018-08-31 17:37:55 INFO
>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token
>> for : ip-10-213-136-25.ec2.internal:8041
>> 2018-08-31 17:38:34 ERROR
>> o.a.flink.runtime.rest.handler.job.JobExecutionResultHandler -
>> Implementation error: Unhandled exception.
>> akka.pattern.AskTimeoutException: Ask timed out on
>> [Actor[akka://flink/user/dispatcher#-219618710]] after [10000 ms].
>> Sender[null] sent message of type
>> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
>> at
>> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
>> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
>> at
>> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
>> at
>> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>> at
>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
>> at
>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
>> at java.lang.Thread.run(Thread.java:748)
>> 2018-08-31 17:38:41 INFO
>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Waiting for
>> application to be successfully unregistered.
>> 2018-08-31 17:38:41 INFO
>> o.a.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - Interrupted
>> while waiting for queue
>> java.lang.InterruptedException: null
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:323)
>> 2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
>> flink-akka.remote.default-remote-dispatcher-81 - Association with remote
>> system [akka.tcp://flink@ip-10-213-142-102.ec2.internal:42027] has failed,
>> address is now gated for [50] ms. Reason: [Disassociated]
>>
>>
>>
>
17:37:37,563 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
resource [file:logback.xml] at [file:logback.xml]
17:37:37,673 |-INFO in ch.qos.logback.core.joran.action.DefinePropertyAction -
About to instantiate property definer of type
[transform.utilities.logback.EnvironmentVariablePropertyDefiner]
17:37:37,681 |-INFO in ch.qos.logback.core.joran.action.DefinePropertyAction -
Popping property definer for property named [region] from the object stack
17:37:37,692 |-INFO in ch.qos.logback.core.joran.action.DefinePropertyAction -
About to instantiate property definer of type
[transform.utilities.logback.EnvironmentVariablePropertyDefiner]
17:37:37,693 |-INFO in ch.qos.logback.core.joran.action.DefinePropertyAction -
Popping property definer for property named [environment] from the object stack
17:37:37,693 |-INFO in
ch.qos.logback.core.joran.util.ConfigurationWatchListUtil@1e67a849 - Adding
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1535731101345_0004/filecache/21/packet-generator-async-assembly-1.0.jar!/logback.elasticsearch-urls.xml]
to configuration watch list.
17:37:37,693 |-INFO in
ch.qos.logback.core.joran.spi.ConfigurationWatchList@57d5872c - URL
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1535731101345_0004/filecache/21/packet-generator-async-assembly-1.0.jar!/logback.elasticsearch-urls.xml]
is not of type file
17:37:37,794 |-INFO in
ch.qos.logback.core.joran.util.ConfigurationWatchListUtil@1e67a849 - Adding
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1535731101345_0004/filecache/21/packet-generator-async-assembly-1.0.jar!/logback.aws.xml]
to configuration watch list.
17:37:37,794 |-INFO in
ch.qos.logback.core.joran.spi.ConfigurationWatchList@57d5872c - URL
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1535731101345_0004/filecache/21/packet-generator-async-assembly-1.0.jar!/logback.aws.xml]
is not of type file
17:37:37,797 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type
[com.internetitem.logback.elasticsearch.ElasticsearchAppender]
17:37:37,799 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [us01.prod.elasticsearch]
17:37:37,803 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.ElasticsearchProperties] for
[properties] property
17:37:37,803 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
17:37:37,804 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
17:37:37,805 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
17:37:37,814 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
17:37:37,814 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
17:37:37,815 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
17:37:37,815 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
17:37:37,815 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.Property] for [property] property
17:37:37,818 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.HttpRequestHeaders] for
[headers] property
17:37:37,819 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type
[com.internetitem.logback.elasticsearch.config.HttpRequestHeader] for [header]
property
17:37:37,846 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type
[ch.qos.logback.core.rolling.RollingFileAppender]
17:37:37,848 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [file]
17:37:37,853 |-INFO in
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@4c39bec8 - Will use zip
compression
17:37:37,856 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder]
for [encoder] property
17:37:37,858 |-WARN in ch.qos.logback.core.rolling.RollingFileAppender[file] -
Append mode is mandatory for RollingFileAppender. Defaulting to append=true.
17:37:37,858 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[file] -
Active log file name: logs/application.log
17:37:37,858 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[file] -
File property is set to [logs/application.log]
17:37:37,859 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
17:37:37,860 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [console]
17:37:37,860 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder]
for [encoder] property
17:37:37,861 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction -
Setting level of ROOT logger to INFO
17:37:37,861 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction -
Attaching appender named [file] to Logger[ROOT]
17:37:37,861 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction -
Attaching appender named [console] to Logger[ROOT]
17:37:37,864 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction -
Attaching appender named [us01.prod.elasticsearch] to Logger[ROOT]
17:37:37,865 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [ch.qos.logback] to WARN
17:37:37,865 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [org.apache.flink] to WARN
17:37:37,865 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem] to
WARN
17:37:37,865 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- End of configuration.
17:37:37,866 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@d21a74c -
Registering current configuration as safe fallback point
2018-08-31 17:37:38 WARN org.apache.flink.configuration.Configuration -
Config uses deprecated configuration key 'web.port' instead of proper key
'rest.port'
2018-08-31 17:37:38 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-08-31 17:37:38 WARN
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory
/tmp/flink-web-5888ee19-cbd7-43cd-931d-ce254672b9c6/flink-web-upload does not
exist, or has been deleted externally. Previously uploaded files are no longer
available.
2018-08-31 17:37:39 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting
to ResourceManager at ip-10-213-140-28.ec2.internal/10.213.140.28:8030
2018-08-31 17:37:39 INFO
o.a.h.yarn.client.api.impl.ContainerManagementProtocolProxy -
yarn.client.max-cached-nodemanagers-proxies : 0
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-189.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-96.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-119.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-113.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-14.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-165.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-23.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-126.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-58.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-81.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-236.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-148.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-83.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-165.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-11.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-41.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-176.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-161.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-10.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-177.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-66.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-254.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-143.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-186.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-172.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-182.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-123.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-124.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-5.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-217.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-16.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-75.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-59.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-253.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-238.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-16.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-91.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-42.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-66.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-234.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-198.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-42.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-214.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-214.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-131.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-221.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-102.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-254.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-51.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-94.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-141.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-176.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-202.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-107.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-45.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-82.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-127.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-139.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-32.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-137.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-181.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-157.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-245.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-141-125.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-175.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-149.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-3.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-5.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-102.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-189.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-246.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-2.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-223.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-145.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-201.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-252.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-52.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-31.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-120.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-137-18.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-107.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-36.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-221.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-208.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-58.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-213.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-79.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-99.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-143-188.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-67.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-186.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-219.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-140-201.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-142-173.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-112.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-138-174.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-5.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-139-66.ec2.internal:8041
2018-08-31 17:37:55 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Received new token for : ip-10-213-136-25.ec2.internal:8041
2018-08-31 17:38:34 ERROR
o.a.flink.runtime.rest.handler.job.JobExecutionResultHandler - Implementation
error: Unhandled exception.
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher#-219618710]] after [10000 ms]. Sender[null]
sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
at java.lang.Thread.run(Thread.java:748)
2018-08-31 17:38:41 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
- Waiting for application to be successfully unregistered.
2018-08-31 17:38:41 INFO
o.a.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - Interrupted while
waiting for queue
java.lang.InterruptedException: null
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:323)
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-81 - Association with remote system
[akka.tcp://flink@ip-10-213-142-102.ec2.internal:42027] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-88 - Association with remote system
[akka.tcp://flink@ip-10-213-137-165.ec2.internal:45025] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-83 - Association with remote system
[akka.tcp://flink@ip-10-213-141-123.ec2.internal:42745] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-83 - Association with remote system
[akka.tcp://flink@ip-10-213-142-217.ec2.internal:37293] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-79 - Association with remote system
[akka.tcp://flink@ip-10-213-138-42.ec2.internal:44033] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-79 - Association with remote system
[akka.tcp://flink@ip-10-213-138-120.ec2.internal:42235] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-13 - Association with remote system
[akka.tcp://flink@ip-10-213-143-188.ec2.internal:33547] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-88 - Association with remote system
[akka.tcp://flink@ip-10-213-138-174.ec2.internal:37121] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-79 - Association with remote system
[akka.tcp://flink@ip-10-213-139-127.ec2.internal:42495] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-79 - Association with remote system
[akka.tcp://flink@ip-10-213-141-16.ec2.internal:39039] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-80 - Association with remote system
[akka.tcp://flink@ip-10-213-143-157.ec2.internal:40071] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-86 - Association with remote system
[akka.tcp://flink@ip-10-213-136-223.ec2.internal:33399] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
2018-08-31 17:38:42 WARN akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-13 - Association with remote system
[akka.tcp://flink@ip-10-213-143-45.ec2.internal:36059] has failed, address is
now gated for [50] ms. Reason: [Disassociated]
