Hello All,
I'm running the WordCount example streaming job and it fails because of
loss of Taskmanagers.
When gone through the logs of the taskmanager it has the following messages
15:14:26,592 INFO org.apache.flink.streaming.runtime.tasks.StreamTask
- State backend is set to heap memory (checkpoint to jobmanager)
15:14:26,703 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Unnamed (15/25) switched to RUNNING
15:14:26,709 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Unnamed (17/25) switched to RUNNING
15:14:26,712 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Unnamed (16/25) switched to RUNNING
15:14:26,714 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Unnamed (18/25) switched to RUNNING
15:27:29,356 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Unnamed (15/25) switched to FAILED with
exception.
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager '
vm-10-155-208-135.cloud.mwn.de/10.155.208.135:37028'. This might indicate
that the remote task manager was lost.
at
org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelInactive(PartitionRequestClientHandler.java:119)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:306)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
at
io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
15:27:29,401 INFO org.apache.flink.runtime.taskmanager.TaskManager
- TaskManager akka://flink/user/taskmanager disconnects from
JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager: JobManager
is no longer reachable
15:27:29,384 WARN akka.remote.RemoteWatcher
- Detected unreachable: [akka.tcp://flink@10.155.208.156:6123]
15:27:29,356 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Unnamed (18/25) switched to FAILED with
exception.
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager '
vm-10-155-208-135.cloud.mwn.de/10.155.208.135:37028'. This might indicate
that the remote task manager was lost.
at
org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelInactive(PartitionRequestClientHandler.java:119)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:306)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
at
io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
15:27:29,356 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Unnamed (16/25) switched to FAILED with
exception.
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager '
vm-10-155-208-135.cloud.mwn.de/10.155.208.135:37028'. This might indicate
that the remote task manager was lost.
at
org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelInactive(PartitionRequestClientHandler.java:119)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:306)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
at
io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
15:27:29,356 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Unnamed (17/25) switched to FAILED with
exception.
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager '
vm-10-155-208-135.cloud.mwn.de/10.155.208.135:37028'. This might indicate
that the remote task manager was lost.
at
org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelInactive(PartitionRequestClientHandler.java:119)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:306)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
at
io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
15:27:29,518 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Cancelling all computations and discarding all cached data.
15:27:29,525 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to fail task externally Keyed Aggregation -> Sink:
Unnamed (17/25)
15:27:29,525 INFO org.apache.flink.runtime.taskmanager.Task
- Task Keyed Aggregation -> Sink: Unnamed (17/25) is already in state
FAILED
15:27:29,525 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to fail task externally Source: Read Text File Source ->
Flat Map (20/25)
15:27:29,525 INFO org.apache.flink.runtime.taskmanager.Task
- Source: Read Text File Source -> Flat Map (20/25) switched to
FAILED with exception.
java.lang.Exception: TaskManager akka://flink/user/taskmanager disconnects
from JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager:
JobManager is no longer reachable
at
org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:826)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:297)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:119)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at
akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
at akka.actor.ActorCell.invoke(ActorCell.scala:486)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15:27:29,526 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Keyed Aggregation -> Sink: Unnamed
(15/25)
15:27:29,525 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Keyed Aggregation -> Sink: Unnamed
(17/25)
15:27:29,526 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Keyed Aggregation -> Sink: Unnamed
(16/25)
15:27:29,527 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Keyed Aggregation -> Sink: Unnamed
(18/25)
15:27:29,528 INFO org.apache.flink.runtime.taskmanager.Task
- Triggering cancellation of task code Source: Read Text File Source
-> Flat Map (20/25) (9cd65cbf15aafe0241b9d59e48b79f52).
15:27:29,531 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to fail task externally Source: Read Text File Source ->
Flat Map (6/25)
15:27:29,531 INFO org.apache.flink.runtime.taskmanager.Task
- Source: Read Text File Source -> Flat Map (6/25) switched to FAILED
with exception.
java.lang.Exception: TaskManager akka://flink/user/taskmanager disconnects
from JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager:
JobManager is no longer reachable
at
org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:826)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:297)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:119)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at
akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
at akka.actor.ActorCell.invoke(ActorCell.scala:486)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15:27:29,532 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Source: Read Text File Source -> Flat
Map (20/25)
15:27:29,535 INFO org.apache.flink.runtime.taskmanager.Task
- Triggering cancellation of task code Source: Read Text File Source
-> Flat Map (6/25) (2151d07383014506c5f1b0283b1986de).
15:27:29,535 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to fail task externally Keyed Aggregation -> Sink:
Unnamed (15/25)
15:27:29,535 INFO org.apache.flink.runtime.taskmanager.Task
- Task Keyed Aggregation -> Sink: Unnamed (15/25) is already in state
FAILED
15:27:29,535 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to fail task externally Source: Read Text File Source ->
Flat Map (13/25)
15:27:29,535 INFO org.apache.flink.runtime.taskmanager.Task
- Source: Read Text File Source -> Flat Map (13/25) switched to
FAILED with exception.
java.lang.Exception: TaskManager akka://flink/user/taskmanager disconnects
from JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager:
JobManager is no longer reachable
at
org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:826)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:297)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:119)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at
akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
at akka.actor.ActorCell.invoke(ActorCell.scala:486)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15:27:29,536 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Source: Read Text File Source -> Flat
Map (6/25)
15:27:29,538 INFO org.apache.flink.runtime.taskmanager.Task
- Triggering cancellation of task code Source: Read Text File Source
-> Flat Map (13/25) (a6a0f05e0362dbf0505c1439893e53e4).
15:27:29,539 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to fail task externally Keyed Aggregation -> Sink:
Unnamed (16/25)
15:27:29,539 INFO org.apache.flink.runtime.taskmanager.Task
- Task Keyed Aggregation -> Sink: Unnamed (16/25) is already in state
FAILED
15:27:29,539 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to fail task externally Source: Read Text File Source ->
Flat Map (24/25)
15:27:29,539 INFO org.apache.flink.runtime.taskmanager.Task
- Source: Read Text File Source -> Flat Map (24/25) switched to
FAILED with exception.
java.lang.Exception: TaskManager akka://flink/user/taskmanager disconnects
from JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager:
JobManager is no longer reachable
at
org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:826)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:297)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:119)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at
akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
at akka.actor.ActorCell.invoke(ActorCell.scala:486)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15:27:29,541 INFO org.apache.flink.runtime.taskmanager.Task
- Triggering cancellation of task code Source: Read Text File Source
-> Flat Map (24/25) (96df0f3c0990c5a573c67a31a6207fa2).
15:27:29,541 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to fail task externally Keyed Aggregation -> Sink:
Unnamed (18/25)
15:27:29,541 INFO org.apache.flink.runtime.taskmanager.Task
- Task Keyed Aggregation -> Sink: Unnamed (18/25) is already in state
FAILED
15:27:29,542 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Source: Read Text File Source -> Flat
Map (24/25)
15:27:29,543 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Source: Read Text File Source -> Flat
Map (13/25)
15:27:29,551 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Disassociating from JobManager
15:27:29,572 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has a UID that has been quarantined. Association aborted.
15:27:29,582 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:27:29,587 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:27:29,623 INFO org.apache.flink.runtime.io.network.netty.NettyClient
- Successful shutdown (took 17 ms).
15:27:29,625 INFO org.apache.flink.runtime.io.network.netty.NettyServer
- Successful shutdown (took 2 ms).
15:27:29,640 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500
milliseconds)
15:27:30,157 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 2, timeout: 1000
milliseconds)
15:27:31,177 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 3, timeout: 2000
milliseconds)
15:27:33,193 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 4, timeout: 4000
milliseconds)
15:27:37,203 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 5, timeout: 8000
milliseconds)
15:27:37,218 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:27:45,215 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 6, timeout: 16000
milliseconds)
15:27:45,235 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:28:01,223 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 7, timeout: 30 seconds)
15:28:01,237 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:28:31,243 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 8, timeout: 30 seconds)
15:28:31,261 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:28:45,960 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:29:01,253 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 9, timeout: 30 seconds)
15:29:01,273 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:30:11,545 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 10, timeout: 30 seconds)
15:30:11,562 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
15:30:25,958 WARN Remoting
- Tried to associate with unreachable remote address [akka.tcp://
flink@10.155.208.156:6123]. Address is now gated for 5000 ms, all messages
to this address will be delivered to dead letters. Reason: The remote
system has quarantined this system. No further associations to the remote
system are possible until this system is restarted.
Could anyone explain what is the reason for the systems getting
disassociated and be quarantined?
Kind Regards,
Ravinder Kaur
