Hello,

I've assembled a standalone cluster of 3 task managers and 3 job
managers(and 3 ZK) following the instructions at

https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/cluster_setup.html
and
https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/jobmanager_high_availability.html

It works ok, but randomly, task managers becomes unavailable. JobManager
has exception like below in logs:


[2018-03-19 00:33:10,211] WARN Association with remote system [akka.tcp://
fl...@qafdsflinkw811.nn.five9lab.com:42413] has failed, address is now
gated for [5000] ms. Reason: [Association failed with [akka.tcp://
fl...@qafdsflinkw811.nn.five9lab.com:42413]] Caused by: [Connection
refused: qafdsflinkw811.nn.five9lab.com/10.5.61.124:42413]
(akka.remote.ReliableDeliverySupervisor)
[2018-03-21 00:30:35,975] ERROR Association to [akka.tcp://
fl...@qafdsflinkw811.nn.five9lab.com:35320] with UID [1500204560]
irrecoverably failed. Quarantining address. (akka.remote.Remoting)
java.util.concurrent.TimeoutException: Remote system has been silent for
too long. (more than 48.0 hours)
        at
akka.remote.ReliableDeliverySupervisor$$anonfun$idle$1.applyOrElse(Endpoint.scala:375)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
        at
akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I can't find a reason for this exception, any ideas?

Thank you,
Alex

Reply via email to