Re: Warnings connecting to Akka

2019-10-18 Thread John Smith
Oh that's fine. I was just wondering why it happened. It seems to have gone away since the reboot. On Fri, 18 Oct 2019 at 10:43, Till Rohrmann wrote: > Hi John, > > the reason why you are seeing these warnings is because Akka tries to > re-establish the connection to a lost endpoint (here a dead

Re: Warnings connecting to Akka

2019-10-18 Thread Till Rohrmann
Hi John, the reason why you are seeing these warnings is because Akka tries to re-establish the connection to a lost endpoint (here a dead TaskExecutor). This should continue until the connection is either quarantined or if the underlying ActorRef to the remote endpoint has been garbage collected.

Re: Warnings connecting to Akka

2019-10-10 Thread John Smith
Ok so it seems there was some sort of network issue. Then leader election. But it seems it had some old state and kept trying to connect to the same task machine over and over...? 2019-09-19 22:26:14,841 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Unable to read addi

Re: Warnings connecting to Akka

2019-10-09 Thread Timothy Victor
We see a very similar (if not the same) error running version 1.9 on Kubernetes. So far what we have discovered is that a taskmanager gets killed and a new one is created, but JM still thinks it needs to connect to the old (now dead TM). I was even able to see the a taskmanager on the same host

Re: Warnings connecting to Akka

2019-10-09 Thread John Smith
Sorry been away on leave. I'll check ASAP. On Thu, 3 Oct 2019 at 20:52, Zili Chen wrote: > Does the log you attached above come from a TaskManager Node? If so, > what state is the Job node it tried to connect to? Did it crash? > > BTW, it would be helpful if you can attach more logs of TM and JM

Re: Warnings connecting to Akka

2019-10-03 Thread Zili Chen
Does the log you attached above come from a TaskManager Node? If so, what state is the Job node it tried to connect to? Did it crash? BTW, it would be helpful if you can attach more logs of TM and JM except two lines said akka connection refused. John Smith 于2019年10月4日周五 上午2:08写道: > So I guess

Re: Warnings connecting to Akka

2019-10-03 Thread John Smith
So I guess it had some older state? On Thu., Oct. 3, 2019, 11:29 a.m. John Smith, wrote: > I'm running standalone cluster with Zookeeper. It seems it was trying to > connect to an older node. I rebooted the Job node tha was complaining. It > seems to be ok now... > > I have 3 Zookeepers, 3 Job N

Re: Warnings connecting to Akka

2019-10-03 Thread John Smith
I'm running standalone cluster with Zookeeper. It seems it was trying to connect to an older node. I rebooted the Job node tha was complaining. It seems to be ok now... I have 3 Zookeepers, 3 Job Nodes and 3 Tasks Nodes On Thu, 3 Oct 2019 at 11:15, Zili Chen wrote: > Hi John, > > could you prov

Re: Warnings connecting to Akka

2019-10-03 Thread Zili Chen
Hi John, could you provide some details such as which mode you runs on(standalone/YARN) and related configuration(jobmanager.address jobmanager.port and so on)? Best, tison. John Smith 于2019年10月3日周四 下午11:02写道: > Hi running 1.8 the cluster seems to be OK but I see these warnings in the > logs.

Warnings connecting to Akka

2019-10-03 Thread John Smith
Hi running 1.8 the cluster seems to be OK but I see these warnings in the logs... 2019-10-03 14:57:25,152 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /xxx.xxx.xxx.65:46167 2019-10-03 14:5