Re: Fencing token exceptions from Job Manager High Availability mode

2019-10-11 Thread Till Rohrmann
aybe 5% of the time. >>> >>> >>> >>> We’re running Flink 1.7.1. This issue only happens when we run in Job >>> Manager High Availability mode. We provision two Job Managers, a 3-node >>> zookeeper cluster, task managers and our monitor all in their

Re: Fencing token exceptions from Job Manager High Availability mode

2019-10-10 Thread Joshua Fan
We provision two Job Managers, a 3-node >> zookeeper cluster, task managers and our monitor all in their own >> Kubernetes namespace. I can send you Zookeeper logs too if that would be >> helpful. >> >> >> >> Thanks in advance for any help you c

Re: Fencing token exceptions from Job Manager High Availability mode

2019-10-10 Thread Joshua Fan
, October 2, 2019 at 6:10 AM > *To: *Fabian Hueske > *Cc: *"Hanson, Bruce" , "user@flink.apache.org" < > user@flink.apache.org> > *Subject: *Re: Fencing token exceptions from Job Manager High > Availability mode > > > > Hi Bruce, are you able to pro

Re: Fencing token exceptions from Job Manager High Availability mode

2019-10-02 Thread Till Rohrmann
Hi Bruce, are you able to provide us with the full debug logs? From the excerpt itself it is hard to tell what is going on. Cheers, Till On Wed, Oct 2, 2019 at 2:24 PM Fabian Hueske wrote: > Hi Bruce, > > I haven't seen such an exception yet, but maybe Till (in CC) can help. > > Best, > Fabian

Re: Fencing token exceptions from Job Manager High Availability mode

2019-10-02 Thread Fabian Hueske
Hi Bruce, I haven't seen such an exception yet, but maybe Till (in CC) can help. Best, Fabian Am Di., 1. Okt. 2019 um 05:51 Uhr schrieb Hanson, Bruce < bruce.han...@here.com>: > Hi all, > > > > We are running some of our Flink jobs with Job Manager High Availability. > Occasionally we get a clu

Fencing token exceptions from Job Manager High Availability mode

2019-09-30 Thread Hanson, Bruce
Hi all, We are running some of our Flink jobs with Job Manager High Availability. Occasionally we get a cluster that comes up improperly and doesn’t respond. Attempts to submit the job seem to hang and when we hit the /overview REST endpoint in the Job Manager we get a 500 error and a fencing t