Thanks again, Peter for sharing your logs. I looked into the issue with the help of Chesnay. Essentially, it's FLINK-27354 [1] that is causing this issue. We couldn't come up with a reason why it should have popped up just now with 1.15. The bug itself is already present in 1.14. You can find more details on the investigation in FLINK-27354 [1] itself.
Best, Matthias [1] https://issues.apache.org/jira/browse/FLINK-27354 On Mon, Apr 25, 2022 at 2:00 PM Matthias Pohl <matth...@ververica.com> wrote: > Thanks Peter, we're looking into it... > > On Mon, Apr 25, 2022 at 11:54 AM Peter Schrott <pe...@bluerootlabs.io> > wrote: > >> Hi, >> >> sorry for the late reply. It took me quite some time to get the logs out >> of the system. I have attached them now. >> >> Its logs of 2 jobmanagers and 2 taskamangers. It can be seen on jm 1 that >> the job starts crashing and recovering a few times. This happens >> until 2022-04-20 12:12:14,607. After that the above described behavior can >> be seen. >> >> I hope this helps. >> >> Best, Peter >> >> On Fri, Apr 22, 2022 at 12:06 PM Matthias Pohl <matth...@ververica.com> >> wrote: >> >>> FYI: I created FLINK-27354 [1] to cover the issue of retrying to connect >>> to the RM while shutting down the JobMaster. >>> >>> This doesn't explain your issue though, Peter. It's still unclear why >>> the JobMaster is still around as stated in my previous email. >>> >>> Matthias >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-27354 >>> >>> On Fri, Apr 22, 2022 at 11:54 AM Matthias Pohl <matth...@ververica.com> >>> wrote: >>> >>>> Just by looking through the code, it appears that these logs could be >>>> produced while stopping the job. The ResourceManager sends a confirmation >>>> of the JobMaster being disconnected at the end back to the JobMaster. If >>>> the JobMaster is still around to process the request, it would try to >>>> reconnect (I'd consider that a bug because the JobMaster is in shutdown >>>> mode already and wouldn't need to re-establish the connection). This method >>>> would have been swallowed otherwise if the JobMaster was already >>>> terminated. >>>> >>>> The only explanation I can come up with right now (without having any >>>> logs) is that stopping the JobMaster didn't finish for some reason. For >>>> that it would be helpful to look at the logs to see whether there is some >>>> other issue that causes the JobMaster to stop entirely. >>>> >>>> On Fri, Apr 22, 2022 at 10:14 AM Matthias Pohl <matth...@ververica.com> >>>> wrote: >>>> >>>>> ...if possible it would be good to get debug rather than only info >>>>> logs. Did you encounter anything odd in the TaskManager logs as well. >>>>> Sharing those might be of value as well. >>>>> >>>>> On Fri, Apr 22, 2022 at 8:57 AM Matthias Pohl <matth...@ververica.com> >>>>> wrote: >>>>> >>>>>> Hi Peter, >>>>>> thanks for sharing. That doesn't sound right. May you provide the >>>>>> entire jobmanager logs? >>>>>> >>>>>> Best, >>>>>> Matthias >>>>>> >>>>>> On Thu, Apr 21, 2022 at 6:08 PM Peter Schrott <pe...@bluerootlabs.io> >>>>>> wrote: >>>>>> >>>>>>> Hi Flink-Users, >>>>>>> >>>>>>> I am not sure if this does something to my cluster or not. But since >>>>>>> updating to Flink 1.15 (atm rc4) I find the following logs: >>>>>>> >>>>>>> INFO: Registering job manager ab7db9ff0ebd26b3b89c3e2e56684762 >>>>>>> @akka.tcp:// >>>>>>> fl...@flink-jobmanager-xxx.com:40015/user/rpc/jobmanager_2 for job >>>>>>> 5566648d9b1aac6c1a1b78187fd56975. >>>>>>> >>>>>>> as many times as number of parallelisms (here 10 times). These logs >>>>>>> are triggered every 5 minutes. >>>>>>> >>>>>>> Then they are followed by: >>>>>>> >>>>>>> INFO: Registration of job manager ab7db9ff0ebd26b3b89c3e2e56684762 >>>>>>> @akka.tcp:// >>>>>>> fl...@flink-jobmanager-xxx.com:40015/user/rpc/jobmanager_2 failed. >>>>>>> >>>>>>> also 10 log entries. >>>>>>> >>>>>>> I followed the lifetime of the job (5566648d9b1aac6c1a1b78187fd56975), >>>>>>> it was a long-running sql streaming job, started on Apr 13th on a >>>>>>> standalone cluster. After some recovery attempts it finally failed >>>>>>> (using >>>>>>> the failover strategy) on the 20th Apr (yesterday) for good. Then those >>>>>>> logs started to appear. Now there was no other job running on my cluster >>>>>>> anymore but the logs appeared every 5 minutes until I restarted this >>>>>>> jobmanager service. >>>>>>> >>>>>>> This job was just an example, it happens to other jobs too. >>>>>>> >>>>>>> It's just INFO logs but it does not look healthy either. >>>>>>> >>>>>>> Thanks & Best >>>>>>> Peter >>>>>>> >>>>>> -- Matthias Pohl | Engineer Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton Wehner
