Hi Flink Community,
Most of the Flink applications run on 1.14 at my company. After upgrading
the Flink Operator to 1.6, we've seen many jobmanager pods show
"JobManagerDeploymentStatus: MISSING".
Here are some logs from the operator pod on one of our Flink applications:
[m [33m2023-10-18 02:02:40,823 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][nemo/nemo-streaming-users-identi-updates] >>> Event | Warning |
SAVEPOINTERROR | Savepoint failed for savepointTriggerNonce: null
...
[m [33m2023-10-18 02:02:40,883 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][nemo/nemo-streaming-users-identi-updates] >>> Event | Warning |
CLUSTERDEPLOYMENTEXCEPTION | Status have been modified externally in
version 17447422864 Previous: <redacted>
...
[m [33m2023-10-18 02:02:40,919 [m [36mi.j.o.p.e.ReconciliationDispatcher [m
[1;31m[ERROR][nemo/nemo-streaming-users-identi-updates] Error during event
processing ExecutionScope{ resource id:
ResourceID{name='nemo-streaming-users-identi-updates', namespace='nemo'},
version: 17447420285} failed.
...
org.apache.flink.kubernetes.operator.exception.ReconciliationException:
org.apache.flink.kubernetes.operator.exception.StatusConflictException:
Status have been modified externally in version 17447422864 Previous:
<redacted>
...
[m [33m2023-10-18 02:03:03,273 [m [36mo.a.f.k.o.o.d.ApplicationObserver [m
[1;31m[ERROR][nemo/nemo-streaming-users-identi-updates] Missing JobManager
deployment
...
[m [33m2023-10-18 02:03:03,295 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][nemo/nemo-streaming-users-identi-updates] >>> Event | Warning | MISSING |
Missing JobManager deployment
[m [33m2023-10-18 02:03:03,295 [m [36mo.a.f.c.Configuration [m [33m[WARN
][nemo/nemo-streaming-users-identi-updates] Config uses deprecated
configuration key 'high-availability' instead of proper key
'high-availability.type'
This seems related to this email thread:
https://www.mail-archive.com/user@flink.apache.org/msg51439.html. However,
I believe that we're not seeing the HA metadata getting deleted.
What could cause the JobManagerDeploymentStatus to be MISSING?
Thanks,
Tony
--
<http://www.robinhood.com/>
Tony Chen
Software Engineer
Menlo Park, CA
Don't copy, share, or use this email without permission. If you received it
by accident, please let us know and then delete it right away.
