liting liu created FLINK-40384:
----------------------------------
Summary: Periodic FlinkStateSnapshot cleanup causes repeated
SavepointError events
Key: FLINK-40384
URL: https://issues.apache.org/jira/browse/FLINK-40384
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: liting liu
{{When periodic FlinkStateSnapshot creation is enabled, active snapshot
resources may be deleted while their status is still being updated.}}
{{The operator repeatedly reports errors similar to:
\{code}
io.fabric8.kubernetes.client.KubernetesClientException:
Failure executing PATCH at:
.../namespaces/<namespace>/flinkstatesnapshots/<snapshot-name>
Operation cannot be fulfilled on
flinkstatesnapshots.flink.apache.org "<snapshot-name>":
the object has been modified; please apply your changes to the latest version
and try again
HTTP status: 409 Conflict
\{code}
Each conflict produces a SavepointError Kubernetes Event.
The affected FlinkStateSnapshot CR is normally deleted shortly afterward, but
its Event remains until the Kubernetes Event TTL expires. Therefore, the
namespace may contain many SavepointError Events referring to
FlinkStateSnapshot resources that no longer exist.}}
{{h3. Effect}}
{{* Every periodic savepoint cycle can generate additional unnecessary
SavepointError Events.
* A large number of failure Events accumulate over time, even though only a few
FlinkStateSnapshot CRs currently exist.
* Many Events refer to already-deleted resources, making the reported failure
impossible to inspect afterward.
* Operator logs and Kubernetes Events are continuously polluted by 409 Conflict
errors.
* Genuine savepoint failures become difficult to distinguish from these
deletion and status-update conflicts.
* Monitoring and alerting systems may report continuous savepoint failures even
when the errors are caused by the operator's internal resource lifecycle
handling.
* Periodic savepoint attempts may be deleted before they complete.}}
{{h3. How to reproduce}}
# {{Deploy Flink Kubernetes Operator 1.10 or later with the FlinkStateSnapshot
CRD enabled. }}
# {{Create a running FlinkDeployment.}}
# {{Configure periodic savepoints and retain only one snapshot:
\{code}
kubernetes.operator.periodic.savepoint.interval: 10s
kubernetes.operator.savepoint.history.max.count: "1"
kubernetes.operator.savepoint.dispose-on-delete: "true"
\{code}}}
# {{Wait for the first periodic FlinkStateSnapshot to reach COMPLETED.}}
# {{Wait for the next periodic FlinkStateSnapshot to be created.}}
# {{{}{}}}{{{}Inspect the Kubernetes Events:
\{code}
kubectl get events -n <namespace> \
--field-selector reason=SavepointError \
--sort-by=.metadata.creationTimestamp
\{code}{}}}
# {{Continue observing several periodic savepoint cycles.
The operator repeatedly produces PATCH 409 Conflict errors and SavepointError
Events. After the corresponding FlinkStateSnapshot CRs are deleted, their
failure Events remain visible in the namespace.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
