Hi everyone, Flink 2.3 introduces the application as a first-class concept above jobs, with its own ID, state, and durable result store. The Kubernetes Operator is job-centric and has no representation of it. That produces one immediate bug and one longer-term gap. I'd like directional input on the first, and to open a discussion on the second.
1. Deployments wedge on Flink 2.3 (FLINK-40467) On Flink 2.3 a deployment can come up healthy but never submit its job. The JobManager runs, REST responds, and the operator reports success -- nothing in the logs says otherwise. When an application terminates, Flink persists a terminal ApplicationResult keyed by Application ID, under a path derived from the HA cluster id. For the operator both are the CR name, which never changes across redeployments, so the next deployment finds the old record and Flink declines to re-run it. It is intermittent: Flink deletes that record once cleanup commits, and it only survives when the operator tears down the JobManager while cleanup is still running. There is an open PR for this already by contributors in the community: https://github.com/apache/flink-kubernetes-operator/pull/1199 -- a unique `application-result-store.storage-path` per deployment. Same shape as the operator's existing `setRandomJobResultStorePath` (FLINK-27569). We arrived at the same fix independently in our fork and have verified it on 2.3. The tradeoff is a deliberate resource leak: every deployment leaves a directory behind that nothing ever cleans up, and users are expected to prune them by hand. That is the same deal the job result store already makes, but this doubles it. *Question:* Is this an approach the community can accept? 2. The operator has no model of the application layer Beyond this bug, the operator cannot observe or reason about applications at all. There is no Application ID in the CRD or status, and no use of the `/applications` REST endpoints. Consequences we have run into: - Application-level failures surface as "job not found", or not at all. - Multi-job applications are unavailable. Under HA, 2.3 supports multiple batch jobs per application, but the operator sets `submit-failed-job-on-application-error=true`, which caps the job count at 1. If it is worth supporting, the operator would need to model the application at some point, and we are unsure how far that should go. Some directions, roughly in order of how much they change: (a) Observation only -- surface application state and its jobs in status, emit events. No behavioural change, and no CRD change beyond status. (b) The operator acts on what it observes -- application state drives lifecycle decisions rather than a single job's state, so an application that is still starting, one that failed before submitting a job, and one that was never submitted at all become distinguishable instead of all surfacing as "job not found". (c) Fully application-centric -- lift the single-job assumption in the reconciler and the status model, which is what multi-batch actually requires. (a) and (b) seem useful on their own even if multi-job never happens, but we may be missing reasons not to. (c) is where we assume the real design questions are -- what a per-job status looks like, how snapshots and upgrades are addressed with more than one job, and what happens to `status.jobStatus`. So: is multi-job something the operator wants to support, and if so does that ordering seem reasonable? And is anyone already looking at this? We would love to hear from the community. Thanks, James
