I've added a description of how the launcher could be implemented to the same Google Doc at https://docs.google.com/document/d/1dtGjPYcsBkx1vxHPs1QnDtPxeH_Acz_pl8gx4b1BLB4/edit?usp=sharing
Having looked into it more, it'll be lot smaller and simpler to implement than I first thought. Kind regards D -- dalelane.co.uk On Thursday, 27 August 2026 at 19:42, Robert Metzger <[email protected]> wrote: > I think the launcher can go into this FLIP. > > > On Tue, Aug 25, 2026 at 8:44 PM Dale Lane <[email protected]> > wrote: > > > > 3. Launcher placement - Here I'd go the opposite way of the current doc. > > > Currently, every deployment shape keeps its bootstrap in the Flink > > > distribution, and that convention is what decouples operator version from > > > Flink version. My opinion is that the MiniCluster launcher should follow > > > it. In practice, the launcher is close to `flink run -t local`, only > > adding > > > HA-aware resubmission and pod exit-code semantics on top, so it is core > > > code either way. I'd also say it makes sense to ship it in the > > distribution > > > (ideally with a bin/ script, so the same setup runs locally too) because > > > this means the operator depends on no MiniCluster Java API at all, and it > > > can manage the deployment in the same unified way as the other shapes. > > > > I really like this idea, thanks - that's much better than what I'd > > originally thought of. It simplifies a few things, such as migration > > between MiniCluster and Deployment by letting you just use the same image > > for both as-is. I'll have a proper think through what it would need and > > update the doc. > > > > Do you think a Flink launcher would need a separate/dedicated FLIP to get > > broader scrutiny? > > > > Kind regards > > > > D > > -- > > dalelane.co.uk > > > > On Tuesday, 25 August 2026 at 14:10, Dennis-Mircea Ciupitu < > > [email protected]> wrote: > > > > > Hi Dale, > > > > > > My response to the open points is the following: > > > > > > 1. MiniCluster vs 2 containers in one pod - Robert's numbers and > > > argument make it clear to me. So, in order to ensure the lowest > > possible > > > resource consumption, we need to go with the MiniCluster approach and > > not > > > rely on something intermediary that does not reach this end goal and > > is > > > only better suited from the implementation/design perspective. > > > 2. Separate CRD vs a third mode - Because the MiniCluster approach is > > > the way to go, I'd say that we should have an implementation that > > > integrates with it in the best way possible. For me, I see a separate > > CRD > > > as the best way to proceed here, as MiniCluster is different from many > > > points of view from a normal Flink cluster, and by going with a third > > mode > > > I think we just complicate ourselves and we introduce too much blast > > radius. > > > 3. Launcher placement - Here I'd go the opposite way of the current > > doc. > > > Currently, every deployment shape keeps its bootstrap in the Flink > > > distribution, and that convention is what decouples operator version > > from > > > Flink version. My opinion is that the MiniCluster launcher should > > follow > > > it. In practice, the launcher is close to `flink run -t local`, only > > adding > > > HA-aware resubmission and pod exit-code semantics on top, so it is > > core > > > code either way. I'd also say it makes sense to ship it in the > > distribution > > > (ideally with a bin/ script, so the same setup runs locally too) > > because > > > this means the operator depends on no MiniCluster Java API at all, > > and it > > > can manage the deployment in the same unified way as the other shapes. > > > 4. Promoting the MiniCluster - Here I say if the entrypoint lands in > > > Flink core, then I'd follow Gyula's and Robert's guidance on this > > side, if > > > there is something that can be done further here in order to promote > > it > > > properly. > > > 5. Resource and memory mapping - Deferring the exact ratios to > > > implementation is fine with me. > > > > > > > > > Best regards, > > > Dennis > > > > > > > > > On Mon, Aug 24, 2026 at 2:00 PM Dale Lane < > > [email protected]> > > > wrote: > > > > > > > > Whether it should be a different CRD I think should > > > > > depend on the overlap with the FlinkDeployment. If > > > > > we are exposing essentially the same spec (podTemplate, > > > > > jobSpec, flinkConfig, ingress..) then it feels like we > > > > > should reuse the CRD and simply introduce a new > > > > > deployment mode next to native and standalone. > > > > > > > > @Gyula, @Dennis - How would you like to proceed with this question? > > > > > > > > I can absolutely see arguments for/against both approaches. In the end > > I > > > > leaned more towards a separate CRD, partly from an API purity > > perspective > > > > (feeling that the separate topology deserved a separate API, and avoids > > > > describing the pod spec and status as "jobManager" or "taskManager" > > when > > > > it's both) and partly because it dodged the automated topology > > migration. > > > > > > > > But maybe we don't mind the "a MiniCluster includes a Job Manager so we > > > > can just describe it as jobManager (and ignore the taskManager > > references)" > > > > API smell. It could change the manual migration path I describe to > > > > transition jobs between MiniCluster and full distributed clusters to > > just > > > > flipping a mode value. And we can always reject/block such a mode > > switch in > > > > the meantime unless/until the Operator is ready to support topology > > > > transition/migration. > > > > > > > > Kind regards > > > > > > > > Dale > > > > -- > > > > dalelane.co.uk > > > > > > > > Sent with Proton Mail secure email. > > > > > > > > On Thursday, 20 August 2026 at 10:47, Gyula Fóra <[email protected] > > > > > > > wrote: > > > > > > > > > Thanks @Robert Metzger for the explanation, > > > > > > > > > > Overall I think this is a good idea and hopefully we can reuse most > > of > > > > > the controller flow that we already have. > > > > > It would be nice to share the spec/status as much as reasonably > > possible. > > > > > > > > > > Whether it should be a different CRD I think should depend on the > > > > > overlap with the FlinkDeployment. If we are exposing essentially the > > > > > same spec (podTemplate, jobSpec, flinkConfig, ingress..) then it > > feels > > > > > like we should reuse the CRD and simply introduce a new deployment > > > > > mode next to native and standalone. > > > > > > > > > > Cheers > > > > > Gyula > > > > > > > > > > On Thu, Aug 20, 2026 at 11:26 AM Dale Lane > > > > > <[email protected]> wrote: > > > > > > > > > > > > This bit that I added to the FLIP is perhaps worthy of a bit more > > > > scrutiny, as it's based on my assumptions (misunderstandings?) of how > > the > > > > FKO project works. > > > > > > > > > > > > In the section "Promoting MiniCluster to a stable API" I asserted: > > > > > > > > > > > > > For the Flink Kubernetes Operator project to provide a launcher > > > > > > > as an example project would depend on the core Flink project > > > > > > > promoting some MiniCluster components to @PublicEvolving ... > > > > > > > This is not a technical requirement, as the current > > > > > > > proof-of-concept demonstrates that the existing API is sufficient > > > > > > > - the promotion recommended here is of MiniCluster’s existing > > > > > > > submission surface. The requirement is about the Operator project > > > > > > > managing risk by building upon a stable contract. ... > > > > > > > > > > > > Is that really a requirement or have I just invented that? :) > > > > > > > > > > > > Looking more closely at the existing code, I can see we already > > have > > > > plenty of uses of flink-runtime classes that don't have @Public / > > > > @PublicEvolving annotations, so my usage of flink-runtime classes like > > the > > > > MiniCluster and MiniClusterConfiguration wouldn't be without precedent. > > > > > > > > > > > > While I still think it'd be lovely to build this feature solely on > > > > stable APIs, maybe I'm creating an unreasonably high bar to clear by > > > > framing it this so strongly. > > > > > > > > > > > > What do you think? What are the norms here? > > > > > > > > > > > > Kind regards > > > > > > > > > > > > Dale > > > > > > -- > > > > > > dalelane.co.uk > > > > > > > > > > > > > > > > > > On Wednesday, 5 August 2026 at 16:02, Dale Lane < > > > > [email protected]> wrote: > > > > > > > > > > > > > I'd like to start a discussion on > > > > > > > FLIP-XXX : Running Flink jobs in MiniCluster using the Kubernetes > > > > Operator > > > > > > > > > > > > > https://docs.google.com/document/d/1dtGjPYcsBkx1vxHPs1QnDtPxeH_Acz_pl8gx4b1BLB4/edit?usp=sharing > > > > > > > > > > > > > > The aim of the FLIP is to extend the Flink Kubernetes Operator to > > > > offer a single-pod, light-weight deployment option for low-throughput > > jobs. > > > > > > > > > > > > > > From the motivation: > > > > > > > A single-pod, self-contained Flink job that starts fast and > > needs no > > > > multi-pod coordination could be a good fit for low-throughput jobs that > > > > aren't suitable for session clusters because they need isolation. > > > > > > > > > > > > > > Looking forward to feedback, both on the general motivation (Have > > > > you seen a need for very small lightweight Flink jobs where fast > > > > crash-consistent resume is good enough without a full distributed Flink > > > > cluster?) and the suggested implementation approach (Do you think a new > > > > custom resource kind is the best way to represent this capability?) > > > > > > > > > > > > > > Kind regards > > > > > > > > > > > > > > Dale > > > > > > > -- > > > > > > > dalelane.co.uk > > > > > > > > > > > > > > > > > > > > > >
