Hi Ryan / Daniel / Gyula, I've finished the last round of asks, and PR #1043 is now out of draft and ready for review: https://github.com/apache/flink-kubernetes-operator/pull/1043. I rewrote the description, so it should be a quicker read, with a suggested reading order at the top.
Highlights: 1. Gate-clear timeout. A new kubernetes.operator.bluegreen.gate.timeout bounds the wait for the gate, and the transition aborts past it. Every abort hands the gate back to the deployment that keeps running. 2. SQL / Table API support. The gate reads each record's timestamp from exactly one configured strategy. field-index reads a BIGINT, TIMESTAMP(p) or TIMESTAMP_LTZ(p) column of RowData by its type, for SQL and Table API jobs. The gate uses record time only, with no wall-clock fallback, and late records are no longer dropped after the hand-over. 3. No-dup / no-loss e2e test. The new e2e-tests/test_bluegreen_advanced.sh runs a transition of a purpose-built job and checks that every record is written exactly once, late records included. It's wired into the CI matrix next to the existing Blue/Green tests, which I fixed and enabled as well. All 32 jobs pass on my fork across Flink 1.20, 2.0, 2.1 and 2.2, natively and standalone: https://github.com/schongloo/flink-kubernetes-operator/actions/runs/37846378915. The abort-after-cutover scenario isn't covered end to end yet; it's a TODO in the script. 4. Gate chaining and parallelism. The gate joins its neighbour's chain, keeps the edge it replaces, and copies the neighbour's parallelism and max parallelism. So it adds no vertex that could be rescaled on its own. 5. Docs. There's a new Advanced Transition Mode section in the Blue/Green Deployments page: how the hand-over works and what it guarantees, teardown and aborts, requirements, the extraction strategies, gate placement, SQL jobs and configuration. The related pages are updated to match. On your exactly-once sinks point: the old deployment is now torn down only once every one of its gate subtasks has handed over, and it's stopped with a savepoint first, so sinks that commit on checkpoints commit what it wrote. Running the e2e tests also surfaced a few fixes, all in the PR. The main one: only the standby sets the cutover point now, which closes a race that could duplicate records. I also documented a Flink 2.x limitation: jobs using the async state API can't take canonical savepoints, so they need the native format for transitions. The operator now explains that failure and rejects state backend changes that a native savepoint can't restore. The known limitations are listed in the PR description. Could you continue your evaluation when you get a chance? Happy to walk through any part of it. Thanks, > On Aug 18, 2026, at 11:13 AM, Sergio Chong Loo <[email protected]> wrote: > > Thanks for the feedback Ryan, I’ll re-focus on this soon and will keep you > posted. > > FYI: @Gyula > > - Sergio > >> On Aug 10, 2026, at 10:12 AM, Ryan van Huuksloot via dev >> <[email protected]> wrote: >> >> Hi Sergio, >> >> Thanks for driving this. We are still interested but are currently caught >> up in other work, which explains the delay. >> >> Re: What we should include. >> >> The 4 points are good. >> Additionally, your deferral is unclear. Are SQL/Table pipelines supported >> if they use watermarks? That should be included. >> I also think that having an e2e harness might need to be part of the >> success criteria. If we have one test we may as well build it into the >> harness. >> >> I agree with what is out of scope for Phase 2. >> >> The last thing I would point out is that the documentation will be really >> important for this feature. Your callouts are great and I would do +100 to >> what you have for documentation. >> >> Thanks >> >> Ryan van Huuksloot >> Staff Engineer, Infrastructure | Streaming Platform >> [image: Shopify] >> <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> >> >> >> On Mon, Jul 6, 2026 at 12:50 PM Sergio Chong Loo <[email protected]> >> wrote: >> >>> Hi Daniel / Ryan / Gyula, >>> >>> Overdue update, and apologies for the long gap since May. A no-code >>> watermark option jumped >>> the queue (a few teams were blocked without it), so I pushed that >>> through first. Quick >>> summary of what landed, then a proposal to lock the remaining scope so >>> we share a finish >>> line. I know it’s a bit lengthy but please bear with me. >>> >>> Landed since May: >>> - No-code watermark extraction: bluegreen.gate.watermark.field-path. >>> Instead of a custom >>> extractor-class, point the gate at a dot-notation field path: >>> bluegreen.gate.strategy: "WATERMARK" >>> bluegreen.gate.watermark.field-path: "eventTime" # or nested: >>> "metadata.timestamp" >>> It reads an epoch-ms long off the record (public field, class >>> hierarchy, or getXxx() >>> getter), cached with no per-record reflection, and works out of the >>> box with protobuf >>> classes. extractor-class stays for anything a field path cannot >>> express. Scope note, since >>> it is close to your WatermarkGenerator point but not the same: it >>> removes extractor code >>> for DataStream/POJO jobs, but it still reads an existing field and >>> does not cover the >>> SQL/Table case you raised (that stays open, see Defer). >>> - transitionMode: you were right that the unset case was a real gap (it >>> errored instead of >>> defaulting), so it now defaults to BASIC (existing basic deployments >>> upgrade cleanly) and >>> is immutable after the initial deployment (a change is rejected with a >>> validation error >>> and a Warning event). >>> - Rebased onto current operator main, and the gate-strategy layer is now >>> extensible >>> (watermark is one strategy). >>> >>> Proposed finish line: Phase 2 to be marked done when advanced Blue/Green >>> is correct >>> within a documented envelope, never hangs silently, and its >>> no-dup/no-loss claim is proven >>> by at least one real end-to-end test. Anything that widens the envelope >>> is a follow-up. >>> >>> Must-have: >>> 1. Gate-clear timeout. If a transition's gate never clears (watermark >>> stalls), the >>> controller currently waits indefinitely. Add a deadline so it fails >>> with a clear error >>> instead of hanging. >>> 2. One real no-dup/no-loss end-to-end test for the supported shape: >>> single source, >>> idempotent or transactional sink, happy-path cutover plus a failure >>> injected around the >>> cutover. This turns the headline claim from asserted to proven, and I >>> am happy to help >>> wire it into the e2e suite. Daniel, if your sandbox pipeline is >>> close, this is exactly it. >>> 3. Documented envelope and limitations: one meaningful watermark at the >>> gate point; sink >>> must be idempotent or transactional for end-to-end exactly-once; the >>> idleness caveat; >>> JVM-only auto-injection. >>> 4. Gate parallelism (your point). The agent already injects the gate at >>> the adjacent >>> operator's parallelism with a forward connection, so it adds no >>> shuffle and normally >>> chains into that vertex. The remaining work is to make that chaining >>> reliable (also match >>> the neighbor's max-parallelism and set the chaining strategy >>> explicitly) so the gate is >>> never a standalone vertex the autoscaler can rescale on its own, plus >>> a documented >>> pin/exclude fallback for the rare non-chainable case. That closes the >>> bottleneck-under-autoscaling concern. >>> >>> Defer (follow-ups): >>> - Full conformance matrix (every sink class by failure point by >>> idleness), beyond the one >>> core test. >>> - SQL/Table watermark-generator injection so table defs need no >>> timestamp field. This is >>> really SQL support, a separate surface from DataStream advanced BG; >>> proposing it as a >>> Phase 3 item. >>> - A broader shared e2e harness beyond the core test. >>> >>> Out of scope for Phase 2 (proposing we close these): >>> - Multiple sources with diverging watermarks, and mixed source types. >>> The gate assumes one >>> meaningful watermark; per-input readiness is a future gate strategy. >>> - Guaranteed no-loss when a gated source uses idleness. Documented >>> limitation; a future >>> per-source-readiness strategy could address it, at the cost of the >>> very stall the >>> gate-clear timeout is there to catch. >>> - Pyflink auto-injection. JVM-only this phase; the agent rewrites the >>> JVM job graph and >>> reads records as Java objects. >>> >>> Asks: >>> - Agree on the four must-haves as the finish line, and sign off on the >>> deferred and >>> out-of-scope lists. >>> - Daniel: how close is the sandbox pipeline to serving as the core test >>> (item 2 below)? And can >>> you confirm the exactly-once expectation to assert, that green's sink >>> transactions do not >>> commit until blue has taken and committed its final checkpoint? >>> >>> Thanks. This last stretch is short if we agree on the boundary. >>> >>> - Sergio >>> >>>> Hi Daniel / Ryan, >>>> >>>> Again thanks a lot for the feedback. Here are some thoughts: >>>> >>>> The following are excellent points and I indeed think we can work on >>> them immediately to incorporate them. I already have some ideas on >>> implementation but let me flesh them out more before I share them: >>>> - WatermarkGenerator >>>> - Gate Parallelism >>>> - TransitionMode default (trivial) >>>> >>>> Thanks for the Misc Testing proposal (especially with SQL), it will >>> indeed facilitate development and make sure the outcome/output is precise. >>> Let me know if/when you choose to start this effort (or any other item for >>> that matter). >>>> >>>> Exactly-once sinks. This needs explicit testing indeed. I haven't >>> verified this one yet. I believe the gate operator should ensure green's >>> sink transactions don't begin until blue has completed its final checkpoint >>> and committed. This is a good candidate for a dedicated test case before >>> declaring phase 1 stable. >>>> >>>> Non-idempotent sinks. I'm still not sure this is a problem meant to be >>> solved by Blue/Green deployments. If we think closely about it, even within >>> a single Flink pipeline, strict global ordering is only guaranteed within a >>> single subtask's partition. The moment you have parallelism > 1, different >>> subtasks process different keys/partitions independently and emit records >>> at different rates. There's no global wall-clock ordering across subtasks. >>> Flink's watermark mechanism gives you event-time ordering, not >>> arrival-order consistency across the whole topology. For pipelines where >>> strict ordering to a non-idempotent sink is a hard requirement, the right >>> pattern is a savepoint based stop/restart rather than concurrent blue/green >>> execution. >>>> >>>> On the other hand, assuming we pursue this, your proposal sounds >>> straightforward, but using state would imply the new pipeline needs to >>> buffer an unbounded amount of data while waiting for the first deployment >>> to finish, which makes memory management and state sizing very difficult. >>> The gate watermark barrier already provides a temporal ordering guarantee >>> (green doesn't advance past blue's watermark), which covers the majority of >>> idempotency-sensitive sinks. Perhaps I'm looking at this from an overly >>> simplistic perspective, is there a concrete example you can share to >>> illustrate the scenario you have in mind? We should definitely keep this >>> topic open as we make progress on the other items. >>>> >>>> I'll keep you posted on progress for the items at the top. >>>> >>>> ⁃ Sergio >>> >>>> On Jul 2, 2026, at 4:52 PM, Sergio Chong Loo <[email protected]> >>> wrote: >>>> >>>> Sorry for the late reply Vamshi, these are the right questions for a >>> zero-loss/zero-dup rollout. >>>> >>>> One thing up front, because it answers most of these: the gate works at >>> a single point in >>>> the job, right after a source or right before a sink. There it reads >>> one watermark >>>> (whatever Flink has combined at that point) and checks each record's >>> event-time once >>>> against a single cutover value. So it assumes a simple, linear spot >>> where one watermark >>>> cleanly governs the records passing through. In short: the pipeline has >>> to give the gate >>>> one event-time per record to compare against one cutover, once. Most of >>> your questions are >>>> really "what if it isn't that simple," and in this Advanced mode (Phase >>> 2) the answer is: >>>> place the gate on a stretch where it stays simple. >>>> >>>> 1. Multiple sources / diverging watermarks. No special rule: the gate >>> uses whatever >>>> watermark Flink has already combined at that point (for multiple >>> inputs, the minimum across >>>> them), plus each record's own event-time. So it effectively waits for >>> the slowest input; it >>>> doesn't track inputs individually. Jobs that need per-input logic >>> should keep the gate on a >>>> single-watermark stretch. >>>> >>>> 2. Idleness / lag. A slow-but-active source just holds the watermark >>> back, so the cutover >>>> waits. That is safe (no loss) but can stall, which is why we are adding >>> a deadline so a >>>> permanent stall fails loudly rather than hanging. Idleness is the sharp >>> edge, and a real >>>> caveat on no-loss: a source marked idle is dropped from the >>> current/minimum watermark >>>> (by design, for liveness), so the gate can cut over ahead of it. >>> Records it emits afterward carry >>>> an event-time before the cutover, and once the outgoing deployment is >>> torn down they have >>>> nowhere to land, so they are dropped. This is the same >>> completeness-versus-liveness >>>> tradeoff idleness always makes, turned into a hard boundary by the >>> cutover. Stated plainly: >>>> no-loss holds only if no source feeding the gate is idle during the >>> transition. If your >>>> sources use idleness, treat no-loss as best-effort across the cutover. >>> (A future gate >>>> strategy could prob. wait for per-source readiness, including idle >>> sources, at the cost of >>>> re-introducing that stall). >>>> >>>> 3. Mixed sources / event-time quality. Works for: event-time jobs with >>> one solid watermark >>>> and a clear per-record event-time where the gate sits. Out of scope for >>> Phase 2: weak or >>>> absent event-time, uneven event-time quality across inputs, or anything >>> that needs several >>>> watermarks combined. >>>> >>>> 4. Sink guarantees. The gate makes a clean split (the old job emits up >>> to the cutover, the >>>> new job after it), so normally each record goes out once. But the gate >>> only coordinates the >>>> cutover; it can't take back a duplicate the sink already wrote. During >>> the overlap, and >>>> especially if a job restarts and replays, true exactly-once still needs >>> an idempotent or >>>> transactional sink. So I'd treat idempotent-or-transactional sinks as a >>> requirement for the >>>> no-dup/no-loss claim. For Kafka, EOS transactions are the main >>> backstop. The gate narrows >>>> the window; the sink closes it. >>>> >>>> 5. Conformance matrix. Not yet: we unit-test the controller and the >>> gate's record-level >>>> logic, but there's no end-to-end no-dup/no-loss test grid, and that's >>> the real gap (and my >>>> next step, happy to build it with you). The grid would cross: number of >>> sources, watermark >>>> divergence, idleness, where a failure happens, and sink type, each >>> checked for duplicates >>>> and loss. Honest status: the single-source and idleness rows are the >>> near-term ones; the >>>> failover + sink-type rows (the ones that matter most for a hard >>> zero-loss/zero-dup bar) are >>>> the bigger lift and tie into the cutover-recovery work, so I wouldn't >>> call failover >>>> correctness proven yet. Send the scenarios that matter to you and I'll >>> add them. >>>> >>>> And none of the "out of scope" items are dead-ends. The watermark gate >>> is just one strategy >>>> (bluegreen.gate.strategy=WATERMARK) on an extensible gate layer. The >>> whole idea is that >>>> more complex cutover logic (multiple watermarks, per-input readiness, >>> custom rules) can be >>>> added as a new gate implementation instead of complicating this one. So >>> "out of scope for >>>> Phase 2" really means "a future strategy," and that's exactly where I'd >>> welcome help. >>>> >>>> This is the pressure-testing Phase 2 needs. Much appreciated. >>>> >>>>> Hi Sergio, >>>>> Thanks for driving FLIP-504 forward. This is great and the top most need >>>>> for Flink B/G cutovers. I am evaluating Phase 2 against our current >>>>> production use cases, where zero-loss/zero-duplication during cutover >>> is a >>>>> hard requirement. I would appreciate a few clarifications on the >>> intended >>>>> implementation semantics. >>>>> >>>>> 1. For multi-source jobs, if watermarks diverge, what is the intended >>>>> cutover rule (min-watermark, per-input readiness, or other)? >>>>> 2. How should idleness/temporary lag on one source affect transition >>>>> readiness to avoid loss? >>>>> 3. For mixed source types/event-time quality, what behavior is >>> supported >>>>> vs out of scope in Phase 2? >>>>> 4. For “no-dup/no-loss” expectations, what sink guarantees are assumed >>>>> (idempotent/transactional required or recommended)? For Kafka sinks >>>>> specifically, should EOS transactional mode be treated as the primary >>>>> backstop? >>>>> 5. Is there a minimal conformance test matrix planned (divergence, >>>>> idleness, failover, duplicate/loss verification)? Any guidance here >>> would >>>>> really help teams roll out safely with clear correctness boundaries. >>>>> >>>>> Thanks, >>>>> Vamshi >>>> >>>> >>>>> On Jun 21, 2026, at 2:57 PM, Jing-Jia Hung <[email protected]> >>> wrote: >>>>> >>>>> Hi Sergio, >>>>> >>>>> Late to the thread. The gate auto injection via the Java agent is cool. >>>>> >>>>> Most of the discussion so far has centered on data-plane correctness. >>> The >>>>> area I'd like to understand better is the control-plane side, >>> specifically >>>>> recovery when a transition doesn't complete. The FLIP notes the >>> controller >>>>> can be left in a bad state if a deployment fails mid-transition. From >>>>> reading the draft it looks like the teardown step waits on the gate >>>>> signaling CLEAR_TO_TEARDOWN. If the gate stalls (watermark never >>> advances, >>>>> or the job fails before it signals), how do you picture the transition >>>>> unwinding? Is there a deadline that forces a rollback, or is that part >>> of >>>>> the error handling still being worked out? >>>>> >>>>> Also, I wonder how this approach would work with Pyflink DataStream >>> jobs. >>>>> My understanding is that Java agent may not be able to inject the gate >>> in >>>>> that case. Curious to hear your thoughts. >>>>> >>>>> Thank you! >>>>> Jing >>>>> >>>>> >>>>> >>>>> On 2026/05/04 17:56:58 Sergio Chong Loo wrote: > Hi Daniel / Ryan, > > >>>>> Again thanks a lot for the feedback. Here are some thoughts: > > The >>>>> following are excellent points and I indeed think we can work on them >>>>> immediately to incorporate them. I already have some ideas on >>>>> implementation but let me flesh them out more before I share them: > - >>>>> WatermarkGenerator > - Gate Parallelism > - TransitionMode default >>>>> (trivial) > > Thanks for the Misc Testing proposal (especially with >>> SQL), >>>>> it will indeed facilitate development and make sure the outcome/output >>> is >>>>> precise. Let me know if/when you choose to start this effort (or any >>> other >>>>> item for that matter). > > Exactly-once sinks. This needs explicit >>> testing >>>>> indeed. I haven't verified this one yet. I believe the gate operator >>> should >>>>> ensure green's sink transactions don't begin until blue has completed >>> its >>>>> final checkpoint and committed. This is a good candidate for a dedicated >>>>> test case before declaring phase 1 stable. > > Non-idempotent sinks. I'm >>>>> still not sure this is a problem meant to be solved by Blue/Green >>>>> deployments. If we think closely about it, even within a single Flink >>>>> pipeline, strict global ordering is only guaranteed within a single >>>>> subtask's partition. The moment you have parallelism > 1, different >>>>> subtasks process different keys/partitions independently and emit >>> records >>>>> at different rates. There's no global wall-clock ordering across >>> subtasks. >>>>> Flink's watermark mechanism gives you event-time ordering, not >>>>> arrival-order consistency across the whole topology. For pipelines where >>>>> strict ordering to a non-idempotent sink is a hard requirement, the >>> right >>>>> pattern is a savepoint based stop/restart rather than concurrent >>> blue/green >>>>> execution. > > On the other hand, assuming we pursue this, your proposal >>>>> sounds straightforward, but using state would imply the new pipeline >>> needs >>>>> to buffer an unbounded amount of data while waiting for the first >>>>> deployment to finish, which makes memory management and state sizing >>> very >>>>> difficult. The gate watermark barrier already provides a temporal >>> ordering >>>>> guarantee (green doesn't advance past blue's watermark), which covers >>> the >>>>> majority of idempotency-sensitive sinks. Perhaps I'm looking at this >>> from >>>>> an overly simplistic perspective, is there a concrete example you can >>> share >>>>> to illustrate the scenario you have in mind? We should definitely keep >>> this >>>>> topic open as we make progress on the other items. > > I'll keep you >>> posted >>>>> on progress for the items at the top. > > ⁃ Sergio > > > > On Apr 20, >>> 2026, >>>>> at 8:40 AM, Sergio Chong Loo <[email protected]> wrote: > > > > Hi >>> @Daniel / >>>>> @Ryan, > > > > Thanks a lot for the input. Similarly we’re swamped in a >>>>> time crunch here but I’ll be taking a deep dive into your feedback >>>>> hopefully before EoW. > > > > Stay tuned! > > > > - Sergio > > > >> On >>> Apr >>>>> 15, 2026, at 3:57 PM, Daniel Rossos <[email protected]> wrote: > >> > >>>>> >>>>> Hey Sergio, > >> > >> I got around to running locally as well as doing a >>>>> deeper dive into the current implementation details (Sorry about the >>>>> delay). This all looks super awesome and huge thanks for taking the >>> time to >>>>> put this together. I have some comments / questions below. Some I think >>>>> will be answered as we test further and some are potential next steps / >>>>> features that might be nice for phase 2. > >> > >> Non-Idempotent sinks >>>> >>>>>>> I believe we will still have non-idempotent sinks issues by using >>> these >>>>> gate functions. Brief recap of when this was raised before, records in >>> the >>>>> “green” deployment could be produced before the “blue” deployment >>> causing >>>>> an out-of-order delivery in record processing order which could have >>>>> implications for downstream sinks. One potential solution I was thinking >>>>> about that fits into this PR was to have the gate-operator be stateful >>> and >>>>> have your “green” pipeline accumulate messages after watermark barrier >>> and >>>>> wait for the “blue” to communicate (via the configmap) that it is done >>>>> processing before passing its records on. This would introduce a minimal >>>>> processing delay (poll rate of the “green” on configmap), but would >>> ensure >>>>> that the ordering of the streams to down stream sinks remains >>> consistent. >>>>> Other complications arise here with regards to handling state, but want >>> to >>>>> get your opinion here. > >> > >> WatermarkGenerator > >> Inserting a >>>>> watermark generator instead of requiring watermark in data. On the SQL >>> side >>>>> of things I noticed it is required that a field be a timestamp field >>> that >>>>> can be converted into a watermark. I was wondering if an alternative >>> would >>>>> be to inject a watermark generator step (generate based off some other >>>>> value), that way existing table defs won’t need to be changed to >>>>> accommodate this new feature. > >> > >> Exactly-once-sinks > >> How does >>>>> this work with exactly_once downstream sinks compatibility wise? For >>>>> example we should test using exactly_once kafka sinks to see if there >>> will >>>>> be any conflicts there. > >> > >> Gate Parallelism > >> From my >>>>> understanding, the parallelism of the gate operator has to be same as >>> the >>>>> sink/source operator it is tied to. With autoscaling how do these stay >>> in >>>>> sync? Could this cause problems? Can this gate become a performance >>>>> bottleneck somehow? > >> > >> TransitionMode Default > >> >>> `transitionMode` >>>>> not being set to default `BASIC` means all phase 1 Blue-Green deployment >>>>> would be broken specs on upgrade. I think we will want to include that >>>>> default > >> > >> Misc Testing > >> This is something for the future >>> (and >>>>> something I might play around with as I test), is adding a case to the >>>>> e2e-tests that creates a Flink pipeline (sql and/or non-sql) and checks >>>>> output from sink to ensure there are no duplicates. I don’t have a >>>>> convenient pipeline on hand > >> > >> Going forward, I am going to try >>> to >>>>> get a good test pipeline created and test this on our sandbox >>> environment >>>>> to see if I can get a good e2e run on prod-like conditions. > >> > >> >>>>> Thanks again, > >> > >> Daniel > >> > >> On Wed, Apr 1, 2026 at 12:25 PM >>>>> Sergio Chong Loo <[email protected] <[email protected]>> wrote: > >>> >>>>> Absolutely no rush, take your time. > >>> > >>> I’m also still working >>> on >>>>> some details around error handling. I’ll reach out offline so you can >>>>> always work on the latest. > >>> > >>> Thank you both as well for the >>>>> feedback! > >>> > >>> - Sergio > >>> > >>> > >>>> On Apr 1, 2026, at >>>>> 8:16 AM, Daniel Rossos <[email protected] <[email protected]>> >>>>> wrote: > >>>> > >>>> Hi Sergio, > >>>> > >>>> Sorry about the delay on >>> my >>>>> end (just got back after a few weeks off). I have caught up with + agree >>>>> with all the feedback Ryan provided in this thread. > >>>> > >>>> The >>> Gate >>>>> auto-injection idea is really cool. I'm going to try and make some time >>>>> this week to test out your PR on my side and provide feedback and >>> thoughts. >>>>>>>>>>>>>>> Thanks again for driving this, > >>>> Daniel > >>>> > >>>> > >>>>>>>>> On Thu, Mar 26, 2026 at 11:13 AM Ryan van Huuksloot via dev < >>>>> [email protected] <[email protected]>> wrote: > >>>>> Awesome! >>>>> Thanks for the update, Sergio. I'm excited to see the plan - it is > >>>>>>>> >>>>> a cool idea so I'm glad it is working. > >>>>> > >>>>> Ryan van >>> Huuksloot > >>>>>>>>>> Staff Engineer, Infrastructure | Streaming Platform > >>>>> [image: >>>>> Shopify] > >>>>> < >>>>> https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> >>>> >>>>>>>>>>>>>>>>>>>>>> On Thu, Mar 26, 2026 at 1:46 AM Sergio Chong Loo < >>>>> [email protected] <[email protected]>> > >>>>> wrote: > >>>>> > >>>>> > >>>>> @Ryan / @Daniel, > >>>>> > > >>>>> > Good news! The >>>>> “GateInjectorPipelineExecutor” idea is successful!! > >>>>> > > >>>>> > >>>>> While the original approach of simply activating it with > >>>>> > >>>>> “execution.target” did not quite work, I was able to implement it via >>> the >>>>> *Instrumentation > >>>>> > API with a Java Agent* that injects it… the >>> user >>>>> doesn’t have to touch > >>>>> > their pipelines and I added 2 options, >>> at >>>>> least for now, to place/inject > >>>>> > the Gate after the source or >>>>> before the sink (complex DAG cases with > >>>>> > multiple sources or >>> sinks >>>>> for now are not supported). > >>>>> > > >>>>> > I’m documenting >>> everything >>>>> and prepping the Draft PR for your review, > >>>>> > probably a couple >>> more >>>>> days. > >>>>> > > >>>>> > Thanks, stay tuned. > >>>>> > > >>>>> > - >>> Sergio >>>>>>>>>>>>>>>>>>>>>>>>>> On Mar 16, 2026, at 3:30 PM, Sergio Chong Loo >>>>> <[email protected] <[email protected]>> wrote: > >>>>> > > >>>>> > >>> Thanks >>>>> for the ideas and the offer to help out Ryan! It’s invaluable to > >>>>>>>>> >>>>> learn about how other users/teams scenarios. > >>>>> > > >>>>> > Indeed >>> I >>>>> have to pursue and evaluate the GateInjectorExecutor nonetheless > >>>>>>>>> >>>>> for our internal development. Ideally it’d be great if the user can >>> simply >>>>>>>>>>>> “invoke” the functionality, even give the user an option to >>>>> specify “where" > >>>>> > the gating mechanism to be placed (e.g. right >>>>> after the source or before > >>>>> > sink), or for the most flexibility >>>>> they can incorporate and place the Gate > >>>>> > manually just like it >>> is >>>>> now. > >>>>> > > >>>>> > I’ll share the progress asap and we can all >>> take >>>>> it from there (this > >>>>> > should not exceed a couple weeks). I’ll >>>>> definitely need more of your > >>>>> > feedback to verify this with >>> Flink >>>>> SQL. > >>>>> > > >>>>> > Thanks again, > >>>>> > Sergio > >>>>> > > >>>>>>>>> >>>>>>>>>>>> On Mar 16, 2026, at 7:01 AM, Ryan van Huuksloot < > >>>>> > >>>>> [email protected] <[email protected]>> wrote: > >>>>> > > >>>>>>>> >>>>>> Hi Sergio, > >>>>> > > >>>>> > re: 1.1 > >>>>> > My thought is that a >>>>> BlueGreen Mixin isn't Kubernetes specific and could > >>>>> > be reused >>> by >>>>> other deployment control planes. However, I do agree that > >>>>> > >>>>> attaching it to the sink has other implications so I am happy to pivot >>> if > >>>>>>>>>>> we can find an alternative solution. > >>>>> > > >>>>> > re: 2 > >>>>>>>>>>> I'm happy to leave it out of the Phase 2 implementation, but I >>>>> think it > >>>>> > should be possible. For example we use Phase 1 with >>>>> cross cluster > >>>>> > migrations today. Phase 2 within a single >>> cluster >>>>> isn't particularly useful > >>>>> > for us. > >>>>> > > >>>>> > re: >>>>> GateInjectorExecutor > >>>>> > This sounds like a neat idea. I need to >>> read >>>>> more about how it would work > >>>>> > but from a high level, injecting >>> an >>>>> operator before your sinks sounds like > >>>>> > a good idea. Better >>>>> isolation, possible with SQL, no mixins, etc. > >>>>> > > >>>>> > I will >>>>> mention that part of the reason I want it before the sinks is > >>>>> > >>>>> because nine out of ten people building pipelines struggle to >>> understand > >>>>>>>>>>> where their state is and how Phase 2 would affect the correctness >>>>> of their > >>>>> > state depending on where they put the gate. I >>> understand >>>>> that if you have a > >>>>> > remote lookup and want to save bandwidth, >>> you >>>>> could optimize your pipeline > >>>>> > by moving the gate before the >>> remote >>>>> call; however, that seems like an > >>>>> > optimization that can be >>> made >>>>> later. > >>>>> > > >>>>> > Thanks for driving this! Let me know how we >>> can >>>>> help. > >>>>> > > >>>>> > Ryan van Huuksloot > >>>>> > Staff Engineer, >>>>> Infrastructure | Streaming Platform > >>>>> > [image: Shopify] > >>>>> >>>> < >>>>> https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 16, 2026 at 2:21 AM Sergio Chong >>>>> Loo <[email protected] <[email protected]>> > >>>>> > wrote: > >>>>> > >>>> >>>>>>>>>>>> Hi Ryan > >>>>> >> > >>>>> >> Thanks a lot for these details. For >>>>> sure some of these observations > >>>>> >> popped up during our initial >>>>> discussions, and that’s why our initial goal > >>>>> >> was to introduce >>>>> this as simple as possible and gradually enhance it to > >>>>> >> cover >>>>> gaps. > >>>>> >> > >>>>> >> Allow me to address your concerns: > >>>>> >>>>>> >>>>>>>>>>>> 1. I’m happy you stressed the point of “disruption to existing > >>>>>>>>>>>> pipelines”. However, there’s a few points about attempting to >>>>> build this > >>>>> >> functionality into the sinks (or sources) right >>> off >>>>> the bat (read further > >>>>> >> below for my alternative): > >>>>> >> >>> 1. >>>>> Kubernetes centric: as of now the Blue/Green Deployments > >>>>> >> >>> support >>>>> is a Kubernetes specific solution, adding a mixin directly > >>>>> >> >>>>> available to sinks would “leak” this support outside of K8s > >>>>> >> >>> 2. A >>>>> sink being aware of these deployment phases violates single > >>>>> >> >>>>> responsibility, but more importantly… > >>>>> >> 3. Flink currently has >>>>> many connectors, with the majority being > >>>>> >> maintained outside >>> of >>>>> the Flink code base, by separate teams, separate > >>>>> >> repos, >>> separate >>>>> release cycles. This would complicate things significantly > >>>>> >> >>> as to >>>>> try and add support for this for every potential flink connector > >>>>>>>>>> >>>>> project out there would be a cumbersome. Blue/Green Phase 2 then only >>> would >>>>>>>>>>>>> works with "gate-aware" sinks. > >>>>> >> 2. I’d leave the >>>>> conversation about migrating jobs between K8s > >>>>> >> clusters >>> outside >>>>> of this scope, even Phase 1 is meant to only work in a > >>>>> >> single >>>>> cluster… > >>>>> >> 3. Watermarking, excellent point, it’s indeed a >>>>> requirement so I’ll > >>>>> >> make sure this is validated where >>> applicable >>>>> (by the concrete > >>>>> >> implementation) > >>>>> >> > >>>>> >> > >>>>>>>> >>>>>>> Having said what I said about point 1.1 above, I’m currently working >>> on >>>>>>>>>>>>> an approach which uses a “GateInjectorPipelineExecutor” so to >>>>> speak; in > >>>>> >> other words a custom PipelineExecutor that would be >>>>> shipped with the K8s > >>>>> >> Operator, invoked by Flink Configuration >>>>> (via “execution.target:”). This > >>>>> >> custom piece would >>> instantiate >>>>> and inject the Gate at a fixed point in the > >>>>> >> StreamGraph right >>>>> before job submission. I still have to validate and > >>>>> >> ensure a >>> few >>>>> things are correctly taken care of (like Type Information, > >>>>> >> >>> etc.) >>>>> but the theory looks promising. > >>>>> >> > >>>>> >> For the most part >>>>> this works well with Flink SQL (same configuration), > >>>>> >> here’s >>> my >>>>> estimation: > >>>>> >> > >>>>> >> tEnv.executeSql("INSERT INTO my_sink >>>>> ...") > >>>>> >> └─> SQL planner → ExecNodeGraph → Transformation[] > >>>>>>>> >>>>>>> └─> StreamGraph > >>>>> >> └─> GateInjectorExecutor injects >>>>> GateProcessFunction > >>>>> >> └─> StreamGraph' (mutated) → JobGraph > >>>>>>>>>>>> └─> Submit Job > >>>>> >> > >>>>> >> I’m aiming to share some >>>>> updates along these lines in the next few weeks > >>>>> >> but hopefully >>>>> this falls inline with your objectives/thoughts overall. > >>>>> >> > >>>>>>>> >>>>>>> Sergio > >>>>> >> > >>>>> >> > >>>>> >> On Mar 6, 2026, at 3:36 PM, >>> Ryan >>>>> van Huuksloot via dev < > >>>>> >> [email protected] < >>>>> [email protected]>> wrote: > >>>>> >> > >>>>> >> Hi Sergio, > >>>>>>>>>> >>>>> Thanks for starting this conversation. > >>>>> >> > >>>>> >> A few >>> thoughts >>>>> regarding BlueGreen Phase 2: > >>>>> >> 1. The Gate Operator is >>> interesting >>>>> but I don't like that we would have to > >>>>> >> modify users' >>> pipelines >>>>> for them to use Phase 2. This gate function seems > >>>>> >> like it >>> could >>>>> be a Mixin that connectors would implement. If you want to > >>>>> >> >>> use >>>>> Phase 2, your sinks must implement this Mixin. I understand that a > >>>>>>>> >>>>>>> unique GateFunction has pros, but it works less well with FlinkSQL - >>> and >>>>>>>>>>>>> the trade-off doesn't seem worthwhile. > >>>>> >> 2. Regarding >>>>> the ConfigMap. We should consider a solution that supports > >>>>> >> >>>>> migrating Flink jobs between Kubernetes clusters. Otherwise Phase 2 is > >>>>>>>>>>>> only > >>>>> >> useful for in cluster operations. > >>>>> >> 3. >>>>> Watermarking is a requirement. Will the Flink Kubernetes Operator > >>>>>>>> >>>>>>> validate that the pipeline is using watermarks? > >>>>> >> > >>>>> >> >>>>> What happens when idleness is configured? Watermarks will get ignored >>> from >>>>>>>>>>>>>>>>>>>>> these “slow” subtasks and advance, could records from >>>>> the ignored subtasks > >>>>> >> eventually be lost? > >>>>> >> Yes they >>>>> would be lost, but that wo [message truncated...] >>>> >>> >>> >
