Thanks for the feedback Ryan, I’ll re-focus on this soon and will keep you posted.
FYI: @Gyula - Sergio > On Aug 10, 2026, at 10:12 AM, Ryan van Huuksloot via dev > <[email protected]> wrote: > > Hi Sergio, > > Thanks for driving this. We are still interested but are currently caught > up in other work, which explains the delay. > > Re: What we should include. > > The 4 points are good. > Additionally, your deferral is unclear. Are SQL/Table pipelines supported > if they use watermarks? That should be included. > I also think that having an e2e harness might need to be part of the > success criteria. If we have one test we may as well build it into the > harness. > > I agree with what is out of scope for Phase 2. > > The last thing I would point out is that the documentation will be really > important for this feature. Your callouts are great and I would do +100 to > what you have for documentation. > > Thanks > > Ryan van Huuksloot > Staff Engineer, Infrastructure | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Mon, Jul 6, 2026 at 12:50 PM Sergio Chong Loo <[email protected]> > wrote: > >> Hi Daniel / Ryan / Gyula, >> >> Overdue update, and apologies for the long gap since May. A no-code >> watermark option jumped >> the queue (a few teams were blocked without it), so I pushed that >> through first. Quick >> summary of what landed, then a proposal to lock the remaining scope so >> we share a finish >> line. I know it’s a bit lengthy but please bear with me. >> >> Landed since May: >> - No-code watermark extraction: bluegreen.gate.watermark.field-path. >> Instead of a custom >> extractor-class, point the gate at a dot-notation field path: >> bluegreen.gate.strategy: "WATERMARK" >> bluegreen.gate.watermark.field-path: "eventTime" # or nested: >> "metadata.timestamp" >> It reads an epoch-ms long off the record (public field, class >> hierarchy, or getXxx() >> getter), cached with no per-record reflection, and works out of the >> box with protobuf >> classes. extractor-class stays for anything a field path cannot >> express. Scope note, since >> it is close to your WatermarkGenerator point but not the same: it >> removes extractor code >> for DataStream/POJO jobs, but it still reads an existing field and >> does not cover the >> SQL/Table case you raised (that stays open, see Defer). >> - transitionMode: you were right that the unset case was a real gap (it >> errored instead of >> defaulting), so it now defaults to BASIC (existing basic deployments >> upgrade cleanly) and >> is immutable after the initial deployment (a change is rejected with a >> validation error >> and a Warning event). >> - Rebased onto current operator main, and the gate-strategy layer is now >> extensible >> (watermark is one strategy). >> >> Proposed finish line: Phase 2 to be marked done when advanced Blue/Green >> is correct >> within a documented envelope, never hangs silently, and its >> no-dup/no-loss claim is proven >> by at least one real end-to-end test. Anything that widens the envelope >> is a follow-up. >> >> Must-have: >> 1. Gate-clear timeout. If a transition's gate never clears (watermark >> stalls), the >> controller currently waits indefinitely. Add a deadline so it fails >> with a clear error >> instead of hanging. >> 2. One real no-dup/no-loss end-to-end test for the supported shape: >> single source, >> idempotent or transactional sink, happy-path cutover plus a failure >> injected around the >> cutover. This turns the headline claim from asserted to proven, and I >> am happy to help >> wire it into the e2e suite. Daniel, if your sandbox pipeline is >> close, this is exactly it. >> 3. Documented envelope and limitations: one meaningful watermark at the >> gate point; sink >> must be idempotent or transactional for end-to-end exactly-once; the >> idleness caveat; >> JVM-only auto-injection. >> 4. Gate parallelism (your point). The agent already injects the gate at >> the adjacent >> operator's parallelism with a forward connection, so it adds no >> shuffle and normally >> chains into that vertex. The remaining work is to make that chaining >> reliable (also match >> the neighbor's max-parallelism and set the chaining strategy >> explicitly) so the gate is >> never a standalone vertex the autoscaler can rescale on its own, plus >> a documented >> pin/exclude fallback for the rare non-chainable case. That closes the >> bottleneck-under-autoscaling concern. >> >> Defer (follow-ups): >> - Full conformance matrix (every sink class by failure point by >> idleness), beyond the one >> core test. >> - SQL/Table watermark-generator injection so table defs need no >> timestamp field. This is >> really SQL support, a separate surface from DataStream advanced BG; >> proposing it as a >> Phase 3 item. >> - A broader shared e2e harness beyond the core test. >> >> Out of scope for Phase 2 (proposing we close these): >> - Multiple sources with diverging watermarks, and mixed source types. >> The gate assumes one >> meaningful watermark; per-input readiness is a future gate strategy. >> - Guaranteed no-loss when a gated source uses idleness. Documented >> limitation; a future >> per-source-readiness strategy could address it, at the cost of the >> very stall the >> gate-clear timeout is there to catch. >> - Pyflink auto-injection. JVM-only this phase; the agent rewrites the >> JVM job graph and >> reads records as Java objects. >> >> Asks: >> - Agree on the four must-haves as the finish line, and sign off on the >> deferred and >> out-of-scope lists. >> - Daniel: how close is the sandbox pipeline to serving as the core test >> (item 2 below)? And can >> you confirm the exactly-once expectation to assert, that green's sink >> transactions do not >> commit until blue has taken and committed its final checkpoint? >> >> Thanks. This last stretch is short if we agree on the boundary. >> >> - Sergio >> >>> Hi Daniel / Ryan, >>> >>> Again thanks a lot for the feedback. Here are some thoughts: >>> >>> The following are excellent points and I indeed think we can work on >> them immediately to incorporate them. I already have some ideas on >> implementation but let me flesh them out more before I share them: >>> - WatermarkGenerator >>> - Gate Parallelism >>> - TransitionMode default (trivial) >>> >>> Thanks for the Misc Testing proposal (especially with SQL), it will >> indeed facilitate development and make sure the outcome/output is precise. >> Let me know if/when you choose to start this effort (or any other item for >> that matter). >>> >>> Exactly-once sinks. This needs explicit testing indeed. I haven't >> verified this one yet. I believe the gate operator should ensure green's >> sink transactions don't begin until blue has completed its final checkpoint >> and committed. This is a good candidate for a dedicated test case before >> declaring phase 1 stable. >>> >>> Non-idempotent sinks. I'm still not sure this is a problem meant to be >> solved by Blue/Green deployments. If we think closely about it, even within >> a single Flink pipeline, strict global ordering is only guaranteed within a >> single subtask's partition. The moment you have parallelism > 1, different >> subtasks process different keys/partitions independently and emit records >> at different rates. There's no global wall-clock ordering across subtasks. >> Flink's watermark mechanism gives you event-time ordering, not >> arrival-order consistency across the whole topology. For pipelines where >> strict ordering to a non-idempotent sink is a hard requirement, the right >> pattern is a savepoint based stop/restart rather than concurrent blue/green >> execution. >>> >>> On the other hand, assuming we pursue this, your proposal sounds >> straightforward, but using state would imply the new pipeline needs to >> buffer an unbounded amount of data while waiting for the first deployment >> to finish, which makes memory management and state sizing very difficult. >> The gate watermark barrier already provides a temporal ordering guarantee >> (green doesn't advance past blue's watermark), which covers the majority of >> idempotency-sensitive sinks. Perhaps I'm looking at this from an overly >> simplistic perspective, is there a concrete example you can share to >> illustrate the scenario you have in mind? We should definitely keep this >> topic open as we make progress on the other items. >>> >>> I'll keep you posted on progress for the items at the top. >>> >>> ⁃ Sergio >> >>> On Jul 2, 2026, at 4:52 PM, Sergio Chong Loo <[email protected]> >> wrote: >>> >>> Sorry for the late reply Vamshi, these are the right questions for a >> zero-loss/zero-dup rollout. >>> >>> One thing up front, because it answers most of these: the gate works at >> a single point in >>> the job, right after a source or right before a sink. There it reads >> one watermark >>> (whatever Flink has combined at that point) and checks each record's >> event-time once >>> against a single cutover value. So it assumes a simple, linear spot >> where one watermark >>> cleanly governs the records passing through. In short: the pipeline has >> to give the gate >>> one event-time per record to compare against one cutover, once. Most of >> your questions are >>> really "what if it isn't that simple," and in this Advanced mode (Phase >> 2) the answer is: >>> place the gate on a stretch where it stays simple. >>> >>> 1. Multiple sources / diverging watermarks. No special rule: the gate >> uses whatever >>> watermark Flink has already combined at that point (for multiple >> inputs, the minimum across >>> them), plus each record's own event-time. So it effectively waits for >> the slowest input; it >>> doesn't track inputs individually. Jobs that need per-input logic >> should keep the gate on a >>> single-watermark stretch. >>> >>> 2. Idleness / lag. A slow-but-active source just holds the watermark >> back, so the cutover >>> waits. That is safe (no loss) but can stall, which is why we are adding >> a deadline so a >>> permanent stall fails loudly rather than hanging. Idleness is the sharp >> edge, and a real >>> caveat on no-loss: a source marked idle is dropped from the >> current/minimum watermark >>> (by design, for liveness), so the gate can cut over ahead of it. >> Records it emits afterward carry >>> an event-time before the cutover, and once the outgoing deployment is >> torn down they have >>> nowhere to land, so they are dropped. This is the same >> completeness-versus-liveness >>> tradeoff idleness always makes, turned into a hard boundary by the >> cutover. Stated plainly: >>> no-loss holds only if no source feeding the gate is idle during the >> transition. If your >>> sources use idleness, treat no-loss as best-effort across the cutover. >> (A future gate >>> strategy could prob. wait for per-source readiness, including idle >> sources, at the cost of >>> re-introducing that stall). >>> >>> 3. Mixed sources / event-time quality. Works for: event-time jobs with >> one solid watermark >>> and a clear per-record event-time where the gate sits. Out of scope for >> Phase 2: weak or >>> absent event-time, uneven event-time quality across inputs, or anything >> that needs several >>> watermarks combined. >>> >>> 4. Sink guarantees. The gate makes a clean split (the old job emits up >> to the cutover, the >>> new job after it), so normally each record goes out once. But the gate >> only coordinates the >>> cutover; it can't take back a duplicate the sink already wrote. During >> the overlap, and >>> especially if a job restarts and replays, true exactly-once still needs >> an idempotent or >>> transactional sink. So I'd treat idempotent-or-transactional sinks as a >> requirement for the >>> no-dup/no-loss claim. For Kafka, EOS transactions are the main >> backstop. The gate narrows >>> the window; the sink closes it. >>> >>> 5. Conformance matrix. Not yet: we unit-test the controller and the >> gate's record-level >>> logic, but there's no end-to-end no-dup/no-loss test grid, and that's >> the real gap (and my >>> next step, happy to build it with you). The grid would cross: number of >> sources, watermark >>> divergence, idleness, where a failure happens, and sink type, each >> checked for duplicates >>> and loss. Honest status: the single-source and idleness rows are the >> near-term ones; the >>> failover + sink-type rows (the ones that matter most for a hard >> zero-loss/zero-dup bar) are >>> the bigger lift and tie into the cutover-recovery work, so I wouldn't >> call failover >>> correctness proven yet. Send the scenarios that matter to you and I'll >> add them. >>> >>> And none of the "out of scope" items are dead-ends. The watermark gate >> is just one strategy >>> (bluegreen.gate.strategy=WATERMARK) on an extensible gate layer. The >> whole idea is that >>> more complex cutover logic (multiple watermarks, per-input readiness, >> custom rules) can be >>> added as a new gate implementation instead of complicating this one. So >> "out of scope for >>> Phase 2" really means "a future strategy," and that's exactly where I'd >> welcome help. >>> >>> This is the pressure-testing Phase 2 needs. Much appreciated. >>> >>>> Hi Sergio, >>>> Thanks for driving FLIP-504 forward. This is great and the top most need >>>> for Flink B/G cutovers. I am evaluating Phase 2 against our current >>>> production use cases, where zero-loss/zero-duplication during cutover >> is a >>>> hard requirement. I would appreciate a few clarifications on the >> intended >>>> implementation semantics. >>>> >>>> 1. For multi-source jobs, if watermarks diverge, what is the intended >>>> cutover rule (min-watermark, per-input readiness, or other)? >>>> 2. How should idleness/temporary lag on one source affect transition >>>> readiness to avoid loss? >>>> 3. For mixed source types/event-time quality, what behavior is >> supported >>>> vs out of scope in Phase 2? >>>> 4. For “no-dup/no-loss” expectations, what sink guarantees are assumed >>>> (idempotent/transactional required or recommended)? For Kafka sinks >>>> specifically, should EOS transactional mode be treated as the primary >>>> backstop? >>>> 5. Is there a minimal conformance test matrix planned (divergence, >>>> idleness, failover, duplicate/loss verification)? Any guidance here >> would >>>> really help teams roll out safely with clear correctness boundaries. >>>> >>>> Thanks, >>>> Vamshi >>> >>> >>>> On Jun 21, 2026, at 2:57 PM, Jing-Jia Hung <[email protected]> >> wrote: >>>> >>>> Hi Sergio, >>>> >>>> Late to the thread. The gate auto injection via the Java agent is cool. >>>> >>>> Most of the discussion so far has centered on data-plane correctness. >> The >>>> area I'd like to understand better is the control-plane side, >> specifically >>>> recovery when a transition doesn't complete. The FLIP notes the >> controller >>>> can be left in a bad state if a deployment fails mid-transition. From >>>> reading the draft it looks like the teardown step waits on the gate >>>> signaling CLEAR_TO_TEARDOWN. If the gate stalls (watermark never >> advances, >>>> or the job fails before it signals), how do you picture the transition >>>> unwinding? Is there a deadline that forces a rollback, or is that part >> of >>>> the error handling still being worked out? >>>> >>>> Also, I wonder how this approach would work with Pyflink DataStream >> jobs. >>>> My understanding is that Java agent may not be able to inject the gate >> in >>>> that case. Curious to hear your thoughts. >>>> >>>> Thank you! >>>> Jing >>>> >>>> >>>> >>>> On 2026/05/04 17:56:58 Sergio Chong Loo wrote: > Hi Daniel / Ryan, > > >>>> Again thanks a lot for the feedback. Here are some thoughts: > > The >>>> following are excellent points and I indeed think we can work on them >>>> immediately to incorporate them. I already have some ideas on >>>> implementation but let me flesh them out more before I share them: > - >>>> WatermarkGenerator > - Gate Parallelism > - TransitionMode default >>>> (trivial) > > Thanks for the Misc Testing proposal (especially with >> SQL), >>>> it will indeed facilitate development and make sure the outcome/output >> is >>>> precise. Let me know if/when you choose to start this effort (or any >> other >>>> item for that matter). > > Exactly-once sinks. This needs explicit >> testing >>>> indeed. I haven't verified this one yet. I believe the gate operator >> should >>>> ensure green's sink transactions don't begin until blue has completed >> its >>>> final checkpoint and committed. This is a good candidate for a dedicated >>>> test case before declaring phase 1 stable. > > Non-idempotent sinks. I'm >>>> still not sure this is a problem meant to be solved by Blue/Green >>>> deployments. If we think closely about it, even within a single Flink >>>> pipeline, strict global ordering is only guaranteed within a single >>>> subtask's partition. The moment you have parallelism > 1, different >>>> subtasks process different keys/partitions independently and emit >> records >>>> at different rates. There's no global wall-clock ordering across >> subtasks. >>>> Flink's watermark mechanism gives you event-time ordering, not >>>> arrival-order consistency across the whole topology. For pipelines where >>>> strict ordering to a non-idempotent sink is a hard requirement, the >> right >>>> pattern is a savepoint based stop/restart rather than concurrent >> blue/green >>>> execution. > > On the other hand, assuming we pursue this, your proposal >>>> sounds straightforward, but using state would imply the new pipeline >> needs >>>> to buffer an unbounded amount of data while waiting for the first >>>> deployment to finish, which makes memory management and state sizing >> very >>>> difficult. The gate watermark barrier already provides a temporal >> ordering >>>> guarantee (green doesn't advance past blue's watermark), which covers >> the >>>> majority of idempotency-sensitive sinks. Perhaps I'm looking at this >> from >>>> an overly simplistic perspective, is there a concrete example you can >> share >>>> to illustrate the scenario you have in mind? We should definitely keep >> this >>>> topic open as we make progress on the other items. > > I'll keep you >> posted >>>> on progress for the items at the top. > > ⁃ Sergio > > > > On Apr 20, >> 2026, >>>> at 8:40 AM, Sergio Chong Loo <[email protected]> wrote: > > > > Hi >> @Daniel / >>>> @Ryan, > > > > Thanks a lot for the input. Similarly we’re swamped in a >>>> time crunch here but I’ll be taking a deep dive into your feedback >>>> hopefully before EoW. > > > > Stay tuned! > > > > - Sergio > > > >> On >> Apr >>>> 15, 2026, at 3:57 PM, Daniel Rossos <[email protected]> wrote: > >> > >>>> >>>> Hey Sergio, > >> > >> I got around to running locally as well as doing a >>>> deeper dive into the current implementation details (Sorry about the >>>> delay). This all looks super awesome and huge thanks for taking the >> time to >>>> put this together. I have some comments / questions below. Some I think >>>> will be answered as we test further and some are potential next steps / >>>> features that might be nice for phase 2. > >> > >> Non-Idempotent sinks >>> >>>>>> I believe we will still have non-idempotent sinks issues by using >> these >>>> gate functions. Brief recap of when this was raised before, records in >> the >>>> “green” deployment could be produced before the “blue” deployment >> causing >>>> an out-of-order delivery in record processing order which could have >>>> implications for downstream sinks. One potential solution I was thinking >>>> about that fits into this PR was to have the gate-operator be stateful >> and >>>> have your “green” pipeline accumulate messages after watermark barrier >> and >>>> wait for the “blue” to communicate (via the configmap) that it is done >>>> processing before passing its records on. This would introduce a minimal >>>> processing delay (poll rate of the “green” on configmap), but would >> ensure >>>> that the ordering of the streams to down stream sinks remains >> consistent. >>>> Other complications arise here with regards to handling state, but want >> to >>>> get your opinion here. > >> > >> WatermarkGenerator > >> Inserting a >>>> watermark generator instead of requiring watermark in data. On the SQL >> side >>>> of things I noticed it is required that a field be a timestamp field >> that >>>> can be converted into a watermark. I was wondering if an alternative >> would >>>> be to inject a watermark generator step (generate based off some other >>>> value), that way existing table defs won’t need to be changed to >>>> accommodate this new feature. > >> > >> Exactly-once-sinks > >> How does >>>> this work with exactly_once downstream sinks compatibility wise? For >>>> example we should test using exactly_once kafka sinks to see if there >> will >>>> be any conflicts there. > >> > >> Gate Parallelism > >> From my >>>> understanding, the parallelism of the gate operator has to be same as >> the >>>> sink/source operator it is tied to. With autoscaling how do these stay >> in >>>> sync? Could this cause problems? Can this gate become a performance >>>> bottleneck somehow? > >> > >> TransitionMode Default > >> >> `transitionMode` >>>> not being set to default `BASIC` means all phase 1 Blue-Green deployment >>>> would be broken specs on upgrade. I think we will want to include that >>>> default > >> > >> Misc Testing > >> This is something for the future >> (and >>>> something I might play around with as I test), is adding a case to the >>>> e2e-tests that creates a Flink pipeline (sql and/or non-sql) and checks >>>> output from sink to ensure there are no duplicates. I don’t have a >>>> convenient pipeline on hand > >> > >> Going forward, I am going to try >> to >>>> get a good test pipeline created and test this on our sandbox >> environment >>>> to see if I can get a good e2e run on prod-like conditions. > >> > >> >>>> Thanks again, > >> > >> Daniel > >> > >> On Wed, Apr 1, 2026 at 12:25 PM >>>> Sergio Chong Loo <[email protected] <[email protected]>> wrote: > >>> >>>> Absolutely no rush, take your time. > >>> > >>> I’m also still working >> on >>>> some details around error handling. I’ll reach out offline so you can >>>> always work on the latest. > >>> > >>> Thank you both as well for the >>>> feedback! > >>> > >>> - Sergio > >>> > >>> > >>>> On Apr 1, 2026, at >>>> 8:16 AM, Daniel Rossos <[email protected] <[email protected]>> >>>> wrote: > >>>> > >>>> Hi Sergio, > >>>> > >>>> Sorry about the delay on >> my >>>> end (just got back after a few weeks off). I have caught up with + agree >>>> with all the feedback Ryan provided in this thread. > >>>> > >>>> The >> Gate >>>> auto-injection idea is really cool. I'm going to try and make some time >>>> this week to test out your PR on my side and provide feedback and >> thoughts. >>>>>>>>>>>>>> Thanks again for driving this, > >>>> Daniel > >>>> > >>>> > >>>>>>>> On Thu, Mar 26, 2026 at 11:13 AM Ryan van Huuksloot via dev < >>>> [email protected] <[email protected]>> wrote: > >>>>> Awesome! >>>> Thanks for the update, Sergio. I'm excited to see the plan - it is > >>>>>>> >>>> a cool idea so I'm glad it is working. > >>>>> > >>>>> Ryan van >> Huuksloot > >>>>>>>>> Staff Engineer, Infrastructure | Streaming Platform > >>>>> [image: >>>> Shopify] > >>>>> < >>>> https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> >>> >>>>>>>>>>>>>>>>>>>>> On Thu, Mar 26, 2026 at 1:46 AM Sergio Chong Loo < >>>> [email protected] <[email protected]>> > >>>>> wrote: > >>>>> > >>>>> > >>>> @Ryan / @Daniel, > >>>>> > > >>>>> > Good news! The >>>> “GateInjectorPipelineExecutor” idea is successful!! > >>>>> > > >>>>> > >>>> While the original approach of simply activating it with > >>>>> > >>>> “execution.target” did not quite work, I was able to implement it via >> the >>>> *Instrumentation > >>>>> > API with a Java Agent* that injects it… the >> user >>>> doesn’t have to touch > >>>>> > their pipelines and I added 2 options, >> at >>>> least for now, to place/inject > >>>>> > the Gate after the source or >>>> before the sink (complex DAG cases with > >>>>> > multiple sources or >> sinks >>>> for now are not supported). > >>>>> > > >>>>> > I’m documenting >> everything >>>> and prepping the Draft PR for your review, > >>>>> > probably a couple >> more >>>> days. > >>>>> > > >>>>> > Thanks, stay tuned. > >>>>> > > >>>>> > - >> Sergio >>>>>>>>>>>>>>>>>>>>>>>>> On Mar 16, 2026, at 3:30 PM, Sergio Chong Loo >>>> <[email protected] <[email protected]>> wrote: > >>>>> > > >>>>> > >> Thanks >>>> for the ideas and the offer to help out Ryan! It’s invaluable to > >>>>>>>> >>>> learn about how other users/teams scenarios. > >>>>> > > >>>>> > Indeed >> I >>>> have to pursue and evaluate the GateInjectorExecutor nonetheless > >>>>>>>> >>>> for our internal development. Ideally it’d be great if the user can >> simply >>>>>>>>>>> “invoke” the functionality, even give the user an option to >>>> specify “where" > >>>>> > the gating mechanism to be placed (e.g. right >>>> after the source or before > >>>>> > sink), or for the most flexibility >>>> they can incorporate and place the Gate > >>>>> > manually just like it >> is >>>> now. > >>>>> > > >>>>> > I’ll share the progress asap and we can all >> take >>>> it from there (this > >>>>> > should not exceed a couple weeks). I’ll >>>> definitely need more of your > >>>>> > feedback to verify this with >> Flink >>>> SQL. > >>>>> > > >>>>> > Thanks again, > >>>>> > Sergio > >>>>> > > >>>>>>>> >>>>>>>>>>> On Mar 16, 2026, at 7:01 AM, Ryan van Huuksloot < > >>>>> > >>>> [email protected] <[email protected]>> wrote: > >>>>> > > >>>>>>> >>>>> Hi Sergio, > >>>>> > > >>>>> > re: 1.1 > >>>>> > My thought is that a >>>> BlueGreen Mixin isn't Kubernetes specific and could > >>>>> > be reused >> by >>>> other deployment control planes. However, I do agree that > >>>>> > >>>> attaching it to the sink has other implications so I am happy to pivot >> if > >>>>>>>>>> we can find an alternative solution. > >>>>> > > >>>>> > re: 2 > >>>>>>>>>> I'm happy to leave it out of the Phase 2 implementation, but I >>>> think it > >>>>> > should be possible. For example we use Phase 1 with >>>> cross cluster > >>>>> > migrations today. Phase 2 within a single >> cluster >>>> isn't particularly useful > >>>>> > for us. > >>>>> > > >>>>> > re: >>>> GateInjectorExecutor > >>>>> > This sounds like a neat idea. I need to >> read >>>> more about how it would work > >>>>> > but from a high level, injecting >> an >>>> operator before your sinks sounds like > >>>>> > a good idea. Better >>>> isolation, possible with SQL, no mixins, etc. > >>>>> > > >>>>> > I will >>>> mention that part of the reason I want it before the sinks is > >>>>> > >>>> because nine out of ten people building pipelines struggle to >> understand > >>>>>>>>>> where their state is and how Phase 2 would affect the correctness >>>> of their > >>>>> > state depending on where they put the gate. I >> understand >>>> that if you have a > >>>>> > remote lookup and want to save bandwidth, >> you >>>> could optimize your pipeline > >>>>> > by moving the gate before the >> remote >>>> call; however, that seems like an > >>>>> > optimization that can be >> made >>>> later. > >>>>> > > >>>>> > Thanks for driving this! Let me know how we >> can >>>> help. > >>>>> > > >>>>> > Ryan van Huuksloot > >>>>> > Staff Engineer, >>>> Infrastructure | Streaming Platform > >>>>> > [image: Shopify] > >>>>> >>> < >>>> https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> >>> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 16, 2026 at 2:21 AM Sergio Chong >>>> Loo <[email protected] <[email protected]>> > >>>>> > wrote: > >>>>> > >>> >>>>>>>>>>> Hi Ryan > >>>>> >> > >>>>> >> Thanks a lot for these details. For >>>> sure some of these observations > >>>>> >> popped up during our initial >>>> discussions, and that’s why our initial goal > >>>>> >> was to introduce >>>> this as simple as possible and gradually enhance it to > >>>>> >> cover >>>> gaps. > >>>>> >> > >>>>> >> Allow me to address your concerns: > >>>>> >>>>> >>>>>>>>>>> 1. I’m happy you stressed the point of “disruption to existing > >>>>>>>>>>> pipelines”. However, there’s a few points about attempting to >>>> build this > >>>>> >> functionality into the sinks (or sources) right >> off >>>> the bat (read further > >>>>> >> below for my alternative): > >>>>> >> >> 1. >>>> Kubernetes centric: as of now the Blue/Green Deployments > >>>>> >> >> support >>>> is a Kubernetes specific solution, adding a mixin directly > >>>>> >> >>>> available to sinks would “leak” this support outside of K8s > >>>>> >> >> 2. A >>>> sink being aware of these deployment phases violates single > >>>>> >> >>>> responsibility, but more importantly… > >>>>> >> 3. Flink currently has >>>> many connectors, with the majority being > >>>>> >> maintained outside >> of >>>> the Flink code base, by separate teams, separate > >>>>> >> repos, >> separate >>>> release cycles. This would complicate things significantly > >>>>> >> >> as to >>>> try and add support for this for every potential flink connector > >>>>>>>>> >>>> project out there would be a cumbersome. Blue/Green Phase 2 then only >> would >>>>>>>>>>>> works with "gate-aware" sinks. > >>>>> >> 2. I’d leave the >>>> conversation about migrating jobs between K8s > >>>>> >> clusters >> outside >>>> of this scope, even Phase 1 is meant to only work in a > >>>>> >> single >>>> cluster… > >>>>> >> 3. Watermarking, excellent point, it’s indeed a >>>> requirement so I’ll > >>>>> >> make sure this is validated where >> applicable >>>> (by the concrete > >>>>> >> implementation) > >>>>> >> > >>>>> >> > >>>>>>> >>>>>> Having said what I said about point 1.1 above, I’m currently working >> on >>>>>>>>>>>> an approach which uses a “GateInjectorPipelineExecutor” so to >>>> speak; in > >>>>> >> other words a custom PipelineExecutor that would be >>>> shipped with the K8s > >>>>> >> Operator, invoked by Flink Configuration >>>> (via “execution.target:”). This > >>>>> >> custom piece would >> instantiate >>>> and inject the Gate at a fixed point in the > >>>>> >> StreamGraph right >>>> before job submission. I still have to validate and > >>>>> >> ensure a >> few >>>> things are correctly taken care of (like Type Information, > >>>>> >> >> etc.) >>>> but the theory looks promising. > >>>>> >> > >>>>> >> For the most part >>>> this works well with Flink SQL (same configuration), > >>>>> >> here’s >> my >>>> estimation: > >>>>> >> > >>>>> >> tEnv.executeSql("INSERT INTO my_sink >>>> ...") > >>>>> >> └─> SQL planner → ExecNodeGraph → Transformation[] > >>>>>>> >>>>>> └─> StreamGraph > >>>>> >> └─> GateInjectorExecutor injects >>>> GateProcessFunction > >>>>> >> └─> StreamGraph' (mutated) → JobGraph > >>>>>>>>>>> └─> Submit Job > >>>>> >> > >>>>> >> I’m aiming to share some >>>> updates along these lines in the next few weeks > >>>>> >> but hopefully >>>> this falls inline with your objectives/thoughts overall. > >>>>> >> > >>>>>>> >>>>>> Sergio > >>>>> >> > >>>>> >> > >>>>> >> On Mar 6, 2026, at 3:36 PM, >> Ryan >>>> van Huuksloot via dev < > >>>>> >> [email protected] < >>>> [email protected]>> wrote: > >>>>> >> > >>>>> >> Hi Sergio, > >>>>>>>>> >>>> Thanks for starting this conversation. > >>>>> >> > >>>>> >> A few >> thoughts >>>> regarding BlueGreen Phase 2: > >>>>> >> 1. The Gate Operator is >> interesting >>>> but I don't like that we would have to > >>>>> >> modify users' >> pipelines >>>> for them to use Phase 2. This gate function seems > >>>>> >> like it >> could >>>> be a Mixin that connectors would implement. If you want to > >>>>> >> >> use >>>> Phase 2, your sinks must implement this Mixin. I understand that a > >>>>>>> >>>>>> unique GateFunction has pros, but it works less well with FlinkSQL - >> and >>>>>>>>>>>> the trade-off doesn't seem worthwhile. > >>>>> >> 2. Regarding >>>> the ConfigMap. We should consider a solution that supports > >>>>> >> >>>> migrating Flink jobs between Kubernetes clusters. Otherwise Phase 2 is > >>>>>>>>>>> only > >>>>> >> useful for in cluster operations. > >>>>> >> 3. >>>> Watermarking is a requirement. Will the Flink Kubernetes Operator > >>>>>>> >>>>>> validate that the pipeline is using watermarks? > >>>>> >> > >>>>> >> >>>> What happens when idleness is configured? Watermarks will get ignored >> from >>>>>>>>>>>>>>>>>>>> these “slow” subtasks and advance, could records from >>>> the ignored subtasks > >>>>> >> eventually be lost? > >>>>> >> Yes they >>>> would be lost, but that wo [message truncated...] >>> >> >>
