Hi Piyush, I checked the AIP again, and it looks much better! Thanks for iterating over this proposal. I left a few minor editorial comments, and I also saw that Christos left comments we should address before voting.
Please also keep in mind that most of the maintainers are currently occupied by the upcoming Airflow Summit. Personally, I would wait with voting for a week or two after the conference to give maintainers time to catch up. Regards, Przemek On 2026/08/19 06:45:30 Piyush Maheshwari wrote: > Hi everyone > Gentle reminder to review the AIP > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning > . > > Thank you. > > On Thu, 13 Aug 2026 at 4:05 PM, Piyush Maheshwari < > [email protected]> wrote: > > > Hi everyone, > > I'd like to request the community's review of the AIP > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning > > . > > If there are no differing opinions I'd like to take it to a vote soon. > > > > Also, thank you Ephraim for the re-review. I've addressed all comments > > from the latest review and updated the AIP accordingly. > > > > Thanks and regards, > > Piyush > > > > On Fri, Aug 7, 2026 at 4:10 PM Ephraim Anierobi < > > [email protected]> wrote: > > > >> Hi Piyush, > >> > >> Thank you for this summary. It is accurate, and it matches where we > >> landed. > >> Thank you also for the many rounds of changes. The document is much > >> clearer > >> now than when we started. > >> > >> I want to state my position plainly for the thread. My main concern was > >> that a pin on a DagVersion could not hold, because the row could change > >> under it and the bundle pointer could move. The pin guard fixes both. With > >> that, plus the retention rule, the stale DAG change, the purity scope, and > >> the divergence warning, I no longer object to the pin target. The AIP now > >> says what it delivers and what it does not. That was my real goal. > >> > >> One thing I want to be clear about. My review was about whether the design > >> holds, not about whether the feature is worth building. Those are two > >> different questions, and the second one belongs to the community. > >> > >> The feature adds pin awareness in several core places: the version write > >> path, db clean, the scheduler's eligibility query for import errors and > >> stale DAGs, DagModel and asset field resolution in the DAG processor, and > >> a > >> synchronous update when a pin changes. Each change is small on its own. > >> Together they mean these paths must consider pins from now on. > >> > >> What it gives back is per-DAG control over when a version takes effect, > >> for > >> pure DAGs that come from versioned bundles. That is real and useful for > >> the > >> teams who need it. Whether it is worth the added complexity in core paths > >> is a judgment call, and voters should make it with both sides stated. I am > >> not voting against it. I only do not want my resolved concerns to be read > >> as a view on that question. > >> > >> I have left my remaining comments on the document itself. They are small > >> and none of them changes the design, but I think they should be settled > >> before the vote. > >> > >> On the vote, I agree with starting it again instead of continuing the old > >> thread. The document has changed a lot. A fresh call, with a short note on > >> what changed, would be fair to everyone who already voted. > >> > >> Thanks again for your patience through all of this. The proposal is in > >> much > >> better shape. > >> > >> Regards, > >> Ephraim > >> > >> On Wed, 5 Aug 2026 at 10:55, Piyush Maheshwari < > >> [email protected]> > >> wrote: > >> > >> > Hi everyone, > >> > > >> > I'd like to thank Ephraim for the depth of this review. Following the > >> > previous email, we had a series of detailed follow-up discussions, and I > >> > believe we have converged. This email summarizes where we landed, > >> answers > >> > the concerns from the previous email for the benefit of the thread, and > >> > describes the updates I have now made to the AIP. I would like the > >> > community's review of the updated AIP ( > >> > > >> > > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > >> > ) > >> > > >> > == Precise scope of the proposal == > >> > > >> > The discussion sharpened what the AIP promises, and it now states this > >> > explicitly: > >> > > >> > - The goal is deterministic, per-DAG control over when a new version of > >> a > >> > DAG takes effect, plus an instant, API-driven rollback to a prior > >> version, > >> > without affecting other DAGs sharing the same bundle. > >> > > >> > - A pin fixes the serialized DAG the scheduler enforces (task set, > >> edges, > >> > schedule, display fields) and the bundle (name and version) the worker > >> > checks out. It provides graph and code-level determinism, not > >> > behavior-level determinism, since a task that reads live external state > >> at > >> > execution time still does so under a pin. > >> > > >> > - Pinning guarantees are stated for DAGs that are "pure", meaning a > >> given > >> > bundle (name and version) always produces the same serialized DAG. The > >> AIP > >> > now defines this purity condition up front. > >> > > >> > - For impure DAGs, different pinning guarantees are now documented for > >> > impure DAGs. A divergence between the pinned serialized DAG's snapshot > >> and > >> > the worker's fresh parse for execution-time fields is silent. The AIP > >> now > >> > explicitly documents this as "silent divergence," a limitation, and adds > >> > detection for it. > >> > > >> > == Answers to the five concerns from the previous email == > >> > > >> > 1. "The latest DagVersion is mutable, so a pin does not freeze content." > >> > Addressed by introducing a pin guard in the DAG Processor's > >> > version-recording path. A pinned DagVersion is treated as referenced, so > >> > its serialized content is never overwritten in place; a content change > >> > creates a new DagVersion instead (the same path that already runs when > >> task > >> > instances reference the row). Note that the DAG processor only mutates > >> the > >> > latest DagVersion row, so a pin on an older version is already immutable > >> > today; the guard changes behavior only when the pinned version is also > >> the > >> > latest. > >> > > >> > 2. "A pin would not pin the source code, because bundle_version is > >> > refreshed in place on hash-identical parses." > >> > Addressed by the same guard, which stops the in-place > >> > bundle_version/version_data refresh from applying to a pinned row. When > >> the > >> > bundle advances without a serialized-DAG change while the pinned > >> version is > >> > the latest, the processor instead creates one new DagVersion carrying > >> the > >> > new bundle_version, which becomes the unpinned latest; subsequent > >> no-change > >> > bundle advances keep updating that new latest in place exactly as today. > >> > This preserves the intent of the in-place refresh (the DAG's latest > >> version > >> > keeps tracking the bundle) while the pinned version's bundle reference > >> > stays frozen. Since new DagRuns will also carry the pinned bundle, and > >> > workers resolve their bundle checkout from the DagRun, the scheduler's > >> view > >> > and the worker's code stay on the same version. Note that this version > >> > creation, without a hash change, occurs only per pin advance, not once > >> per > >> > commit. > >> > > >> > 3. "Version identity is often accidental (impure DAGs)." > >> > Addressed by scoping (guarantees stated for pure DAGs, with the purity > >> > condition defined in the AIP) and by divergence detection, where a > >> > DagWarning is raised when a pinned DAG's fresh parse hash no longer > >> matches > >> > the pinned DagVersion's hash, naming the pinned version. The check is a > >> > no-op for pure DAGs. > >> > > >> > 4. "Cleanup can delete a pinned version and silently unpin via ON DELETE > >> > SET NULL." > >> > Addressed: airflow db clean will retain any DagVersion referenced by an > >> > active pin, the same way it retains task-instance-referenced versions. > >> > > >> > 5. "The deletion rule works against the main use case." > >> > Revised: a dag_id that disappears from the latest parse would still be > >> > marked stale on DagModel as today, but the scheduler's eligibility query > >> > will admit a stale DAG while it has an active pin, so it keeps running > >> on > >> > its pinned version (parallel to the import-error exception already in > >> the > >> > AIP). Actual removal requires an explicit unpin first, after which the > >> > normal stale-to-deleted path applies unchanged. This turns the defect > >> > scenario from a silent removal into an explicit act. > >> > > >> > == On pinning the bundle instead of the DagVersion == > >> > > >> > We discussed this alternative at length, in two variants, and the AIP's > >> > "Alternatives" section now covers both: > >> > > >> > - A global bundle pin (keep a history of bundle versions and freeze the > >> > whole bundle on one) is simple and atomic, but it pins every DAG in the > >> > bundle at once. It cannot keep a subset of DAGs on an older version > >> while > >> > the rest advance, and per-DAG control is the primary use case, since a > >> > breaking change in a shared bundle typically affects only some of the > >> DAGs > >> > it emits. We see it as a potentially complementary, coarser-grained > >> control > >> > rather than a replacement. > >> > > >> > - A per-DAG pin keyed by the bundle (name and version) is not uniquely > >> > resolvable to a serialized DAG for impure DAGs, since the pair can map > >> to > >> > several dag_version rows. It therefore cannot be expressed as a foreign > >> key > >> > to a specific version, and the latest matching row can resolve > >> differently > >> > across invocations, which is behavior drift while supposedly pinned. It > >> is > >> > also unstable over time, because a dag_version's bundle_version is > >> > refreshed in place on hash-identical parses, so a previously valid > >> bundle > >> > version can stop resolving exactly when a rollback needs it. Freezing > >> that > >> > would require an immutable bundle-to-dag_version mapping, which amounts > >> to > >> > pinning the dag_version too and converges towards this proposal. > >> > > >> > == Operating models and rollback availability == > >> > > >> > Ephraim raised an important point about rollback availability. An > >> on-demand > >> > pin can only roll back to versions Airflow already recorded, so a change > >> > that never altered the serialized DAG (e.g. one confined to an imported > >> > helper module) has no distinct version to return to unless it was > >> pinned at > >> > the time. The AIP now states this honestly under a new "Usage Models and > >> > Rollback Availability" section, which describes two models the primitive > >> > serves: > >> > > >> > 1. On-demand pinning (incident response, planned cutovers): rollback > >> > targets are the recorded DAG versions. Changes that altered the > >> serialized > >> > DAG (and had task instances) are reachable; changes that did not are a > >> > stated limitation. > >> > 2. Standing pins driven by an external release orchestrator: every DAG > >> > always carries a pin and the orchestrator advances it per deployment. > >> Every > >> > pinned state remains an immutable rollback target, closing the gap > >> above, > >> > at the cost of operating the orchestrator. The consequences of the same > >> are > >> > also covered in the AIP. > >> > > >> > Airflow itself only ships the pin/unpin primitive; neither model is > >> > mandatory to use this feature. > >> > > >> > == Updated AIP and proposed timeline == > >> > > >> > I have updated AIP-109 with all of the above: > >> > > >> > > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > >> > > >> > I would appreciate the community's review of the updated proposal. If > >> > multiple maintainers review it and if no major open questions or > >> comments > >> > remain by Wednesday, August 12, I plan to restart the vote phase. > >> > > >> > Thanks again to Ephraim for the multiple rounds of thorough review that > >> got > >> > the proposal to this shape. > >> > > >> > Regards, > >> > Piyush > >> > > >> > On Tue, Jul 28, 2026 at 3:47 PM Ephraim Anierobi < > >> > [email protected]> > >> > wrote: > >> > > >> > > Thank you for this proposal, and for the careful work behind it. I > >> took a > >> > > second, deeper look at the document, and I want to share my concerns, > >> > > because I believe they change the direction rather than the details. > >> > > > >> > > First, I want to recognize the real problem the AIP identifies. The > >> parse > >> > > loop controls when new code becomes active, and that timing is not > >> > > deterministic. Teams need a way to control activation time, and they > >> > need a > >> > > fast rollback path. This is a genuine gap, and I am glad the AIP puts > >> a > >> > > spotlight on it. > >> > > > >> > > After the second review, however, I believe the proposal rests on a > >> > > misunderstanding of what a DagVersion is. A DagVersion is a historical > >> > > record of what Airflow observed after a parse. It is an output of the > >> > > system, not an input to it. Dag versioning exists so that running and > >> > past > >> > > task instances keep a faithful record of the code they ran against. > >> > *Bundle > >> > > versioning* is the mechanism that decides which source code a run > >> uses. > >> > The > >> > > AIP reads DagVersion as a release artifact that users can select and > >> > > activate, and the current implementation does not support that > >> reading: > >> > > > >> > > 1. *The latest DagVersion is mutable*. When no task instance > >> references > >> > it, > >> > > write_dag() replaces its serialized content, hash, and Dag code in > >> place, > >> > > on the same version ID. A pin does not create a task instance > >> reference. > >> > So > >> > > a user could pin the current version before a risky merge, and the > >> next > >> > > parse could replace the content behind that pin. The pin would not > >> give > >> > the > >> > > determinism it promises. > >> > > > >> > > 2. *A pin on the current DagVersion would not pin the source code*. > >> When > >> > a > >> > > new commit does not change the serialized content, `write_dag()` > >> updates > >> > > bundle_version on the latest DagVersion in place. A change to a helper > >> > > module or an SQL file does not change the serialized hash, so workers > >> > would > >> > > run the new commit under the pinned version. > >> > > > >> > > 3. *Version identity is often accidental.* Parse-time variability and > >> > > serializer defects can create versions that no one intended. Airflow > >> > ships > >> > > an inflation checker and a dedicated Dag warning type because this > >> > problem > >> > > is real and common. Dag factories at large scale, the AIP's own key > >> > > example, are the most exposed. A pin would often select an accidental > >> > > snapshot rather than a deliberate release. > >> > > > >> > > 4. *The proposal does not address cleanup*. Today, `airflow db clean` > >> > keeps > >> > > the latest version per Dag and skips only versions that task instances > >> > > reference. An older pinned version with no task instances is eligible > >> for > >> > > deletion, and the proposed ON DELETE SET NULL behavior would then > >> > silently > >> > > return the Dag to the latest version, exactly when the pin matters > >> most. > >> > > The document does not cover this interaction. > >> > > > >> > > 5. *The deletion rule works against the main use case*. The AIP states > >> > that > >> > > removal from a new bundle version deletes the Dag, even when it is > >> > pinned. > >> > > A defect in a dag-factory file that changes or drops Dag IDs removes > >> Dags > >> > > from the parse results. In that case, Airflow would delete the Dag and > >> > > discard the pin, so the protection would be absent in one of the most > >> > > common failure modes the AIP is motivated by. > >> > > > >> > > These are not implementation gaps that more safeguards can close. They > >> > all > >> > > follow from the same source: the design asks a history table to act > >> as a > >> > > deployment mechanism. Making that work would mean redefining what > >> > > DagVersion is, and I do not think we should. > >> > > > >> > > The goal itself is worth pursuing at the right layer. Deterministic > >> > cutover > >> > > and instant rollback are operations on the *source artifact*. Bundle > >> > > versions already represent the source, and pinning or rolling back at > >> the > >> > > bundle layer would be atomic for all Dags and all support files at > >> once. > >> > I > >> > > would be glad to help explore that direction. > >> > > > >> > > Thanks again for raising this discussion. The problem is real, and I > >> hope > >> > > we can solve it together on a foundation that supports it. > >> > > > >> > > - Ephraim > >> > > > >> > > On Mon, 27 Jul 2026 at 06:53, Piyush Maheshwari < > >> > > [email protected]> > >> > > wrote: > >> > > > >> > > > Thank you, Sumit. > >> > > > > >> > > > I've started a voting thread as suggested: > >> > > > https://lists.apache.org/thread/bff758c5p68v1mtzhflytn7yzv0d8l1t > >> > > > > >> > > > Regards, > >> > > > Piyush > >> > > > > >> > > > On Fri, Jul 24, 2026 at 12:08 PM Sumit Maheshwari < > >> > > [email protected]> > >> > > > wrote: > >> > > > > >> > > > > I've reviewed it again today and left some comments. Overall, it > >> > looks > >> > > > > great and would be a great feature add in Airflow. I think you can > >> > > start > >> > > > an > >> > > > > official voting thread on this AIP. > >> > > > > > >> > > > > > >> > > > > On Sun, Jul 19, 2026 at 7:32 PM Piyush Maheshwari < > >> > > > > [email protected]> wrote: > >> > > > > > >> > > > > > Hi everyone, > >> > > > > > Thanks for the reviews on AIP-109 ( > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > >> > > > > > ). > >> > > > > > Over the past few weeks, I implemented the feature in our fork > >> > > (nearly > >> > > > > > done). > >> > > > > > Based on the insights and learnings from this process, I have > >> > further > >> > > > > > refined the AIP while addressing all comments. > >> > > > > > > >> > > > > > Please review the updated version. I'll wait to collect some > >> > feedback > >> > > > and > >> > > > > > then take it to a vote. > >> > > > > > I can start contributing PRs as soon as the AIP is approved. > >> > > > > > > >> > > > > > Thanks and regards, > >> > > > > > Piyush > >> > > > > > > >> > > > > > On Wed, Jun 3, 2026 at 2:01 PM Ephraim Anierobi < > >> > > > > [email protected] > >> > > > > > > > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hi Piyush, > >> > > > > > > > >> > > > > > > Thanks for the AIP. I’ve added a few comments to the doc. > >> Please > >> > > > take a > >> > > > > > > look, as I think we should iron out those areas and fully > >> > > understand > >> > > > > the > >> > > > > > > implications before moving forward. > >> > > > > > > > >> > > > > > > Regards > >> > > > > > > - Ephraim > >> > > > > > > > >> > > > > > > On Tue, 26 May 2026 at 18:24, Przemysław Mirowski < > >> > > [email protected] > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > I checked the API - +1. Thanks for writing this up! > >> > > > > > > > > >> > > > > > > > Thanks Nathan for mentioning the #63884 PR. It is nice > >> addition > >> > > > > > (already > >> > > > > > > > merged), and I think that it will be really useful for users > >> > who > >> > > > got > >> > > > > > used > >> > > > > > > > to Airflow 2 retry mechanism. I think also that your PR and > >> > > API-109 > >> > > > > are > >> > > > > > > > complementary as one is focusing on the versioning behaviour > >> > for > >> > > > > > retrying > >> > > > > > > > task, the latter is focusing on versioning behaviour for new > >> > Dag > >> > > > > Runs. > >> > > > > > > > ________________________________ > >> > > > > > > > From: Christos Bisias <[email protected]> > >> > > > > > > > Sent: 25 May 2026 09:46 > >> > > > > > > > To: [email protected] <[email protected]> > >> > > > > > > > Subject: Re: [DISCUSS] DAG Version Pinning for Deployment > >> > Gating > >> > > > > > > (Building > >> > > > > > > > on AIP-63) > >> > > > > > > > > >> > > > > > > > Hello, > >> > > > > > > > > >> > > > > > > > I'm a little late on the discussion but I just came across > >> the > >> > > AIP > >> > > > > and > >> > > > > > I > >> > > > > > > > like this idea. I've actually been thinking of working on > >> > > something > >> > > > > > > > similar, to allow people to handle a bad rollout by > >> reverting > >> > to > >> > > > old > >> > > > > > code > >> > > > > > > > for that run without a full slow release pipeline. And this > >> > > covers > >> > > > > it. > >> > > > > > > > > >> > > > > > > > In my opinion, this seems more like a natural step towards > >> what > >> > > dag > >> > > > > > > > versioning is supposed to do, than a new feature. > >> > > > > > > > > >> > > > > > > > Thank you, > >> > > > > > > > Christos > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Mon, May 25, 2026 at 8:15 AM Piyush Maheshwari < > >> > > > > > > > [email protected]> wrote: > >> > > > > > > > > >> > > > > > > > > Hi everyone, > >> > > > > > > > > I wanted to send a gentle nudge to review AIP-109: DAG > >> > Version > >> > > > > > Pinning > >> > > > > > > ( > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > >> > > > > > > > > ). > >> > > > > > > > > If there are no major concerns, I would like to take this > >> to > >> > a > >> > > > vote > >> > > > > > > soon. > >> > > > > > > > > > >> > > > > > > > > Thanks, > >> > > > > > > > > Piyush > >> > > > > > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 7:44 AM Piyush Maheshwari < > >> > > > > > > > > [email protected]> wrote: > >> > > > > > > > > > >> > > > > > > > > > Thanks for the note, Sumit. Based on the feedback, I've > >> > > drafted > >> > > > > an > >> > > > > > > AIP > >> > > > > > > > > > that is now up for review. > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > >> > > > > > > > > > > >> > > > > > > > > > Would like to get the community's feedback on the same. > >> > > > > > > > > > > >> > > > > > > > > > Nathan, I remember seeing your work (the issue and an > >> older > >> > > PR) > >> > > > > > while > >> > > > > > > > > > reviewing all ongoing work related to DAG versions. I > >> agree > >> > > > with > >> > > > > > the > >> > > > > > > > PR's > >> > > > > > > > > > intent, although I haven't reviewed it yet. > >> > > > > > > > > > I understand your PR makes the version-pinned execution > >> > > > behavior > >> > > > > of > >> > > > > > > > > reruns > >> > > > > > > > > > and backfills configurable. > >> > > > > > > > > > However, this discussion revolves around the behavior > >> for > >> > new > >> > > > > runs > >> > > > > > > > only. > >> > > > > > > > > > We need the capability to pin a DAG to a specific > >> version > >> > for > >> > > > > > future > >> > > > > > > > runs > >> > > > > > > > > > instead. > >> > > > > > > > > > Hope that clarifies. > >> > > > > > > > > > > >> > > > > > > > > > Regards, > >> > > > > > > > > > Piyush > >> > > > > > > > > > > >> > > > > > > > > > On Thu, May 14, 2026 at 5:11 PM Nathan Hadfield < > >> > > > > > > > > [email protected]> > >> > > > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > >> Hello, > >> > > > > > > > > >> > >> > > > > > > > > >> I saw AIP-109 that was created in relation to this > >> > > discussion > >> > > > > and > >> > > > > > > > > thought > >> > > > > > > > > >> I’d better mention this PR that I’ve been working on > >> for a > >> > > > while > >> > > > > > and > >> > > > > > > > is > >> > > > > > > > > >> close to being approved. > >> > > > > > > > > >> > >> > > > > > > > > >> https://github.com/apache/airflow/pull/63884 > >> > > > > > > > > >> > >> > > > > > > > > >> It is very much related to the motivations described > >> and > >> > > > > > implements > >> > > > > > > > the > >> > > > > > > > > >> desire for control over the behaviour when > >> > > > clearing/backfilling > >> > > > > > > runs. > >> > > > > > > > > >> > >> > > > > > > > > >> Happy to discuss best steps for this here or on the PR. > >> > > > > > > > > >> > >> > > > > > > > > >> Cheers, > >> > > > > > > > > >> > >> > > > > > > > > >> Nathan > >> > > > > > > > > >> > >> > > > > > > > > >> From: Przemysław Mirowski <[email protected]> > >> > > > > > > > > >> Date: Tuesday, 28 April 2026 at 21:54 > >> > > > > > > > > >> To: [email protected] <[email protected]> > >> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for > >> Deployment > >> > > > Gating > >> > > > > > > > > >> (Building on AIP-63) > >> > > > > > > > > >> > >> > > > > > > > > >> This Message Is From an External Sender > >> > > > > > > > > >> This message came from outside your organization. > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > > >> > P.S. In my opinion, what can be done in/around git, > >> > should > >> > > > be > >> > > > > > done > >> > > > > > > > > >> there. Recreation of CI/CD in any form inside of > >> Airflow > >> > > > itself > >> > > > > is > >> > > > > > > > > >> something which should not be done. > >> > > > > > > > > >> > I'm glad we agree on this :) I suppose we just > >> disagree > >> > on > >> > > > > what > >> > > > > > is > >> > > > > > > > > >> possible outside of Airflow :p > >> > > > > > > > > >> > >> > > > > > > > > >> I think that we just disagree on what the issue is, > >> not on > >> > > > what > >> > > > > is > >> > > > > > > > > >> possible/should be outside of Airflow. > >> > > > > > > > > >> > >> > > > > > > > > >> > I think we are trying to duplicate what we already > >> have > >> > in > >> > > > > Git. > >> > > > > > > > > >> > >> > > > > > > > > >> Not really if we are only referring to version > >> pinning. As > >> > > far > >> > > > > as > >> > > > > > I > >> > > > > > > am > >> > > > > > > > > >> aware of how things are working, there is no > >> possibility > >> > to > >> > > > > > > determine > >> > > > > > > > > that > >> > > > > > > > > >> Dag after e.g. deployment on 1:00:00 PM will be exactly > >> > > parsed > >> > > > > and > >> > > > > > > > used > >> > > > > > > > > >> since 1:01:00 PM forward. Basically, what version > >> pinning > >> > > > would > >> > > > > > > > provide > >> > > > > > > > > is > >> > > > > > > > > >> the full control of the time since the given version > >> will > >> > be > >> > > > > used > >> > > > > > > > > >> (currently we can only have more-or-less timing which > >> in > >> > > some > >> > > > > > cases, > >> > > > > > > > is > >> > > > > > > > > not > >> > > > > > > > > >> sufficient). The "quick revert" is the consequence of > >> > having > >> > > > > above > >> > > > > > > > > >> possibility. > >> > > > > > > > > >> > >> > > > > > > > > >> Looking at the general concerns, with having that > >> feature > >> > or > >> > > > > not, > >> > > > > > > > users > >> > > > > > > > > >> can pretty easily test things on production, but it > >> just > >> > > > > requires > >> > > > > > > more > >> > > > > > > > > time > >> > > > > > > > > >> between iterations without it. IMHO it will not change > >> the > >> > > > need > >> > > > > > for > >> > > > > > > > > >> Airflow-related platform teams which makes sure, by > >> > > > > > > > standards/policies, > >> > > > > > > > > >> that things are properly tested before production > >> > > deployment. > >> > > > I > >> > > > > > > think > >> > > > > > > > > that > >> > > > > > > > > >> assumption that some users will misuse this feature is > >> > true > >> > > > > (like > >> > > > > > > with > >> > > > > > > > > most > >> > > > > > > > > >> of the features really), but on the other hand it would > >> > > > provide > >> > > > > > more > >> > > > > > > > > >> control for other users. The other solution possibly > >> would > >> > > be > >> > > > to > >> > > > > > > make > >> > > > > > > > > Dag > >> > > > > > > > > >> Processor work more on "events" instead of "simple" > >> > parsing > >> > > > loop > >> > > > > > (I > >> > > > > > > > > recall > >> > > > > > > > > >> that there was some PR couple years ago with PoC of > >> that, > >> > > but > >> > > > I > >> > > > > > > > couldn't > >> > > > > > > > > >> quickly find it). > >> > > > > > > > > >> > >> > > > > > > > > >> ________________________________ > >> > > > > > > > > >> From: Jarek Potiuk <[email protected]> > >> > > > > > > > > >> Sent: 28 April 2026 17:02 > >> > > > > > > > > >> To: [email protected] <[email protected]> > >> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for > >> Deployment > >> > > > Gating > >> > > > > > > > > >> (Building on AIP-63) > >> > > > > > > > > >> > >> > > > > > > > > >> Same concerns. I think we are trying to duplicate what > >> we > >> > > > > already > >> > > > > > > have > >> > > > > > > > > in > >> > > > > > > > > >> Git—branches and reverts, for example—by moving what > >> > should > >> > > be > >> > > > > > > managed > >> > > > > > > > > as > >> > > > > > > > > >> part of the development process to Airflow UI. > >> > > > > > > > > >> > >> > > > > > > > > >> Almost everything you describe can be done with: > >> > > > > > > > > >> > >> > > > > > > > > >> * having a dev/staging system configured properly to > >> use > >> > > > > > dev/staging > >> > > > > > > > > >> branches > >> > > > > > > > > >> * Having a process of managing development and a proper > >> > > > > branching > >> > > > > > > > > strategy > >> > > > > > > > > >> * single git command (for example, `git revert XXXX` > >> > > followed > >> > > > by > >> > > > > > > push > >> > > > > > > > to > >> > > > > > > > > >> the right branch) > >> > > > > > > > > >> > >> > > > > > > > > >> J. > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > > >> On Tue, Apr 28, 2026 at 10:34 AM Pierre Jeambrun < > >> > > > > > > > [email protected] > >> > > > > > > > > > > >> > > > > > > > > >> wrote: > >> > > > > > > > > >> > >> > > > > > > > > >> > At first glance I tend to agree with Jens and Niko. > >> > > > > > > > > >> > > >> > > > > > > > > >> > I understand the request, but I agree that this > >> resolves > >> > > > CI/CD > >> > > > > > and > >> > > > > > > > > >> testing > >> > > > > > > > > >> > issues that should probably be remain outside > >> Airflow. > >> > > > > > > > > >> > > >> > > > > > > > > >> > On Mon, Apr 27, 2026 at 7:43 PM Oliveira, Niko < > >> > > > > > > [email protected] > >> > > > > > > > > > >> > > > > > > > > >> > wrote: > >> > > > > > > > > >> > > >> > > > > > > > > >> > > Hey folks! > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > P.S. In my opinion, what can be done in/around > >> git, > >> > > > should > >> > > > > > be > >> > > > > > > > done > >> > > > > > > > > >> > > there. Recreation of CI/CD in any form inside of > >> > Airflow > >> > > > > > itself > >> > > > > > > is > >> > > > > > > > > >> > > something which should not be done. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > I'm glad we agree on this :) I suppose we just > >> > disagree > >> > > on > >> > > > > > what > >> > > > > > > is > >> > > > > > > > > >> > > possible outside of Airflow :p > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > But at this point I will bow out of the > >> conversation > >> > and > >> > > > let > >> > > > > > > > others > >> > > > > > > > > >> weigh > >> > > > > > > > > >> > > in. I'm not fully convinced any of these requested > >> > > > > behaviours > >> > > > > > > > > require > >> > > > > > > > > >> > > changes to Airflow (I think that's just masking > >> some > >> > dev > >> > > > ops > >> > > > > > > > work). > >> > > > > > > > > >> But > >> > > > > > > > > >> > > also I'm not completely opposed to the change > >> either, > >> > > I'm > >> > > > > more > >> > > > > > > on > >> > > > > > > > > the > >> > > > > > > > > >> > > fence, so if others love the feature by all means > >> > > > implement > >> > > > > > it! > >> > > > > > > :) > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Cheers, > >> > > > > > > > > >> > > Niko > >> > > > > > > > > >> > > ________________________________ > >> > > > > > > > > >> > > From: Przemysław Mirowski <[email protected]> > >> > > > > > > > > >> > > Sent: Thursday, April 23, 2026 3:06 PM > >> > > > > > > > > >> > > To: [email protected] <[email protected] > >> > > >> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning > >> for > >> > > > > > Deployment > >> > > > > > > > > Gating > >> > > > > > > > > >> > > (Building on AIP-63) > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > CAUTION: This email originated from outside of the > >> > > > > > organization. > >> > > > > > > > Do > >> > > > > > > > > >> not > >> > > > > > > > > >> > > click links or open attachments unless you can > >> confirm > >> > > the > >> > > > > > > sender > >> > > > > > > > > and > >> > > > > > > > > >> > know > >> > > > > > > > > >> > > the content is safe. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient > >> d’un > >> > > > > > expéditeur > >> > > > > > > > > >> externe. > >> > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce > >> > > jointe > >> > > > si > >> > > > > > > vous > >> > > > > > > > ne > >> > > > > > > > > >> > pouvez > >> > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous > >> > > n’êtes > >> > > > > pas > >> > > > > > > > > certain > >> > > > > > > > > >> > que > >> > > > > > > > > >> > > le contenu ne présente aucun risque. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Hi, > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > I think that CI/CD and version pining are a little > >> two > >> > > > > > different > >> > > > > > > > > >> things > >> > > > > > > > > >> > > here. In a use cases with some critical systems > >> > > involved, > >> > > > > the > >> > > > > > > > > >> situation > >> > > > > > > > > >> > > when the Dag changes the version to the latest > >> without > >> > > > > > > possibility > >> > > > > > > > > to > >> > > > > > > > > >> > > determine when it will exactly happen (CI/CD will > >> have > >> > > > some > >> > > > > > > > > >> more-or-less > >> > > > > > > > > >> > > time to deploy the change, the same goes for Dag > >> > > Processor > >> > > > > > > parsing > >> > > > > > > > > >> time) > >> > > > > > > > > >> > is > >> > > > > > > > > >> > > rather hard to do and in some systems it can make > >> > change > >> > > > > > > > deployment > >> > > > > > > > > >> > harder > >> > > > > > > > > >> > > and less safe. Of course, the ideal solution would > >> be > >> > to > >> > > > > have > >> > > > > > > > proper > >> > > > > > > > > >> > > non-prod environment, which is fully > >> representative in > >> > > > > > > comparison > >> > > > > > > > to > >> > > > > > > > > >> > > production (in some cases exposing non-prod to prod > >> > > > > > > > > data/traffic/etc. > >> > > > > > > > > >> is, > >> > > > > > > > > >> > > just, not an option - e.g. security), but it is not > >> > > always > >> > > > > > > > possible > >> > > > > > > > > >> to do > >> > > > > > > > > >> > > due to various reasons like costs, licenses, space > >> > > and/or > >> > > > > > > vendors. > >> > > > > > > > > I'm > >> > > > > > > > > >> > > agreeing especially with point 5 of Piyush latest > >> > > message. > >> > > > > > > Having > >> > > > > > > > > >> above > >> > > > > > > > > >> > in > >> > > > > > > > > >> > > mind, I think that version pinning would be a nice > >> > > > addition > >> > > > > to > >> > > > > > > the > >> > > > > > > > > Dag > >> > > > > > > > > >> > > Versioning feature with an assumption that it is > >> for > >> > > > > critical > >> > > > > > > > > Airflow > >> > > > > > > > > >> > Dags > >> > > > > > > > > >> > > when full control of the Dags version change time > >> is > >> > > > > required > >> > > > > > > > (maybe > >> > > > > > > > > >> > there > >> > > > > > > > > >> > > is also another way to achieve that). > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > P.S. In my opinion, what can be done in/around git, > >> > > should > >> > > > > be > >> > > > > > > done > >> > > > > > > > > >> there. > >> > > > > > > > > >> > > Recreation of CI/CD in any form inside of Airflow > >> > itself > >> > > > is > >> > > > > > > > > something > >> > > > > > > > > >> > which > >> > > > > > > > > >> > > should not be done. > >> > > > > > > > > >> > > ________________________________ > >> > > > > > > > > >> > > From: Oliveira, Niko <[email protected]> > >> > > > > > > > > >> > > Sent: 23 April 2026 01:50 > >> > > > > > > > > >> > > To: [email protected] <[email protected] > >> > > >> > > > > > > > > >> > > Subject: Re: [DISCUSS] DAG Version Pinning for > >> > > Deployment > >> > > > > > Gating > >> > > > > > > > > >> > (Building > >> > > > > > > > > >> > > on AIP-63) > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Hey Piyush, > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Thanks for your reply, I do love how clearly it is > >> > > written > >> > > > > > and I > >> > > > > > > > see > >> > > > > > > > > >> > > exactly the problem you're trying to solve! > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > I'm still just not convinced this needs to be done > >> in > >> > > > > Airflow, > >> > > > > > > at > >> > > > > > > > > >> least > >> > > > > > > > > >> > > not with a first class feature. As interesting as I > >> > > think > >> > > > > your > >> > > > > > > > > >> > microservice > >> > > > > > > > > >> > > analogy is, Airflow is not a microservice > >> component, > >> > it > >> > > > is a > >> > > > > > > > (very, > >> > > > > > > > > >> very) > >> > > > > > > > > >> > > fancy cron scheduler. And I'm not sure the > >> complexity > >> > is > >> > > > > worth > >> > > > > > > the > >> > > > > > > > > use > >> > > > > > > > > >> > > case. Since any new code added to Airflow must be > >> > > > maintained > >> > > > > > by > >> > > > > > > > this > >> > > > > > > > > >> > > community and we must be cautious that any new > >> pieces > >> > > > serves > >> > > > > > > > enough > >> > > > > > > > > >> use > >> > > > > > > > > >> > > cases/users to make it worth it. > >> > > > > > > > > >> > > To me this should either be managed outside of an > >> > > > individual > >> > > > > > > > Airflow > >> > > > > > > > > >> > > environment e.g. you have an entirely separate > >> > > > > > staging/gamma/dev > >> > > > > > > > > >> Airflow > >> > > > > > > > > >> > > environment, which is exposed to some level of > >> > > production > >> > > > > > > traffic > >> > > > > > > > > (to > >> > > > > > > > > >> > > borrow your microservice analogy) until it can > >> > graduate > >> > > to > >> > > > > the > >> > > > > > > > > >> production > >> > > > > > > > > >> > > environment. And if you really need on the fly > >> > toggling > >> > > > of a > >> > > > > > > > > version, > >> > > > > > > > > >> as > >> > > > > > > > > >> > > you say, Airflow does this quite responsively, if > >> you > >> > > > > deploy a > >> > > > > > > new > >> > > > > > > > > >> > version > >> > > > > > > > > >> > > of your dags it will parse and start using that new > >> > > > version > >> > > > > > > > > >> immediately > >> > > > > > > > > >> > > (the problem you're trying to solve can be a > >> benefit > >> > > > here). > >> > > > > > You > >> > > > > > > > can > >> > > > > > > > > >> even > >> > > > > > > > > >> > > have multiple versions of your dags deployed at > >> once > >> > and > >> > > > use > >> > > > > > > > > >> > configuration > >> > > > > > > > > >> > > to control which dag directory Airflow reads from > >> (or > >> > > > > > > move/symlink > >> > > > > > > > > >> Dags > >> > > > > > > > > >> > in > >> > > > > > > > > >> > > and out of the Dags directory as needed from a > >> known > >> > > good > >> > > > or > >> > > > > > > > pinned > >> > > > > > > > > >> > > source). Or use variables or some other parameter > >> > store > >> > > to > >> > > > > > > control > >> > > > > > > > > >> other > >> > > > > > > > > >> > > pieces of runtime behaviour inside the Dags > >> > themselves. > >> > > > > > Between > >> > > > > > > > > CI/CD, > >> > > > > > > > > >> > dev > >> > > > > > > > > >> > > ops and making use of existing Airflow primitives I > >> > > think > >> > > > > you > >> > > > > > > can > >> > > > > > > > > >> achieve > >> > > > > > > > > >> > > what you're looking for. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > But as always, this is open and community based > >> > > software, > >> > > > so > >> > > > > > I'm > >> > > > > > > > > >> happy to > >> > > > > > > > > >> > > disagree and commit if the rest of the community > >> > thinks > >> > > > this > >> > > > > > is > >> > > > > > > a > >> > > > > > > > > >> > valuable > >> > > > > > > > > >> > > feature :) > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Cheers, > >> > > > > > > > > >> > > Niko > >> > > > > > > > > >> > > ________________________________ > >> > > > > > > > > >> > > From: Piyush Maheshwari <[email protected]> > >> > > > > > > > > >> > > Sent: Tuesday, April 21, 2026 10:46 PM > >> > > > > > > > > >> > > To: [email protected] <[email protected] > >> > > >> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning > >> for > >> > > > > > Deployment > >> > > > > > > > > Gating > >> > > > > > > > > >> > > (Building on AIP-63) > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > CAUTION: This email originated from outside of the > >> > > > > > organization. > >> > > > > > > > Do > >> > > > > > > > > >> not > >> > > > > > > > > >> > > click links or open attachments unless you can > >> confirm > >> > > the > >> > > > > > > sender > >> > > > > > > > > and > >> > > > > > > > > >> > know > >> > > > > > > > > >> > > the content is safe. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient > >> d’un > >> > > > > > expéditeur > >> > > > > > > > > >> externe. > >> > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce > >> > > jointe > >> > > > si > >> > > > > > > vous > >> > > > > > > > ne > >> > > > > > > > > >> > pouvez > >> > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous > >> > > n’êtes > >> > > > > pas > >> > > > > > > > > certain > >> > > > > > > > > >> > que > >> > > > > > > > > >> > > le contenu ne présente aucun risque. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Hi Ephraim, Jarek, Jens, and Niko, > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Thank you for the candid feedback. I want to > >> clarify a > >> > > few > >> > > > > > > things, > >> > > > > > > > > as > >> > > > > > > > > >> I > >> > > > > > > > > >> > > completely agree with Jens and Niko that "testing > >> in > >> > > > > > production" > >> > > > > > > > is > >> > > > > > > > > an > >> > > > > > > > > >> > > anti-pattern. That is absolutely not the intention > >> > here. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > 1. I view this as bringing standard > >> microservice-like > >> > > > > > deployment > >> > > > > > > > > >> maturity > >> > > > > > > > > >> > > to DAGs. > >> > > > > > > > > >> > > Before service deployments in our org, code is > >> tested > >> > > > > locally, > >> > > > > > > in > >> > > > > > > > a > >> > > > > > > > > >> dev > >> > > > > > > > > >> > > environment, and via strict unit/e2e integration > >> tests > >> > > > > before > >> > > > > > it > >> > > > > > > > > ever > >> > > > > > > > > >> > makes > >> > > > > > > > > >> > > it to main. But even after merging and passing > >> those > >> > CI > >> > > > > > > pipelines, > >> > > > > > > > > we > >> > > > > > > > > >> > still > >> > > > > > > > > >> > > use load tests, pre-prod soak times, shadow > >> traffic, > >> > and > >> > > > > gated > >> > > > > > > > > >> production > >> > > > > > > > > >> > > rollouts with automated rollback triggers. Having > >> > > > deployment > >> > > > > > > gates > >> > > > > > > > > for > >> > > > > > > > > >> > the > >> > > > > > > > > >> > > production environment doesn't mean the pre-merge > >> > checks > >> > > > > > weren't > >> > > > > > > > > >> strict > >> > > > > > > > > >> > or > >> > > > > > > > > >> > > that the change wasn't tested beforehand -- it just > >> > > allows > >> > > > > us > >> > > > > > to > >> > > > > > > > > place > >> > > > > > > > > >> > > additional safety gates for the code to take > >> effect, > >> > > > exactly > >> > > > > > > like > >> > > > > > > > in > >> > > > > > > > > >> the > >> > > > > > > > > >> > > service world. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > 2. The core issue we are trying to solve is that > >> > Airflow > >> > > > > > > currently > >> > > > > > > > > >> > > inseparably links Code Distribution (a file > >> arriving > >> > on > >> > > > the > >> > > > > > > > > >> dag-processor > >> > > > > > > > > >> > > and being parsed) with Release Activation (the > >> > scheduler > >> > > > > > > executing > >> > > > > > > > > >> that > >> > > > > > > > > >> > > code). > >> > > > > > > > > >> > > To extend the microservices analogy, I can think of > >> > the > >> > > > DAG > >> > > > > > > > > processor > >> > > > > > > > > >> > > parsing all files as "building the artifact(s)," > >> while > >> > > the > >> > > > > > > > scheduler > >> > > > > > > > > >> and > >> > > > > > > > > >> > > executor acting on the DAG versions created > >> thereafter > >> > > as > >> > > > > > > > > "deploying" > >> > > > > > > > > >> or > >> > > > > > > > > >> > > running the changed code. > >> > > > > > > > > >> > > We simply want to decouple the build from the > >> > > deployment. > >> > > > > This > >> > > > > > > > does > >> > > > > > > > > >> not > >> > > > > > > > > >> > > mean that the code arriving on the dag-processor > >> will > >> > be > >> > > > > > tested > >> > > > > > > > for > >> > > > > > > > > >> the > >> > > > > > > > > >> > > first time straight in production. It should've > >> > already > >> > > > > > passed a > >> > > > > > > > set > >> > > > > > > > > >> of > >> > > > > > > > > >> > > checks in the CI pipeline. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > 3. It is also worth calling out that Airflow > >> already > >> > > > > supports > >> > > > > > > this > >> > > > > > > > > >> > > decoupled behavior at the run level for task > >> re-runs > >> > and > >> > > > > > > > > mid-execution > >> > > > > > > > > >> > DAG > >> > > > > > > > > >> > > version bumps (by pinning the version for the rest > >> of > >> > > the > >> > > > > > > > execution > >> > > > > > > > > or > >> > > > > > > > > >> > the > >> > > > > > > > > >> > > rerun). We are simply trying to expose this > >> existing > >> > > > > > capability > >> > > > > > > at > >> > > > > > > > > the > >> > > > > > > > > >> > DAG > >> > > > > > > > > >> > > level so users can govern which version new > >> scheduled > >> > > runs > >> > > > > are > >> > > > > > > > > created > >> > > > > > > > > >> > > with. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > 4. I also agree that Airflow itself should not be > >> > aware > >> > > of > >> > > > > our > >> > > > > > > > CI/CD > >> > > > > > > > > >> > > pipeline, nor would it manage the deployment > >> > > orchestration > >> > > > > or > >> > > > > > > > > testing. > >> > > > > > > > > >> > > For our requirements, I just need Airflow to expose > >> > APIs > >> > > > to > >> > > > > > > deploy > >> > > > > > > > > >> (pin) > >> > > > > > > > > >> > a > >> > > > > > > > > >> > > DAG version, and to remove the pin (to > >> restore/enable > >> > > the > >> > > > > > > default > >> > > > > > > > > >> > > "auto-deploy latest" behavior). > >> > > > > > > > > >> > > Beyond that, we intend to use an external release > >> > > > > orchestrator > >> > > > > > > > that > >> > > > > > > > > >> can > >> > > > > > > > > >> > > explicitly tell Airflow when a parsed version is > >> > > actually > >> > > > > > > allowed > >> > > > > > > > to > >> > > > > > > > > >> run. > >> > > > > > > > > >> > > Until that API call is made, the previously pinned > >> > > version > >> > > > > > > remains > >> > > > > > > > > >> > active. > >> > > > > > > > > >> > > This ensures we don't introduce assumptions or > >> > awareness > >> > > > of > >> > > > > > the > >> > > > > > > > > >> presence > >> > > > > > > > > >> > of > >> > > > > > > > > >> > > any external gating mechanisms to Airflow. > >> > > > > > > > > >> > > Also note that the intention is to keep the default > >> > > > > > auto-deploy > >> > > > > > > > > >> behavior > >> > > > > > > > > >> > > unless a user (or a system on their behalf) > >> explicitly > >> > > > asks > >> > > > > > > > Airflow > >> > > > > > > > > to > >> > > > > > > > > >> > pin > >> > > > > > > > > >> > > a DAG to a specific version. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > 5. Most importantly, this feature provides an > >> incident > >> > > > > > response > >> > > > > > > > > >> > "rollback" > >> > > > > > > > > >> > > behavior. If a bad DAG version slips through CI/CD > >> > into > >> > > > > > > > production, > >> > > > > > > > > >> > either > >> > > > > > > > > >> > > an on-call engineer or a rollback-trigger > >> > > > (airflow-external) > >> > > > > > can > >> > > > > > > > > >> > instantly > >> > > > > > > > > >> > > roll back to the previous pinned version via the > >> > API/UI > >> > > to > >> > > > > > > > mitigate. > >> > > > > > > > > >> > > Without this, users have to revert the code in Git > >> and > >> > > > wait > >> > > > > > for > >> > > > > > > > the > >> > > > > > > > > >> > entire > >> > > > > > > > > >> > > CI/CD pipeline and file-sync process to run, which > >> is > >> > > > often > >> > > > > > too > >> > > > > > > > slow > >> > > > > > > > > >> > during > >> > > > > > > > > >> > > an outage. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > 6. Jarek - You are right, database schema changes > >> can > >> > be > >> > > > > > > discussed > >> > > > > > > > > >> later. > >> > > > > > > > > >> > > My intention was only to share a very brief > >> summary of > >> > > > how I > >> > > > > > > > deemed > >> > > > > > > > > >> it to > >> > > > > > > > > >> > > be technically feasible for early feedback. I did > >> > > briefly > >> > > > > > share > >> > > > > > > > the > >> > > > > > > > > >> > > high-level use cases ("Safe Deployment Gating" and > >> > > > "Instant > >> > > > > > > > > >> Rollbacks") > >> > > > > > > > > >> > in > >> > > > > > > > > >> > > the original mail, but I completely agree that > >> > aligning > >> > > on > >> > > > > the > >> > > > > > > UX > >> > > > > > > > > >> first > >> > > > > > > > > >> > > would be a good next step. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > If there are no major remaining concerns after this > >> > > > > response, > >> > > > > > I > >> > > > > > > > can > >> > > > > > > > > >> draft > >> > > > > > > > > >> > > and share an AIP to detail the UX, followed by a > >> > > > high-level > >> > > > > > > > > proposal, > >> > > > > > > > > >> > > caveats and next steps. > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > Thanks for your time. > >> > > > > > > > > >> > > Regards, > >> > > > > > > > > >> > > Piyush > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > On Tue, Apr 21, 2026 at 5:59 PM Oliveira, Niko < > >> > > > > > > > [email protected] > >> > > > > > > > > > > >> > > > > > > > > >> > > wrote: > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > I am with Jens on this one. I think we're > >> > complicating > >> > > > > > Airflow > >> > > > > > > > to > >> > > > > > > > > >> get > >> > > > > > > > > >> > > > around a bad practice. If stability of your Dags > >> is > >> > > > > critical > >> > > > > > > and > >> > > > > > > > > >> they > >> > > > > > > > > >> > are > >> > > > > > > > > >> > > > highly versioned then I think as Jens suggested > >> > > running > >> > > > > them > >> > > > > > > > > >> through a > >> > > > > > > > > >> > > > pipeline that first deploys them to a dev or > >> gamma > >> > > > > > environment > >> > > > > > > > > which > >> > > > > > > > > >> > > > verifies that quality of the Dags is what you > >> > expect. > >> > > If > >> > > > > > > > something > >> > > > > > > > > >> > slips > >> > > > > > > > > >> > > > through, then it's just normal software > >> practices of > >> > > > > either > >> > > > > > > > > >> reverting > >> > > > > > > > > >> > and > >> > > > > > > > > >> > > > rolling back or rolling forward with a fix pushed > >> > > > through > >> > > > > > the > >> > > > > > > > > >> > pipeline. I > >> > > > > > > > > >> > > > don't think Airflow should be aware of that > >> process > >> > or > >> > > > > > > > opinionated > >> > > > > > > > > >> > about > >> > > > > > > > > >> > > it. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > Cheers, > >> > > > > > > > > >> > > > Niko > >> > > > > > > > > >> > > > ------------------------------ > >> > > > > > > > > >> > > > *From:* Jens Scheffler <[email protected]> > >> > > > > > > > > >> > > > *Sent:* Monday, April 20, 2026 11:17 AM > >> > > > > > > > > >> > > > *To:* [email protected] < > >> > [email protected]> > >> > > > > > > > > >> > > > *Subject:* RE: [EXT] [DISCUSS] DAG Version > >> Pinning > >> > for > >> > > > > > > > Deployment > >> > > > > > > > > >> > Gating > >> > > > > > > > > >> > > > (Building on AIP-63) > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > CAUTION: This email originated from outside of > >> the > >> > > > > > > organization. > >> > > > > > > > > Do > >> > > > > > > > > >> not > >> > > > > > > > > >> > > > click links or open attachments unless you can > >> > confirm > >> > > > the > >> > > > > > > > sender > >> > > > > > > > > >> and > >> > > > > > > > > >> > > know > >> > > > > > > > > >> > > > the content is safe. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > AVERTISSEMENT: Ce courrier électronique provient > >> > d’un > >> > > > > > > expéditeur > >> > > > > > > > > >> > externe. > >> > > > > > > > > >> > > > Ne cliquez sur aucun lien et n’ouvrez aucune > >> pièce > >> > > > jointe > >> > > > > si > >> > > > > > > > vous > >> > > > > > > > > ne > >> > > > > > > > > >> > > pouvez > >> > > > > > > > > >> > > > pas confirmer l’identité de l’expéditeur et si > >> vous > >> > > > n’êtes > >> > > > > > pas > >> > > > > > > > > >> certain > >> > > > > > > > > >> > > que > >> > > > > > > > > >> > > > le contenu ne présente aucun risque. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > Hi, > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > I am still quite sceptical. Yes, if such pinning > >> is > >> > > > made, > >> > > > > > then > >> > > > > > > > per > >> > > > > > > > > >> Dag > >> > > > > > > > > >> > a > >> > > > > > > > > >> > > > change need to be possible via UI and API. But I > >> > still > >> > > > see > >> > > > > > it > >> > > > > > > as > >> > > > > > > > > >> > > > checken-and-egg - so you want to run a pinned > >> > version > >> > > > but > >> > > > > > then > >> > > > > > > > how > >> > > > > > > > > >> do > >> > > > > > > > > >> > > > you test the changes (w/o moving a version pin)? > >> > Then > >> > > > > again > >> > > > > > > some > >> > > > > > > > > >> test > >> > > > > > > > > >> > > > mode is needed or per run you need to make a > >> "test > >> > > run" > >> > > > > with > >> > > > > > > > > another > >> > > > > > > > > >> > > > version. Smells a bit like mis-using a production > >> > > system > >> > > > > for > >> > > > > > > > > >> testing. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > On the other hand, yes if all Dags share the same > >> > Git > >> > > > repo > >> > > > > > > then > >> > > > > > > > > >> merging > >> > > > > > > > > >> > > > a branch to some other will switch all Dags at > >> the > >> > > same > >> > > > > > time. > >> > > > > > > > > Still > >> > > > > > > > > >> you > >> > > > > > > > > >> > > > could utilize standard Git tools and cherry-pick > >> > > > > individual > >> > > > > > > > > changes > >> > > > > > > > > >> and > >> > > > > > > > > >> > > > no force to always make a full rollout. At least > >> 80% > >> > > > > > possible > >> > > > > > > > with > >> > > > > > > > > >> > > > standard CI/CD tools and Git. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > TLDR I see the danger that instead of a proper > >> CI/CD > >> > > and > >> > > > > > test > >> > > > > > > > > system > >> > > > > > > > > >> > > > such a feature might feel like you can easily > >> test > >> > on > >> > > a > >> > > > > > > > production > >> > > > > > > > > >> > > > system. Effectively it would be needed allowing > >> to > >> > > > start a > >> > > > > > Dag > >> > > > > > > > > with > >> > > > > > > > > >> any > >> > > > > > > > > >> > > > version to also be able to jump back as a > >> reversion. > >> > > > Even > >> > > > > > > > though, > >> > > > > > > > > >> yes, > >> > > > > > > > > >> > > > agree, all is technically possible. > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > Jens > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > On 20.04.26 16:40, Jarek Potiuk wrote: > >> > > > > > > > > >> > > > > +1 to what Ephraim wrote. I think that was a > >> > natural > >> > > > > next > >> > > > > > > step > >> > > > > > > > > we > >> > > > > > > > > >> > > > > discussed, but it needs significant refinement, > >> > > > starting > >> > > > > > > with > >> > > > > > > > > the > >> > > > > > > > > >> > > actual > >> > > > > > > > > >> > > > > use cases it should serve and the UX for user > >> > > > > > interaction. I > >> > > > > > > > > think > >> > > > > > > > > >> > > > related > >> > > > > > > > > >> > > > > database changes are pretty secondary. Use > >> cases > >> > > cover > >> > > > > > runs, > >> > > > > > > > > >> re-runs, > >> > > > > > > > > >> > > > > backfills, CI testing, rollbacks, etc. > >> Following > >> > the > >> > > > > > > > > >> "documentation > >> > > > > > > > > >> > > > first" > >> > > > > > > > > >> > > > > approach discussed in separate thread, > >> describing > >> > > the > >> > > > > > > context > >> > > > > > > > > and > >> > > > > > > > > >> > > > intention > >> > > > > > > > > >> > > > > of what we want to achieve is much more > >> important > >> > > than > >> > > > > DB > >> > > > > > > > schema > >> > > > > > > > > >> > > changes. > >> > > > > > > > > >> > > > > Once we know which use cases we want to serve, > >> the > >> > > DB > >> > > > > > schema > >> > > > > > > > > >> changes > >> > > > > > > > > >> > > and > >> > > > > > > > > >> > > > > other related items will emerge naturally. > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > On Mon, Apr 20, 2026 at 3:15 PM Ephraim > >> Anierobi < > >> > > > > > > > > >> > > > [email protected]> > >> > > > > > > > > >> > > > > wrote: > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > >> Hi Piyush, thanks for starting this > >> discussion. > >> > > > > > > > > >> > > > >> > >> > > > > > > > > >> > > > >> I like the proposal. We can introduce an > >> active > >> > > > > execution > >> > > > > > > > > version > >> > > > > > > > > >> > for > >> > > > > > > > > >> > > > >> "versioned bundles" and make scheduler/API > >> > resolve > >> > > > > > through > >> > > > > > > > it. > >> > > > > > > > > >> The > >> > > > > > > > > >> > > hard > >> > > > > > > > > >> > > > >> part of this is making airflow able to > >> > distinguish > >> > > > the > >> > > > > > > latest > >> > > > > > > > > >> parsed > >> > > > > > > > > >> > > > >> dagmodel's metadata from active scheduling > >> > > metadata. > >> > > > I > >> > > > > > will > >> > > > > > > > > >> suggest > >> > > > > > > > > >> > > you > >> > > > > > > > > >> > > > >> draft this in a google docs and share for > >> further > >> > > > > > > > discussions. > >> > > > > > > > > >> > > > >> > >> > > > > > > > > >> > > > >> Regards > >> > > > > > > > > >> > > > >> - Ephraim > >> > > > > > > > > >> > > > >> > >> > > > > > > > > >> > > > >> On Mon, 20 Apr 2026 at 01:31, Piyush > >> Maheshwari < > >> > > > > > > > > >> > > [email protected]> > >> > > > > > > > > >> > > > >> wrote: > >> > > > > > > > > >> > > > >> > >> > > > > > > > > >> > > > >>> Thanks for sharing your thoughts Jens. > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>>> be able to test it? … a Q&A/Testing > >> environment > >> > > to > >> > > > be > >> > > > > > > able > >> > > > > > > > to > >> > > > > > > > > >> > > sign-off > >> > > > > > > > > >> > > > >>> changes. > >> > > > > > > > > >> > > > >>> Yes, we’ve have built an isolated airflow > >> > > > environment > >> > > > > to > >> > > > > > > run > >> > > > > > > > > >> > > regression > >> > > > > > > > > >> > > > >>> checks before promoting to production. > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> As you suggested, we’re already running both > >> > > generic > >> > > > > and > >> > > > > > > > > >> DAG-custom > >> > > > > > > > > >> > > > >> static > >> > > > > > > > > >> > > > >>> checks in a CI job as a required step to > >> merge > >> > to > >> > > > the > >> > > > > > main > >> > > > > > > > > >> branch. > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>>> But then the "main" branch might be best > >> suited > >> > > if > >> > > > > > > > > >> > > > >>> implemented on the test system > >> > > > > > > > > >> > > > >>> In this case, problematic commits on “main” > >> can > >> > > > choke > >> > > > > > > other > >> > > > > > > > > >> > unrelated > >> > > > > > > > > >> > > > >>> changes. > >> > > > > > > > > >> > > > >>> So the other option would be to revert the > >> > > > problematic > >> > > > > > > > commits > >> > > > > > > > > >> and > >> > > > > > > > > >> > > > deploy > >> > > > > > > > > >> > > > >>> forward. > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> However, a key limitation with this approach > >> > that > >> > > > > > remains > >> > > > > > > is > >> > > > > > > > > >> that a > >> > > > > > > > > >> > > > >> commit > >> > > > > > > > > >> > > > >>> affecting multiple DAGs goes live for either > >> all > >> > > > DAGs > >> > > > > or > >> > > > > > > > none. > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> Second important feature we get with this is > >> > > instant > >> > > > > > > > DAG-level > >> > > > > > > > > >> > > rollback > >> > > > > > > > > >> > > > >>> without waiting for a revert commit to merge > >> and > >> > > be > >> > > > > > picked > >> > > > > > > > by > >> > > > > > > > > >> > > airflow. > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> I think DAG-level version pinning can also > >> > unlock > >> > > a > >> > > > > lot > >> > > > > > of > >> > > > > > > > > >> > > flexibility > >> > > > > > > > > >> > > > >> for > >> > > > > > > > > >> > > > >>> deployments including tiered rollouts, > >> > > auto-rollback > >> > > > > > > > triggers, > >> > > > > > > > > >> > timed > >> > > > > > > > > >> > > > >>> deployment windows and so on. > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> Looking forward to hear your thoughts. > >> > > > > > > > > >> > > > >>> Regards, > >> > > > > > > > > >> > > > >>> Piyush > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> On Sun, 19 Apr 2026 at 3:12 PM, Jens > >> Scheffler < > >> > > > > > > > > >> > [email protected]> > >> > > > > > > > > >> > > > >>> wrote: > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>>> Thanks Piyush for dropping the discussion! > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> I think in general QA processes are > >> important > >> > > and a > >> > > > > > valid > >> > > > > > > > use > >> > > > > > > > > >> > case. > >> > > > > > > > > >> > > So > >> > > > > > > > > >> > > > >> a > >> > > > > > > > > >> > > > >>>> kind of pinning Dag versions really is > >> > important. > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> Thinking about it, if you pin the version > >> ... > >> > how > >> > > > > would > >> > > > > > > you > >> > > > > > > > > >> then > >> > > > > > > > > >> > be > >> > > > > > > > > >> > > > >> able > >> > > > > > > > > >> > > > >>>> to test it? I assume you would need (and > >> should > >> > > > have > >> > > > > or > >> > > > > > > > > invest > >> > > > > > > > > >> > > into) a > >> > > > > > > > > >> > > > >>>> Q&A/Testing environment to be able to > >> sign-off > >> > > > > changes. > >> > > > > > > > Both > >> > > > > > > > > in > >> > > > > > > > > >> > > > >>>> infrastructure but also for Dag changes. > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> If you are changing Dags first of all static > >> > > checks > >> > > > > on > >> > > > > > > Dag > >> > > > > > > > > code > >> > > > > > > > > >> > are > >> > > > > > > > > >> > > > >> very > >> > > > > > > > > >> > > > >>>> much proposed as well as you can have tests > >> > > > > implemented > >> > > > > > > and > >> > > > > > > > > >> test > >> > > > > > > > > >> > > your > >> > > > > > > > > >> > > > >>>> Dags and logic. Similar like software a > >> CI/CD > >> > > > system > >> > > > > > will > >> > > > > > > > be > >> > > > > > > > > a > >> > > > > > > > > >> > good > >> > > > > > > > > >> > > > >>>> setup. Alongside Dag changes also have > >> logical > >> > > > > changes > >> > > > > > > that > >> > > > > > > > > >> mostly > >> > > > > > > > > >> > > can > >> > > > > > > > > >> > > > >>>> only be tested in a live system and not as > >> > static > >> > > > > > checks. > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> Have you considered using Git and a set of > >> > > branches > >> > > > > for > >> > > > > > > > > >> > implementing > >> > > > > > > > > >> > > > >>>> such staging? E.g. you have a git repo and > >> you > >> > > plan > >> > > > > to > >> > > > > > > make > >> > > > > > > > > >> > changes. > >> > > > > > > > > >> > > > >>>> Then you would open a PR for the change and > >> > merge > >> > > > it > >> > > > > to > >> > > > > > > the > >> > > > > > > > > >> "main" > >> > > > > > > > > >> > > > >>>> branch - and there in your CI/CD you can > >> check > >> > > all > >> > > > > > sorts > >> > > > > > > of > >> > > > > > > > > >> static > >> > > > > > > > > >> > > > >>>> checks and tests. But then the "main" branch > >> > > might > >> > > > be > >> > > > > > > best > >> > > > > > > > > >> suited > >> > > > > > > > > >> > if > >> > > > > > > > > >> > > > >>>> implemented on the test system. Once you > >> > validate > >> > > > the > >> > > > > > > > changes > >> > > > > > > > > >> > > > >> end-to-end > >> > > > > > > > > >> > > > >>>> you could make another PR for example to a > >> > "prod" > >> > > > > > branch. > >> > > > > > > > And > >> > > > > > > > > >> if > >> > > > > > > > > >> > > your > >> > > > > > > > > >> > > > >>>> production system is only pulling Dags from > >> the > >> > > > > "prod" > >> > > > > > > > branch > >> > > > > > > > > >> then > >> > > > > > > > > >> > > you > >> > > > > > > > > >> > > > >>>> can have this merging strategy as a staging > >> > > setup. > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> Would this resolve your PING problem? Or > >> which > >> > > > other > >> > > > > > > detail > >> > > > > > > > > in > >> > > > > > > > > >> the > >> > > > > > > > > >> > > use > >> > > > > > > > > >> > > > >>>> case would require a PIN on top of a staging > >> > > > > strategy? > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> Jens > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> P.S.: Have enabled your confluence account > >> > after > >> > > it > >> > > > > was > >> > > > > > > > > >> created in > >> > > > > > > > > >> > > > >> order > >> > > > > > > > > >> > > > >>>> to write to Confluence, sorry, typical > >> pitfall > >> > > > after > >> > > > > > > > account > >> > > > > > > > > >> > > creation > >> > > > > > > > > >> > > > >>>> permissions were not set. Now it should > >> work. > >> > Let > >> > > > me > >> > > > > > know > >> > > > > > > > if > >> > > > > > > > > >> not. > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> On 19.04.26 01:40, Piyush Maheshwari wrote: > >> > > > > > > > > >> > > > >>>>> Hi everyone, > >> > > > > > > > > >> > > > >>>>> I'm a new contributor to Airflow. I'd like > >> to > >> > > > > propose > >> > > > > > a > >> > > > > > > > new > >> > > > > > > > > >> > feature > >> > > > > > > > > >> > > > >> for > >> > > > > > > > > >> > > > >>>> Airflow: DAG Version Pinning. > >> > > > > > > > > >> > > > >>>>> Building on the foundation introduced by > >> > AIP-63: > >> > > > DAG > >> > > > > > > > > >> Versioning ( > >> > > > > > > > > >> > > > >> > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63*3A*DAG*Versioning__;JSsr!!Ci6f514n9QsL8ck!l3ZKTOw996h9qu4NR0VT4ouUryUdk_HmXUAVPbwCHwPwn0N2CCptVdx95-V0BoRFjws9huE_1Vy-THL8jw$ > >> > > > > > > > > >> > > > >>> ), > >> > > > > > > > > >> > > > >>>> this proposal aims to extend Airflow's > >> > > capabilities > >> > > > > to > >> > > > > > > > > support > >> > > > > > > > > >> > true > >> > > > > > > > > >> > > > >>>> continuous deployment (CD) gating and safer > >> > > release > >> > > > > > > cycles. > >> > > > > > > > > >> > > > >>>>> The Problem & Use Cases > >> > > > > > > > > >> > > > >>>>> Currently, the scheduler always creates > >> > DagRuns > >> > > > > using > >> > > > > > > the > >> > > > > > > > > >> latest > >> > > > > > > > > >> > > > >> parsed > >> > > > > > > > > >> > > > >>>> DagVersion. This means that the updated DAG > >> > code > >> > > is > >> > > > > > > > deployed > >> > > > > > > > > >> > (takes > >> > > > > > > > > >> > > > >>> effect) > >> > > > > > > > > >> > > > >>>> right after the dag-processor processes it. > >> > While > >> > > > > this > >> > > > > > is > >> > > > > > > > > great > >> > > > > > > > > >> > for > >> > > > > > > > > >> > > > >> rapid > >> > > > > > > > > >> > > > >>>> development, teams running business-critical > >> > > > > pipelines > >> > > > > > > > often > >> > > > > > > > > >> need > >> > > > > > > > > >> > > > >>> stricter > >> > > > > > > > > >> > > > >>>> deployment mechanisms. Specifically: > >> > > > > > > > > >> > > > >>>>> * > >> > > > > > > > > >> > > > >>>>> Safe Deployment Gating: The ability to pin > >> a > >> > DAG > >> > > > to > >> > > > > > its > >> > > > > > > > last > >> > > > > > > > > >> > known > >> > > > > > > > > >> > > > >>>> stable version while new code is parsed in > >> the > >> > > > > > > background. > >> > > > > > > > > This > >> > > > > > > > > >> > > allows > >> > > > > > > > > >> > > > >>> the > >> > > > > > > > > >> > > > >>>> new version to be held back until it passes > >> > > > automated > >> > > > > > > > > >> regression > >> > > > > > > > > >> > > tests > >> > > > > > > > > >> > > > >> or > >> > > > > > > > > >> > > > >>>> receives explicit manual approval. > >> > > > > > > > > >> > > > >>>>> * > >> > > > > > > > > >> > > > >>>>> Instant Rollbacks: If an issue is detected > >> in > >> > a > >> > > > > newly > >> > > > > > > > > promoted > >> > > > > > > > > >> > DAG > >> > > > > > > > > >> > > > >>>> version, users need the capability to > >> instantly > >> > > > roll > >> > > > > > back > >> > > > > > > > to > >> > > > > > > > > a > >> > > > > > > > > >> > > > previous > >> > > > > > > > > >> > > > >>>> version via the UI/API, without having to > >> > revert > >> > > > the > >> > > > > > > > > underlying > >> > > > > > > > > >> > code > >> > > > > > > > > >> > > > >> and > >> > > > > > > > > >> > > > >>>> wait for the repository sync and DAG > >> processing > >> > > > > cycle. > >> > > > > > > > > >> > > > >>>>> High-Level Proposed Solution > >> > > > > > > > > >> > > > >>>>> Introduce an optional > >> active_dag_version_id to > >> > > the > >> > > > > > > > DagModel. > >> > > > > > > > > >> This > >> > > > > > > > > >> > > > >> field > >> > > > > > > > > >> > > > >>>> can be used to pin a DAG version for > >> scheduling > >> > > and > >> > > > > > > > > execution, > >> > > > > > > > > >> > while > >> > > > > > > > > >> > > > >> the > >> > > > > > > > > >> > > > >>>> dag-processor can continue to parse and > >> > register > >> > > > > newer > >> > > > > > > DAG > >> > > > > > > > > >> > versions. > >> > > > > > > > > >> > > > >>>>> * > >> > > > > > > > > >> > > > >>>>> When this pin is set, the scheduler and API > >> > will > >> > > > > > respect > >> > > > > > > > the > >> > > > > > > > > >> > pinned > >> > > > > > > > > >> > > > >>>> version for creating runs and executing > >> tasks, > >> > > > > > separating > >> > > > > > > > the > >> > > > > > > > > >> > > parsing > >> > > > > > > > > >> > > > >> of > >> > > > > > > > > >> > > > >>>> new code from the execution of new code. > >> > > > > > > > > >> > > > >>>>> * > >> > > > > > > > > >> > > > >>>>> If the pin is NULL, the system defaults to > >> the > >> > > > > current > >> > > > > > > > > >> behavior > >> > > > > > > > > >> > > > >> (always > >> > > > > > > > > >> > > > >>>> executing the latest parsed version). This > >> way, > >> > > we > >> > > > > can > >> > > > > > > > > maintain > >> > > > > > > > > >> > > > >> complete > >> > > > > > > > > >> > > > >>>> backwards compatibility. > >> > > > > > > > > >> > > > >>>>> I have put together some detailed notes > >> > covering > >> > > > the > >> > > > > > > data > >> > > > > > > > > >> model > >> > > > > > > > > >> > > > >>> changes, > >> > > > > > > > > >> > > > >>>> database migrations, and edge cases with > >> this > >> > > > > approach. > >> > > > > > > If > >> > > > > > > > > >> there > >> > > > > > > > > >> > is > >> > > > > > > > > >> > > > >>> general > >> > > > > > > > > >> > > > >>>> alignment that this fits the vision for > >> > Airflow, > >> > > I > >> > > > > > would > >> > > > > > > > like > >> > > > > > > > > >> to > >> > > > > > > > > >> > > take > >> > > > > > > > > >> > > > >>> this > >> > > > > > > > > >> > > > >>>> proposal through the formal AIP review > >> process. > >> > > > > > > > > >> > > > >>>>> But I would love to get the community's > >> > feedback > >> > > > on > >> > > > > > the > >> > > > > > > > > >> feature > >> > > > > > > > > >> > and > >> > > > > > > > > >> > > > >> the > >> > > > > > > > > >> > > > >>>> high-level approach. > >> > > > > > > > > >> > > > >>>>> I'll also need someone to grant me access > >> to > >> > > > create > >> > > > > > > > content > >> > > > > > > > > on > >> > > > > > > > > >> > the > >> > > > > > > > > >> > > > >>>> Airflow Confluence wiki. > >> > > > > > > > > >> > > > >>>>> Thanks for your time! > >> > > > > > > > > >> > > > >>>>> Regards, > >> > > > > > > > > >> > > > >>>>> Piyush > >> > > > > > > > > >> > > > >>>>> > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >> > > > > > > > > > >> > > > > > >> --------------------------------------------------------------------- > >> > > > > > > > > >> > > > >>>> To unsubscribe, e-mail: > >> > > > > > > [email protected] > >> > > > > > > > > >> > > > >>>> For additional commands, e-mail: > >> > > > > > > > [email protected] > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > >>>> > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > >> > > > > > > > >> > > --------------------------------------------------------------------- > >> > > > > > > > > >> > > > To unsubscribe, e-mail: > >> > > > > [email protected] > >> > > > > > > > > >> > > > For additional commands, e-mail: > >> > > > > > [email protected] > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
