Hi Piyush, thanks for starting this discussion. I like the proposal. We can introduce an active execution version for "versioned bundles" and make scheduler/API resolve through it. The hard part of this is making airflow able to distinguish the latest parsed dagmodel's metadata from active scheduling metadata. I will suggest you draft this in a google docs and share for further discussions.
Regards - Ephraim On Mon, 20 Apr 2026 at 01:31, Piyush Maheshwari <[email protected]> wrote: > Thanks for sharing your thoughts Jens. > > > be able to test it? … a Q&A/Testing environment to be able to sign-off > changes. > Yes, we’ve have built an isolated airflow environment to run regression > checks before promoting to production. > > As you suggested, we’re already running both generic and DAG-custom static > checks in a CI job as a required step to merge to the main branch. > > > But then the "main" branch might be best suited if > implemented on the test system > In this case, problematic commits on “main” can choke other unrelated > changes. > So the other option would be to revert the problematic commits and deploy > forward. > > However, a key limitation with this approach that remains is that a commit > affecting multiple DAGs goes live for either all DAGs or none. > > Second important feature we get with this is instant DAG-level rollback > without waiting for a revert commit to merge and be picked by airflow. > > I think DAG-level version pinning can also unlock a lot of flexibility for > deployments including tiered rollouts, auto-rollback triggers, timed > deployment windows and so on. > > Looking forward to hear your thoughts. > Regards, > Piyush > > On Sun, 19 Apr 2026 at 3:12 PM, Jens Scheffler <[email protected]> > wrote: > > > Thanks Piyush for dropping the discussion! > > > > I think in general QA processes are important and a valid use case. So a > > kind of pinning Dag versions really is important. > > > > Thinking about it, if you pin the version ... how would you then be able > > to test it? I assume you would need (and should have or invest into) a > > Q&A/Testing environment to be able to sign-off changes. Both in > > infrastructure but also for Dag changes. > > > > If you are changing Dags first of all static checks on Dag code are very > > much proposed as well as you can have tests implemented and test your > > Dags and logic. Similar like software a CI/CD system will be a good > > setup. Alongside Dag changes also have logical changes that mostly can > > only be tested in a live system and not as static checks. > > > > Have you considered using Git and a set of branches for implementing > > such staging? E.g. you have a git repo and you plan to make changes. > > Then you would open a PR for the change and merge it to the "main" > > branch - and there in your CI/CD you can check all sorts of static > > checks and tests. But then the "main" branch might be best suited if > > implemented on the test system. Once you validate the changes end-to-end > > you could make another PR for example to a "prod" branch. And if your > > production system is only pulling Dags from the "prod" branch then you > > can have this merging strategy as a staging setup. > > > > Would this resolve your PING problem? Or which other detail in the use > > case would require a PIN on top of a staging strategy? > > > > Jens > > > > P.S.: Have enabled your confluence account after it was created in order > > to write to Confluence, sorry, typical pitfall after account creation > > permissions were not set. Now it should work. Let me know if not. > > > > On 19.04.26 01:40, Piyush Maheshwari wrote: > > > Hi everyone, > > > I'm a new contributor to Airflow. I'd like to propose a new feature for > > Airflow: DAG Version Pinning. > > > Building on the foundation introduced by AIP-63: DAG Versioning ( > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning > ), > > this proposal aims to extend Airflow's capabilities to support true > > continuous deployment (CD) gating and safer release cycles. > > > The Problem & Use Cases > > > Currently, the scheduler always creates DagRuns using the latest parsed > > DagVersion. This means that the updated DAG code is deployed (takes > effect) > > right after the dag-processor processes it. While this is great for rapid > > development, teams running business-critical pipelines often need > stricter > > deployment mechanisms. Specifically: > > > > > > * > > > Safe Deployment Gating: The ability to pin a DAG to its last known > > stable version while new code is parsed in the background. This allows > the > > new version to be held back until it passes automated regression tests or > > receives explicit manual approval. > > > * > > > Instant Rollbacks: If an issue is detected in a newly promoted DAG > > version, users need the capability to instantly roll back to a previous > > version via the UI/API, without having to revert the underlying code and > > wait for the repository sync and DAG processing cycle. > > > > > > High-Level Proposed Solution > > > Introduce an optional active_dag_version_id to the DagModel. This field > > can be used to pin a DAG version for scheduling and execution, while the > > dag-processor can continue to parse and register newer DAG versions. > > > > > > * > > > When this pin is set, the scheduler and API will respect the pinned > > version for creating runs and executing tasks, separating the parsing of > > new code from the execution of new code. > > > * > > > If the pin is NULL, the system defaults to the current behavior (always > > executing the latest parsed version). This way, we can maintain complete > > backwards compatibility. > > > > > > I have put together some detailed notes covering the data model > changes, > > database migrations, and edge cases with this approach. If there is > general > > alignment that this fits the vision for Airflow, I would like to take > this > > proposal through the formal AIP review process. > > > But I would love to get the community's feedback on the feature and the > > high-level approach. > > > I'll also need someone to grant me access to create content on the > > Airflow Confluence wiki. > > > > > > Thanks for your time! > > > Regards, > > > Piyush > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
