Thanks Piyush for dropping the discussion!
I think in general QA processes are important and a valid use case. So a
kind of pinning Dag versions really is important.
Thinking about it, if you pin the version ... how would you then be able
to test it? I assume you would need (and should have or invest into) a
Q&A/Testing environment to be able to sign-off changes. Both in
infrastructure but also for Dag changes.
If you are changing Dags first of all static checks on Dag code are very
much proposed as well as you can have tests implemented and test your
Dags and logic. Similar like software a CI/CD system will be a good
setup. Alongside Dag changes also have logical changes that mostly can
only be tested in a live system and not as static checks.
Have you considered using Git and a set of branches for implementing
such staging? E.g. you have a git repo and you plan to make changes.
Then you would open a PR for the change and merge it to the "main"
branch - and there in your CI/CD you can check all sorts of static
checks and tests. But then the "main" branch might be best suited if
implemented on the test system. Once you validate the changes end-to-end
you could make another PR for example to a "prod" branch. And if your
production system is only pulling Dags from the "prod" branch then you
can have this merging strategy as a staging setup.
Would this resolve your PING problem? Or which other detail in the use
case would require a PIN on top of a staging strategy?
Jens
P.S.: Have enabled your confluence account after it was created in order
to write to Confluence, sorry, typical pitfall after account creation
permissions were not set. Now it should work. Let me know if not.
On 19.04.26 01:40, Piyush Maheshwari wrote:
Hi everyone,
I'm a new contributor to Airflow. I'd like to propose a new feature for
Airflow: DAG Version Pinning.
Building on the foundation introduced by AIP-63: DAG Versioning
(https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning),
this proposal aims to extend Airflow's capabilities to support true continuous
deployment (CD) gating and safer release cycles.
The Problem & Use Cases
Currently, the scheduler always creates DagRuns using the latest parsed
DagVersion. This means that the updated DAG code is deployed (takes effect)
right after the dag-processor processes it. While this is great for rapid
development, teams running business-critical pipelines often need stricter
deployment mechanisms. Specifically:
*
Safe Deployment Gating: The ability to pin a DAG to its last known stable
version while new code is parsed in the background. This allows the new version
to be held back until it passes automated regression tests or receives explicit
manual approval.
*
Instant Rollbacks: If an issue is detected in a newly promoted DAG version,
users need the capability to instantly roll back to a previous version via the
UI/API, without having to revert the underlying code and wait for the
repository sync and DAG processing cycle.
High-Level Proposed Solution
Introduce an optional active_dag_version_id to the DagModel. This field can be
used to pin a DAG version for scheduling and execution, while the dag-processor
can continue to parse and register newer DAG versions.
*
When this pin is set, the scheduler and API will respect the pinned version for
creating runs and executing tasks, separating the parsing of new code from the
execution of new code.
*
If the pin is NULL, the system defaults to the current behavior (always
executing the latest parsed version). This way, we can maintain complete
backwards compatibility.
I have put together some detailed notes covering the data model changes,
database migrations, and edge cases with this approach. If there is general
alignment that this fits the vision for Airflow, I would like to take this
proposal through the formal AIP review process.
But I would love to get the community's feedback on the feature and the
high-level approach.
I'll also need someone to grant me access to create content on the Airflow
Confluence wiki.
Thanks for your time!
Regards,
Piyush
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
