Hi,
I am still quite sceptical. Yes, if such pinning is made, then per Dag a
change need to be possible via UI and API. But I still see it as
checken-and-egg - so you want to run a pinned version but then how do
you test the changes (w/o moving a version pin)? Then again some test
mode is needed or per run you need to make a "test run" with another
version. Smells a bit like mis-using a production system for testing.
On the other hand, yes if all Dags share the same Git repo then merging
a branch to some other will switch all Dags at the same time. Still you
could utilize standard Git tools and cherry-pick individual changes and
no force to always make a full rollout. At least 80% possible with
standard CI/CD tools and Git.
TLDR I see the danger that instead of a proper CI/CD and test system
such a feature might feel like you can easily test on a production
system. Effectively it would be needed allowing to start a Dag with any
version to also be able to jump back as a reversion. Even though, yes,
agree, all is technically possible.
Jens
On 20.04.26 16:40, Jarek Potiuk wrote:
+1 to what Ephraim wrote. I think that was a natural next step we
discussed, but it needs significant refinement, starting with the actual
use cases it should serve and the UX for user interaction. I think related
database changes are pretty secondary. Use cases cover runs, re-runs,
backfills, CI testing, rollbacks, etc. Following the "documentation first"
approach discussed in separate thread, describing the context and intention
of what we want to achieve is much more important than DB schema changes.
Once we know which use cases we want to serve, the DB schema changes and
other related items will emerge naturally.
On Mon, Apr 20, 2026 at 3:15 PM Ephraim Anierobi <[email protected]>
wrote:
Hi Piyush, thanks for starting this discussion.
I like the proposal. We can introduce an active execution version for
"versioned bundles" and make scheduler/API resolve through it. The hard
part of this is making airflow able to distinguish the latest parsed
dagmodel's metadata from active scheduling metadata. I will suggest you
draft this in a google docs and share for further discussions.
Regards
- Ephraim
On Mon, 20 Apr 2026 at 01:31, Piyush Maheshwari <[email protected]>
wrote:
Thanks for sharing your thoughts Jens.
be able to test it? … a Q&A/Testing environment to be able to sign-off
changes.
Yes, we’ve have built an isolated airflow environment to run regression
checks before promoting to production.
As you suggested, we’re already running both generic and DAG-custom
static
checks in a CI job as a required step to merge to the main branch.
But then the "main" branch might be best suited if
implemented on the test system
In this case, problematic commits on “main” can choke other unrelated
changes.
So the other option would be to revert the problematic commits and deploy
forward.
However, a key limitation with this approach that remains is that a
commit
affecting multiple DAGs goes live for either all DAGs or none.
Second important feature we get with this is instant DAG-level rollback
without waiting for a revert commit to merge and be picked by airflow.
I think DAG-level version pinning can also unlock a lot of flexibility
for
deployments including tiered rollouts, auto-rollback triggers, timed
deployment windows and so on.
Looking forward to hear your thoughts.
Regards,
Piyush
On Sun, 19 Apr 2026 at 3:12 PM, Jens Scheffler <[email protected]>
wrote:
Thanks Piyush for dropping the discussion!
I think in general QA processes are important and a valid use case. So
a
kind of pinning Dag versions really is important.
Thinking about it, if you pin the version ... how would you then be
able
to test it? I assume you would need (and should have or invest into) a
Q&A/Testing environment to be able to sign-off changes. Both in
infrastructure but also for Dag changes.
If you are changing Dags first of all static checks on Dag code are
very
much proposed as well as you can have tests implemented and test your
Dags and logic. Similar like software a CI/CD system will be a good
setup. Alongside Dag changes also have logical changes that mostly can
only be tested in a live system and not as static checks.
Have you considered using Git and a set of branches for implementing
such staging? E.g. you have a git repo and you plan to make changes.
Then you would open a PR for the change and merge it to the "main"
branch - and there in your CI/CD you can check all sorts of static
checks and tests. But then the "main" branch might be best suited if
implemented on the test system. Once you validate the changes
end-to-end
you could make another PR for example to a "prod" branch. And if your
production system is only pulling Dags from the "prod" branch then you
can have this merging strategy as a staging setup.
Would this resolve your PING problem? Or which other detail in the use
case would require a PIN on top of a staging strategy?
Jens
P.S.: Have enabled your confluence account after it was created in
order
to write to Confluence, sorry, typical pitfall after account creation
permissions were not set. Now it should work. Let me know if not.
On 19.04.26 01:40, Piyush Maheshwari wrote:
Hi everyone,
I'm a new contributor to Airflow. I'd like to propose a new feature
for
Airflow: DAG Version Pinning.
Building on the foundation introduced by AIP-63: DAG Versioning (
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning
),
this proposal aims to extend Airflow's capabilities to support true
continuous deployment (CD) gating and safer release cycles.
The Problem & Use Cases
Currently, the scheduler always creates DagRuns using the latest
parsed
DagVersion. This means that the updated DAG code is deployed (takes
effect)
right after the dag-processor processes it. While this is great for
rapid
development, teams running business-critical pipelines often need
stricter
deployment mechanisms. Specifically:
*
Safe Deployment Gating: The ability to pin a DAG to its last known
stable version while new code is parsed in the background. This allows
the
new version to be held back until it passes automated regression tests
or
receives explicit manual approval.
*
Instant Rollbacks: If an issue is detected in a newly promoted DAG
version, users need the capability to instantly roll back to a previous
version via the UI/API, without having to revert the underlying code
and
wait for the repository sync and DAG processing cycle.
High-Level Proposed Solution
Introduce an optional active_dag_version_id to the DagModel. This
field
can be used to pin a DAG version for scheduling and execution, while
the
dag-processor can continue to parse and register newer DAG versions.
*
When this pin is set, the scheduler and API will respect the pinned
version for creating runs and executing tasks, separating the parsing
of
new code from the execution of new code.
*
If the pin is NULL, the system defaults to the current behavior
(always
executing the latest parsed version). This way, we can maintain
complete
backwards compatibility.
I have put together some detailed notes covering the data model
changes,
database migrations, and edge cases with this approach. If there is
general
alignment that this fits the vision for Airflow, I would like to take
this
proposal through the formal AIP review process.
But I would love to get the community's feedback on the feature and
the
high-level approach.
I'll also need someone to grant me access to create content on the
Airflow Confluence wiki.
Thanks for your time!
Regards,
Piyush
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
