Dear Airflow community,
I'd like to raise a discussion about the behavior change of
`xcom_pull(task_ids=None)` between Airflow 2.x and 3.x, and whether it should
be treated as an unintended behavior change (a bug to fix).
(Related to https://github.com/apache/airflow/issues/51821)
Background
In Airflow 2.x (and in Airflow 1.x as well), the documentation explicitly
states that passing `task_ids=None` removes the filter. In practice, this
resulted in returning the most recent XCom for a given key within the DAG.
In Airflow 3, this changed so that `task_ids=None` defaults to the current
task's `task_id`.
Why this might be considered an unintended change
* The previous behavior was documented, and users have relied on it.
* The new default does not appear to provide meaningful value in most real
scenarios, since tasks typically do not consume their own XCom outputs within
the same run.
* Effectively, a broadly useful default was replaced with behavior that is
often a no-op.
Practical impact
The change introduces several usability regressions:
* Tighter coupling: downstream tasks must explicitly reference upstream
`task_id`s, increasing fragility during refactoring.
* Reduced modularity: reusable DAG components now require knowledge of
upstream internals.
* Lost pattern: it removes a simple and widely used approach-retrieving the
latest value for a key regardless of which task produced it.
On the "ambiguity" concern
A reasonable and consistent interpretation of the previous behavior is:
> return the most recent XCom for the given key within the DAG scope
While there are edge cases (e.g., multiple producers), these are already part
of DAG design considerations and not unique to this behavior.
Workarounds in Airflow 3
It is possible to replicate the old behavior via the public API, but with
notable drawbacks:
* Additional overhead (e.g., REST calls)
* No support for Jinja templating, which limits usage in classic operators
Trade-offs
Restoring the previous behavior would technically introduce a breaking change
for users already on Airflow 3.0-3.2 who may have adapted to the new semantics.
However, the practical impact of this is likely limited:
* The current default behavior (falling back to the current task's
`task_id`) has very low usefulness in real scenarios, so the likelihood that
users intentionally rely on it is low.
* If someone does rely on it, the behavior is trivial to preserve by
explicitly passing the current task's `task_id`.
* A behavior change would only affect cases where:
* an upstream task pushed an XCom with the same key, and
* the current task did not produce a value for that key
Given this, the risk appears constrained and manageable, while restoring the
previous behavior would re-enable a documented and widely useful pattern.
So the question becomes whether:
* the benefits of restoring a documented and broadly useful behavior
outweighs
* the relatively low and mitigatable risk for current Airflow 3 users
Question to the community
Do we consider this change:
* an intended semantic correction, or
* an unintended behavior change that should be treated as a bug fix?
If there is agreement on the latter, I'm happy to contribute to the fix.
Thanks,
Gyorgy Foldvari
________________________________
Information regarding MSCI's processing of personal data may be found at
www.msci.com/privacy-pledge. This email message and any attachments are for the
sole use of the intended recipients and may contain proprietary and/or
confidential information which may be privileged or otherwise protected from
disclosure. Any unauthorized review, use, disclosure or distribution is
prohibited. All rights and remedies are reserved. If you are not an intended
recipient, please contact the sender by reply email and destroy the original
message and any copies of the message as well as any attachments to the
original message. Local registered entity information:
https://www.msci.com/local-registered-entities
Information Classification: GENERAL
