Among, Wei, thanks for the suggestions.
Regarding to the idea to use variables instead. Variables are global, not
specific to DAG runs. Therefore this is not an alternative here. What we need
is a way to get the MOST RECENT value pushed to XCom with a particular key by
ANY upstream task in the same DAG run of the task which is querying XCom.
I also have n alternative solution which also considers the timestamp, so it
guarantees that the most recently stored value will be used. Unfortunately it
involves an API call with all of its overheads. I share it below.
The limitation of these solutions are:
* They are more expensive comparted to the DB query used in Airflow 2
* They cannot be used in a templated field of a classic operator
My solution:
```
def _get_most_recent(xcom_collection_response: XComCollectionResponse) ->
XComResponse | None:
if xcom_collection_response.total_entries < 1:
return None
xcom_entries = xcom_collection_response.xcom_entries
return max(xcom_entries, key=lambda entry: entry.timestamp)
def xcom_pull(
key: str,
task_ids: str | Iterable[str] | None = None,
default: Any = None,
ti: TaskInstance | None = None,
airflow_api_client_provider: AirflowApiClientProvider | None = None
) -> Any | None:
"""Pull an XCom value for a key, optionally scoped to specific task IDs.
If ``task_ids`` is provided, delegates to ``TaskInstance.xcom_pull`` for
those
tasks. Otherwise, it queries the Airflow REST API for XCom entries in the
current DAG run, selects the most recent entry, and pulls the key from that
task.
:param key: XCom key to retrieve.
:param task_ids: Task ID or iterable of task IDs to scope the pull.
:param default: Default value if the XCom is not found.
:param ti: TaskInstance to use; if ``None``, resolved via ``safe_ti``.
:param airflow_api_client_provider: Provider for the Airflow API client;
defaults to ``AirflowApiClientProvider``.
:returns: The XCom value if found, otherwise ``default``.
"""
ti = safe_ti(ti=ti)
if task_ids:
return ti.xcom_pull(task_ids=task_ids, key=key, default=default)
airflow_api_client_provider = airflow_api_client_provider or
AirflowApiClientProvider()
with airflow_api_client_provider.get_api_client() as api_client:
xcom_api = XComApi(api_client=api_client)
xcom_entries = xcom_api.get_xcom_entries(
dag_id=ti.dag_id,
dag_run_id=ti.run_id,
task_id='~',
xcom_key=key
)
latest_xcom_entry = _get_most_recent(xcom_entries)
if latest_xcom_entry:
task_id = latest_xcom_entry.task_id
return ti.xcom_pull(key=key, task_ids=task_id, default=default)
return default
```
AirflowApiClientProvider class is taking care about authentication and JWT key
caching.
safe_ti(ti) is a very simple utility method which is using current context to
get ti if the passed ti value is None.
Information Classification: GENERAL
On 2026/04/17 10:34:05 Wei Lee wrote:
> Thanks Amogh for submitting the PR to clarify this and for suggesting a
> potential workaround. I agree with the points raised here.
>
> The Airflow 2 behavior was a bit confusing, which was one of the reasons we
> chose to remove it in Airflow 3 — and as Amogh pointed out, it was also
> expensive. TP's suggestion sounds like a better alternative going forward.
> Before jumping into changing the Dag code to use Variables, it might be
> worth exploring whether Amogh's plugin approach could ease the migration
> first.
>
> Best,
> Wei
>
> Amogh Desai <[email protected]> 於 2026年4月17日週五 下午5:43寫道：
>
> > On a side note, I think it would be worth while clarifying this in the docs
> > too,
> > and I proposed a PR to do that:
> > https://github.com/apache/airflow/pull/65406
> >
> > And side note to the side note, I experimented around by writing a compat
> > plugin
> > that could help you achieve something like that to approximate AF2
> > behaviour and
> > it works fine.
> >
> > Plugin:
> > ```code
> > from airflow.plugins_manager import AirflowPlugin
> >
> > def xcom_pull_from_upstream(context, key, default=None):
> > """
> > Pull XCom by key from all upstream tasks.
> > Approximates Airflow 2 behavior in Airflow 3.
> > """
> > ti = context['ti']
> > task = context['task']
> >
> > upstream_task_ids = [t.task_id for t in task.upstream_list]
> >
> > if not upstream_task_ids:
> > return default
> >
> > values = ti.xcom_pull(task_ids=upstream_task_ids, key=key)
> >
> > if not values:
> > return default
> >
> > if isinstance(values, list):
> > for value in reversed(values):
> > if value is not None:
> > return value
> > elif values is not None:
> > return values
> >
> > return default
> >
> >
> > class XComCompatPlugin(AirflowPlugin):
> > name = "xcom_compat"
> > macros = [xcom_pull_from_upstream]
> > ```
> >
> > Usage:
> > ```
> > def pull_that_xcom_via_macros(**kwargs):
> > value = kwargs['macros'].xcom_compat.xcom_pull_from_upstream(kwargs,
> > key='num')
> >
> > print(f"Pulled XCom values: {value}")
> >
> >
> > t5 = PythonOperator(
> > task_id='task5',
> > python_callable=pull_that_xcom_via_macros,
> > )
> > ```
> >
> > It's able to fetch fine for a linear dag (you can improvise it as required
> > if you choose
> > to go down this way)
> >
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Fri, Apr 17, 2026 at 1:14 PM Amogh Desai <[email protected]> wrote:
> >
> > > Hello Gyorgy,
> > >
> > > As Ash and others mentioned, it was not unintended.
> > >
> > > The other reason that this was done was also because the older behaviour
> > > was expensive
> > > (scanning all tasks in the dag when task_ids = None) and it would create
> > > unclear semantics
> > > as well - which task's value do you get if multiple tasks push the same
> > > key?
> > >
> > > I am always leaning towards predictability over bad patterns carried over
> > > from legacy code, which
> > > is what we did with Airflow 3.
> > >
> > > I like the ideas presented by TP above if that is the case, otherwise a
> > > slightly more *involved *change
> > > would be to try and mimic airflow 2 behaviour using a macros + plugin
> > > setup (you can use:
> > > `kwargs["ti"].task.upstream_list` to get the upstream task ids to a
> > task)?
> > > Mind you, it might require
> > > some change in both cases.
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Fri, Apr 17, 2026 at 4:22 AM Tzu-ping Chung via dev <
> > > [email protected]> wrote:
> > >
> > >> Would using Variable be an alternative?
> > >>
> > >> From your description, it feels like your team is kind of using XCom as
> > a
> > >> key-value store, which Variable is designed to be. It’s also available
> > on
> > >> Airflow 2 so the migration should be straightforward.
> > >>
> > >> TP
> > >>
> > >> --
> > >> Sent from my iPhone
> > >>
> > >> > On 17 Apr 2026, at 00:37, Foldvari, Gyorgy via dev <
> > >> [email protected]> wrote:
> > >> >
> > >> > Hi Ash,
> > >> >
> > >> > To be very specific: My team is heavily relying on the Airflow 2
> > >> xcom_pull behavior when the key parameter is defined but the task_ids
> > >> parameter is not defined. In this case Airflow 2 returns the most recent
> > >> xcom value pushed by any task in that DAG run using that key, or None
> > if no
> > >> value has been pushed with that key.
> > >> > The change of the behavior introduced in Airflow 3 without any
> > >> equivalent alternative is a blocker for us to migrate to Airflow 3.
> > >> >
> > >> > One option could be to restore the original behavior. I see that there
> > >> will be no consensus on this, and I understand the reasons. Any other
> > >> solution would help if it can be used in template fields as well without
> > >> the need to change the code of classic Operators themselves.
> > >> >
> > >> > As far as I see we already have good proposals.
> > >> >
> > >> > Thanks,
> > >> > Gyorgy Foldvari
> > >> >
> > >> >
> > >> > Information Classification: GENERAL
> > >> > -----Original Message-----
> > >> > From: Ash Berlin-Taylor <[email protected]>
> > >> > Sent: Thursday, April 16, 2026 15:48
> > >> > To: [email protected]
> > >> > Subject: Re: [DISCUSS] Reconsidering `xcom_pull(task_ids=None)`
> > >> behavior change in Airflow 3
> > >> >
> > >> > [You don't often get email from [email protected]. Learn why this is
> > >> important at https://aka.ms/LearnAboutSenderIdentification ]
> > >> >
> > >> > CAUTION: This email originated from outside of the organization. Do
> > not
> > >> click links or open attachments unless you recognize the sender and know
> > >> the content is safe.
> > >> >
> > >> >
> > >> > Gyorgy,
> > >> >
> > >> > Your original post did not mention specifics, and is talking about
> > >> behaviour. The thing that is most relevant is what kind of workload
> > exists
> > >> that means you want this behaviour. XCom is already complicated enough
> > once
> > >> you get beyond “pull this specific task†without adding even more
> > bells
> > >> and whistles.
> > >> >
> > >> > So please be as specific and real as you can in saying why you want
> > >> this feature.
> > >> >
> > >> > Thanks,
> > >> > Ash
> > >> >
> > >> >> On 15 Apr 2026, at 21:25, Jens Scheffler <[email protected]>
> > wrote:
> > >> >>
> > >> >> I do not have the use case and prefer explicit pinning to the task
> > >> where I want XCom from. But maybe my/our use cases are structured and
> > >> repeatable that the questions does not hold.
> > >> >>
> > >> >> So if there is the demand for a non-deterministic retrieval of the
> > >> "OLDEST" / "NEWEST" (single) entry I think using the constant classes
> > is a
> > >> good approach not to overload the function signature. If this is well
> > >> documented in PyDoc then it is accessible. Would be OK for me.
> > >> >>
> > >> >> Otherwise a more generic solution could be providing an API which
> > >> accepts a list of "candidates" to poll meta data and get a lis and this
> > >> list is including dates. Then some logic could sort by date and retrieve
> > >> whatever es desired... if there are more niece use cases this might be
> > an
> > >> alternative. But the proposal is lightweight.
> > >> >>
> > >> >>> On 15.04.26 17:03, Jarek Potiuk wrote:
> > >> >>> I'd wait for others to comment. I want to see what others think.
> > >> >>>
> > >> >>> On Wed, Apr 15, 2026 at 4:51 PM Foldvari, Gyorgy via dev <
> > >> >>> [email protected]> wrote:
> > >> >>>
> > >> >>>> If xcom_pull(task_ids=XCom.ANY, key=†sample†) would return
> > with a
> > >> >>>> list, that would not be the same result what was produced by
> > Airflow
> > >> >>>> 2. Airflow 2 returns with the most recent value and not with a
> > list.
> > >> >>>> I am wondering what would make more sense:
> > >> >>>>
> > >> >>>> 1. Return with one value, the most recent one (on None) instead
> > >> >>>> of a list, copying Airflow 2 behavior.
> > >> >>>> 2. Return with a list but also defining a new optional parameter
> > >> >>>> to override this behavior (XCom.FIRST_OLDEST, XCom.FIRST_NEWEST).
> > >> >>>> I am struggling with coming up a valid use case where getting back
> > a
> > >> >>>> list of values without any further information would make sense. So
> > >> >>>> probably we should go with the simpler first option.
> > >> >>>>
> > >> >>>>
> > >> >>>> Information Classification: GENERAL
> > >> >>>>
> > >> >>>> On 2026/04/15 14:35:35 Jarek Potiuk wrote:
> > >> >>>>>> What do you think about this solution?
> > >> >>>>>> ti.xcom_pull accepting ‘*’ or rather a sentinel value as
> > >> task_ids
> > >> >>>>>> to
> > >> >>>>> explicitly say that ANY task_id should be considered. The result
> > >> >>>>> would be all the values pushed to xcom by any task in that dag run
> > >> >>>>> as a list,
> > >> >>>> sorted
> > >> >>>>> by XCOM timestamp.
> > >> >>>>>> ti.xcom_pull(task_ids= ANY, key=†example†)
> > >> >>>>>
> > >> >>>>> I'd rather (if others see a value for it) have some explicit
> > >> >>>>> sentinel (XCom.FiRST_OLDEST, XCom.FIRST_NEWEST - or whatever makes
> > >> >>>>> sense for
> > >> >>>> those).
> > >> >>>>>> The only challenge is how to use this in classic Operators’
> > >> >>>>>> templated
> > >> >>>>> fields.
> > >> >>>>>
> > >> >>>>> Those sentinels could be easily exposed to JINJA and predefined
> > >> there.
> > >> >>>>>
> > >> >>>>> It might of course be over-the-top, and some might say "ugly" -
> > but
> > >> >>>>> if it makes it easier for people to migrate to 3 and would not
> > make
> > >> >>>>> it more difficult for maintenance, I am all for it.
> > >> >>>>>
> > >> >>>>> J.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Wed, Apr 15, 2026 at 4:22 PM Foldvari, Gyorgy via dev <
> > >> >>>>> [email protected]> wrote:
> > >> >>>>>
> > >> >>>>>> What do you think about this solution?
> > >> >>>>>> ti.xcom_pull accepting ‘*’ or rather a sentinel value as
> > >> task_ids
> > >> >>>>>> to explicitly say that ANY task_id should be considered. The
> > >> >>>>>> result would
> > >> >>>> be
> > >> >>>>>> all the values pushed to xcom by any task in that dag run as a
> > >> >>>>>> list,
> > >> >>>> sorted
> > >> >>>>>> by XCOM timestamp.
> > >> >>>>>> ti.xcom_pull(task_ids= ANY, key=†example†) The only challenge
> > >> is
> > >> >>>>>> how to use this in classic Operators’ templated fields.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> Information Classification: GENERAL
> > >> >>>>>>
> > >> >>>>>> On 2026/04/15 13:34:58 Jarek Potiuk wrote:
> > >> >>>>>>>> This was ambiguous regarding the value you wanted to retrieve
> > >> >>>>>>>> and
> > >> >>>> the
> > >> >>>>>>> intended change.
> > >> >>>>>>>
> > >> >>>>>>> While I see why we cannot (should not) bring back the "exact"
> > >> >>>> behaviour,
> > >> >>>>>> I
> > >> >>>>>>> think we could - potentially at least - make it non-ambiguous
> > and
> > >> >>>>>>> predictable by defining the semantics and ordering and selecting
> > >> >>>>>>> the
> > >> >>>>>> first
> > >> >>>>>>> one. And - I think - we do not **have** to bring it back in this
> > >> >>>>>> particular
> > >> >>>>>>> method. Maybe we can just a new method call with defined
> > ordering
> > >> >>>>>>> and semantics - behaving predictably, similar to Airflow 2—and
> > >> >>>>>>> clearly
> > >> >>>>>> defined
> > >> >>>>>>> semantically. That would at least give people an easier way to
> > >> >>>> migrate?
> > >> >>>>>>> While I think the cat if out-of-the bag and we cannot truly
> > >> >>>>>>> revert
> > >> >>>> the
> > >> >>>>>>> change (because that would again potentially affect 3.0 - 3.2
> > >> >>>> users) -
> > >> >>>>>> but
> > >> >>>>>>> we could at least make it easier for people to cope with it
> > >> >>>>>>> without
> > >> >>>> too
> > >> >>>>>>> much hassle while waiting for the task state to be available in
> > >> >>>>>>> this particular case?
> > >> >>>>>>>
> > >> >>>>>>> Just a thought I had - ... listen to your users and do things
> > >> >>>>>>> easier
> > >> >>>> for
> > >> >>>>>>> them - without breaking our SemVer promises.
> > >> >>>>>>>
> > >> >>>>>>> J.
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Wed, Apr 15, 2026 at 3:20 PM Foldvari, Gyorgy via dev <
> > >> >>>>>>> [email protected]> wrote:
> > >> >>>>>>>
> > >> >>>>>>>> I do not want to use XCom for managing task state. It was just
> > a
> > >> >>>> very
> > >> >>>>>>>> simple - and seemingly misleading - example to explain the
> > >> >>>>>>>> original behavior.
> > >> >>>>>>>>
> > >> >>>>>>>> I gave details about valid use cases and issues caused by this
> > >> >>>> change
> > >> >>>>>> in
> > >> >>>>>>>> the behavior, in my original post. Those are not addressed by
> > >> >>>>>>>> the
> > >> >>>> AIP
> > >> >>>>>> you
> > >> >>>>>>>> are referring to.
> > >> >>>>>>>>
> > >> >>>>>>>> But that AIP would definitely address another valid use case
> > >> >>>>>>>> what
> > >> >>>> I am
> > >> >>>>>>>> missing especially for implementing stateful sensors, so I
> > >> >>>>>>>> really
> > >> >>>> hope
> > >> >>>>>> it
> > >> >>>>>>>> goes through and gets implemented.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> Information Classification: GENERAL -----Original Message-----
> > >> >>>>>>>> From: Ash Berlin-Taylor <[email protected]>
> > >> >>>>>>>> Sent: Wednesday, April 15, 2026 14:49
> > >> >>>>>>>> To: [email protected]
> > >> >>>>>>>> Subject: Re: [DISCUSS] Reconsidering `xcom_pull(task_ids=None)`
> > >> >>>>>> behavior
> > >> >>>>>>>> change in Airflow 3
> > >> >>>>>>>>
> > >> >>>>>>>> [You don't often get email from [email protected]. Learn why this
> > >> >>>>>>>> is important at https://aka.ms/LearnAboutSenderIdentification
> > ]
> > >> >>>>>>>>
> > >> >>>>>>>> CAUTION: This email originated from outside of the
> > organization.
> > >> >>>> Do not
> > >> >>>>>>>> click links or open attachments unless you recognize the sender
> > >> >>>>>>>> and
> > >> >>>>>> know
> > >> >>>>>>>> the content is safe.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> This was ambiguous as to what value you wanted to get, and an
> > >> >>>> intended
> > >> >>>>>>>> change.
> > >> >>>>>>>>
> > >> >>>>>>>> If you want this sort of behaviour, then you probably want to
> > >> >>>>>>>> look
> > >> >>>> at
> > >> >>>>
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
> > >> >>>> iki.apache.org
> > %2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-103%253A%2BTas
> > >> >>>> k%2BState%2BManagement&data=05%7C02%7CGyorgy.Foldvari%40msci.com
> > %7C3
> > >> >>>>
> > f431909517b460c5ea708de9bbed80d%7C7a9376d47c43480f82baa090647f651d%7
> > >> >>>>
> > C0%7C0%7C639119441164612772%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGki
> > >> >>>>
> > OnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy
> > >> >>>>
> > fQ%3D%3D%7C0%7C%7C%7C&sdata=yNOVBbNTbuEw6Niw%2B4ilPG33YqSAp8bEb%2Bf2
> > >> >>>> UDq6xSI%3D&reserved=0
> > >> >>>>>>>> which provides a dedicated way to manage state without many of
> > >> >>>>>>>> the
> > >> >>>>>> quirks
> > >> >>>>>>>> of XCom interface as it stands today. Reading between the
> > lines,
> > >> >>>>>>>> I
> > >> >>>>>> think
> > >> >>>>>>>> this API describes what you want?
> > >> >>>>>>>>
> > >> >>>>>>>> -ash
> > >> >>>>>>>>
> > >> >>>>>>>>> On 15 Apr 2026, at 12:45, Foldvari, Gyorgy via dev <
> > >> >>>>>>>> [email protected]> wrote:
> > >> >>>>>>>>> I see where the confusion is coming from, it is my mistake.
> > >> >>>>>>>>> Sorry
> > >> >>>>>> about
> > >> >>>>>>>> that.
> > >> >>>>>>>>> To clarify, I am taking about the use case where the key
> > >> >>>> parameter is
> > >> >>>>>>>> passed but the task_ids parameter is not or it is None.
> > >> >>>>>>>>> -----Original Message-----
> > >> >>>>>>>>> From: Foldvari, Gyorgy via dev <[email protected]>
> > >> >>>>>>>>> Sent: Wednesday, April 15, 2026 13:42
> > >> >>>>>>>>> To: [email protected]
> > >> >>>>>>>>> Cc: Foldvari, Gyorgy <[email protected]>
> > >> >>>>>>>>> Subject: RE: Re: [DISCUSS] Reconsidering
> > >> >>>> `xcom_pull(task_id
[message truncated...]
________________________________
Information regarding MSCI’s processing of personal data may be found at
www.msci.com/privacy-pledge. This email message and any attachments are for the
sole use of the intended recipients and may contain proprietary and/or
confidential information which may be privileged or otherwise protected from
disclosure. Any unauthorized review, use, disclosure or distribution is
prohibited. All rights and remedies are reserved. If you are not an intended
recipient, please contact the sender by reply email and destroy the original
message and any copies of the message as well as any attachments to the
original message. Local registered entity information:
https://www.msci.com/local-registered-entities
