Small correction on the design: Variable.keys() returning list[str] is
eager. What I had in
mind was a lazy proxy object: fetch the keys from the execution API once on
first access, cache
them, then serve iteration/indexing from that cache. Something like:
keys = Variable.keys(prefix="team_a_config_")
for key in keys:
val = Variable.get(key)
Similar to how lazy_object_proxy.Proxy is used in the template context
today. Keeps the call
site clean without pulling everything upfront.
Re Ash's point about AIP 103 -- task state is for tasks persisting state
across retries, not a shared
config store. So I think you read it right.
And forgot to comment earlier, but re Jens' workaround -- probably better
suited for a centralized static
config, but it will likely fall apart when multiple external systems are
independently writing their own keys.
The use case you have is legitimate and the workaround is honestly awkward
for it.
Regarding secrets backend scoping, I think I am ok with it being on the
metadata db backend for the time
being as long as we document it well.
Thanks & Regards,
Amogh Desai
On Tue, May 5, 2026 at 7:40 PM 김준영 <[email protected]> wrote:
> Hi Ash,
>
> Regarding secrets backends—since they don't currently support listing
> in Airflow, I’ve scoped this feature to the metastore only and will
> document that explicitly. As for AIP-103, Task State is designed for
> tasks to persist their own state across retries, whereas this PR
> focuses on DAGs fetching externally-managed config at runtime, keeping
> the boundary between them clean.
>
> Best regards,
> Jun Yeong
>
> 2026년 5월 5일 (화) 오후 10:48, Ash Berlin-Taylor <[email protected]>님이 작성:
> >
> > Some of these use cases might be better suited to AIP-103 (the State
> storage AIP)
> >
> > One possible issue with exposing Variables.list() (and connections) — do
> the secrets backends support listing variables?
> >
> > -ash
> >
> > > On 5 May 2026, at 11:41, 김준영 <[email protected]> wrote:
> > >
> > > Hi Amogh,
> > >
> > > Thanks for the feedback! I've updated the PR (#66022) to implement the
> > > lazy iterator pattern you suggested:
> > >
> > > for key in Variable.keys(prefix="team_a_config_"):
> > > val = Variable.get(key)
> > >
> > > Now, Variable.keys(prefix=None) returns a list[str] of matching keys,
> > > supported by a new GET /variables/keys?prefix= endpoint in the
> > > Execution API. This allows callers to fetch only the specific values
> > > they need via Variable.get(key).
> > >
> > > Regarding your point about other Airflow resources (Connections,
> > > XComs, etc.) having the same gap: once this PR is merged and the
> > > pattern is established, I'd be happy to extend this keys() approach to
> > > those resources in follow-up PRs. Would that be a direction you'd
> > > encourage?
> > >
> > > The PR is ready for another review whenever you have a moment.
> > >
> > > Best regards,
> > > Jun Yeong
> > >
> > > 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <[email protected]>님이 작성:
> > >>
> > >> Hi Jun Yeong,
> > >>
> > >> Valid gap. There is no GET /variables list endpoint at all, but that
> is
> > >> true for any
> > >> other Airflow artefacts too. So variables isn't the only one missing
> it,
> > >> its all of the
> > >> Airflow resources: connections, xcoms, etc.
> > >>
> > >> One suggestion worth considering: rather than Variable.list()
> returning a
> > >> full list,
> > >>
> > >> a lazy iterator might be a better fit. Something like:
> > >>
> > >>
> > >> for key in Variable.keys(prefix="team_a_config_"):
> > >> val = Variable.get(key)
> > >>
> > >> Happy to review when a PR is ready.
> > >>
> > >> Thanks & Regards,
> > >> Amogh Desai
> > >>
> > >>
> > >> On Fri, May 1, 2026 at 3:02 AM 김준영 <[email protected]> wrote:
> > >>
> > >>> Hi Jens,
> > >>>
> > >>> That is a valid alternative, and using a single JSON variable is
> > >>> indeed a great pattern for centralized or static configurations to
> > >>> reduce I/O.
> > >>>
> > >>> However, for the use cases I'm targeting, the "Single JSON Variable"
> > >>> approach has a few significant drawbacks:
> > >>>
> > >>> 1.Concurrency & Atomic Updates: When multiple external systems or
> > >>> independent CI/CD pipelines need to update their own configurations,
> a
> > >>> single JSON variable creates a race condition. They would have to
> > >>> "Read-Modify-Write" the entire blob, which risks overwriting each
> > >>> other's changes. Separate variables allow for atomic, independent
> > >>> updates.
> > >>>
> > >>> 2.Integration Complexity: Many users integrate Airflow with external
> > >>> tools that independently push values to the Airflow Metadata DB or
> > >>> Secret Backend. Forcing these decoupled systems to coordinate and
> > >>> maintain a single shared JSON structure adds significant integration
> > >>> overhead.
> > >>>
> > >>> 3.Data Modeling Flexibility: While a JSON blob works for some,
> > >>> Variable.list(prefix=) allows Airflow to be unopinionated about how
> > >>> users model their data. It provides a standard "Key-Value store"
> > >>> experience (similar to AWS SSM or Redis) where prefix-based discovery
> > >>> is a first-class citizen.
> > >>>
> > >>> In short, while the JSON approach is a good workaround for specific
> > >>> cases, Variable.list() provides the necessary flexibility for highly
> > >>> dynamic and decoupled environments without forcing a specific data
> > >>> structure on the user.
> > >>>
> > >>> What do you think?
> > >>>
> > >>> Best regards, Jun Yeong
> > >>>
> > >>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <[email protected]>님이 작성:
> > >>>>
> > >>>> Hi Jun Yeong,
> > >>>>
> > >>>> one thought on this: We had similar.
> > >>>>
> > >>>> Our use case: Implemented a custom archiving (Dag) that needed to
> take
> > >>>> care of different retention times in different Dags (maybe bad
> example
> > >>>> because this archival itself accesses the database for archiving...
> > >>>> haha) and we wanted to have different retention times per Dag.
> > >>>>
> > >>>> What we did is we made a JSON structure with all parameters into a
> > >>>> single Variable. Then we did not need to have many IO operations and
> > >>>> Variables but could store all in a single Variable.
> > >>>>
> > >>>> Would this be a viable solution for your case as well?
> > >>>>
> > >>>> Jens
> > >>>>
> > >>>> On 30.04.26 22:47, 김준영 wrote:
> > >>>>> Hi Jens,
> > >>>>>
> > >>>>> Thank you for the thoughtful feedback!
> > >>>>>
> > >>>>> The primary demand for this feature comes from workflows that
> require
> > >>>>> dynamic configuration discovery. A common pattern is grouping
> related
> > >>>>> variables under a shared prefix (e.g., team_a_config_ or
> > >>>>> pipeline_x_param_). In many cases, these keys are generated or
> updated
> > >>>>> dynamically by external systems, meaning the exact set of keys
> isn't
> > >>>>> known at DAG authoring time.
> > >>>>>
> > >>>>> While users in Airflow 2.x relied on session.query(Variable).all()
> as
> > >>>>> a workaround, Airflow 3’s move toward the Task SDK aims to abstract
> > >>>>> away direct ORM/DB access for better security and stability.
> > >>>>> Variable.list(prefix=) provides a supported, clean way to achieve
> this
> > >>>>> discovery within that new architecture.
> > >>>>>
> > >>>>> Regarding the secrets backend limitation, I completely agree. It’s
> > >>>>> important to manage expectations, so I will update the PyDoc to
> > >>>>> explicitly state that this method only lists variables stored in
> the
> > >>>>> metadata database.
> > >>>>>
> > >>>>> Best regards, Jun Yeong
> > >>>>>
> > >>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <[email protected]>님이
> 작성:
> > >>>>>> Hi!
> > >>>>>>
> > >>>>>> thanks for the discussion. While I am not against this I would
> say in
> > >>>>>> Airflow 2 it was also not a "public API" but the DB connecton
> "just
> > >>>>>> used" to list and have a missing API compensated.
> > >>>>>>
> > >>>>>> Can you express what the demand for the missing feature is? What
> > >>>>>> business function did you implement based on listing all
> Variables?
> > >>>>>>
> > >>>>>> As you already stated and also highlighted in the PR the list()
> might
> > >>>>>> not tell about all Variables as the list is not provided from
> secret
> > >>>>>> managers. So it might (small risk thoug) lead to some confusion.
> > >>> Should
> > >>>>>> be explicitly documented in the PyDoc. But this is a nit.
> > >>>>>>
> > >>>>>> Jens
> > >>>>>>
> > >>>>>> On 30.04.26 11:47, 김준영 wrote:
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> I'd like to propose adding Variable.list() to the Task SDK to
> address
> > >>>>>>> the gap left by the removal of direct ORM access in Airflow 3.
> > >>>>>>>
> > >>>>>>> Background:
> > >>>>>>> In Airflow 2.x, users could list all variables via:
> > >>>>>>>
> > >>>>>>> from airflow.models import Variable
> > >>>>>>> from airflow.utils.session import create_session
> > >>>>>>>
> > >>>>>>> with create_session() as session:
> > >>>>>>> variables = session.query(Variable).all()
> > >>>>>>>
> > >>>>>>> In Airflow 3.x, this pattern raises:
> > >>>>>>> RuntimeError: Direct database access via the ORM is not
> > >>> allowed in Airflow
> > >>>>>>> 3.0
> > >>>>>>>
> > >>>>>>> There is currently no supported way to discover variable keys
> > >>> dynamically
> > >>>>>>> when they are not known at DAG authoring time.
> > >>>>>>>
> > >>>>>>> Proposal:
> > >>>>>>> - Add Variable.list(prefix=None) to the Task SDK
> > >>>>>>> - Scope is limited to the metadata database only (same as the
> > >>> old ORM pattern)
> > >>>>>>> - Secrets backend support is intentionally out of scope, as it
> > >>> would
> > >>>>>>> require a broader interface contract change and separate
> > >>> community
> > >>>>>>> discussion
> > >>>>>>>
> > >>>>>>> Related issue: https://github.com/apache/airflow/issues/61166
> > >>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022
> > >>>>>>>
> > >>>>>>> I would appreciate any feedback or concerns from the community
> > >>> before
> > >>>>>>> this moves forward.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jun Yeong Kim
> > >>>>>>>
> > >>>>>>>
> ---------------------------------------------------------------------
> > >>>>>>> To unsubscribe, e-mail: [email protected]
> > >>>>>>> For additional commands, e-mail: [email protected]
> > >>>>>>>
> > >>>>>>
> ---------------------------------------------------------------------
> > >>>>>> To unsubscribe, e-mail: [email protected]
> > >>>>>> For additional commands, e-mail: [email protected]
> > >>>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: [email protected]
> > >>>>> For additional commands, e-mail: [email protected]
> > >>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: [email protected]
> > >>>> For additional commands, e-mail: [email protected]
> > >>>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: [email protected]
> > >>> For additional commands, e-mail: [email protected]
> > >>>
> > >>>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [email protected]
> > > For additional commands, e-mail: [email protected]
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
