Agreed with the design direction. I've pushed the change to return a lazy_object_proxy.Proxy. Would appreciate it if you could take a look.
Best regards, Jun Yeong 2026년 5월 6일 (수) 오전 12:59, Amogh Desai <[email protected]>님이 작성: > > Small correction on the design: Variable.keys() returning list[str] is > eager. What I had in > mind was a lazy proxy object: fetch the keys from the execution API once on > first access, cache > them, then serve iteration/indexing from that cache. Something like: > > keys = Variable.keys(prefix="team_a_config_") > for key in keys: > val = Variable.get(key) > > Similar to how lazy_object_proxy.Proxy is used in the template context > today. Keeps the call > site clean without pulling everything upfront. > > Re Ash's point about AIP 103 -- task state is for tasks persisting state > across retries, not a shared > config store. So I think you read it right. > > And forgot to comment earlier, but re Jens' workaround -- probably better > suited for a centralized static > config, but it will likely fall apart when multiple external systems are > independently writing their own keys. > The use case you have is legitimate and the workaround is honestly awkward > for it. > > Regarding secrets backend scoping, I think I am ok with it being on the > metadata db backend for the time > being as long as we document it well. > > Thanks & Regards, > Amogh Desai > > > On Tue, May 5, 2026 at 7:40 PM 김준영 <[email protected]> wrote: > > > Hi Ash, > > > > Regarding secrets backends—since they don't currently support listing > > in Airflow, I’ve scoped this feature to the metastore only and will > > document that explicitly. As for AIP-103, Task State is designed for > > tasks to persist their own state across retries, whereas this PR > > focuses on DAGs fetching externally-managed config at runtime, keeping > > the boundary between them clean. > > > > Best regards, > > Jun Yeong > > > > 2026년 5월 5일 (화) 오후 10:48, Ash Berlin-Taylor <[email protected]>님이 작성: > > > > > > Some of these use cases might be better suited to AIP-103 (the State > > storage AIP) > > > > > > One possible issue with exposing Variables.list() (and connections) — do > > the secrets backends support listing variables? > > > > > > -ash > > > > > > > On 5 May 2026, at 11:41, 김준영 <[email protected]> wrote: > > > > > > > > Hi Amogh, > > > > > > > > Thanks for the feedback! I've updated the PR (#66022) to implement the > > > > lazy iterator pattern you suggested: > > > > > > > > for key in Variable.keys(prefix="team_a_config_"): > > > > val = Variable.get(key) > > > > > > > > Now, Variable.keys(prefix=None) returns a list[str] of matching keys, > > > > supported by a new GET /variables/keys?prefix= endpoint in the > > > > Execution API. This allows callers to fetch only the specific values > > > > they need via Variable.get(key). > > > > > > > > Regarding your point about other Airflow resources (Connections, > > > > XComs, etc.) having the same gap: once this PR is merged and the > > > > pattern is established, I'd be happy to extend this keys() approach to > > > > those resources in follow-up PRs. Would that be a direction you'd > > > > encourage? > > > > > > > > The PR is ready for another review whenever you have a moment. > > > > > > > > Best regards, > > > > Jun Yeong > > > > > > > > 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <[email protected]>님이 작성: > > > >> > > > >> Hi Jun Yeong, > > > >> > > > >> Valid gap. There is no GET /variables list endpoint at all, but that > > is > > > >> true for any > > > >> other Airflow artefacts too. So variables isn't the only one missing > > it, > > > >> its all of the > > > >> Airflow resources: connections, xcoms, etc. > > > >> > > > >> One suggestion worth considering: rather than Variable.list() > > returning a > > > >> full list, > > > >> > > > >> a lazy iterator might be a better fit. Something like: > > > >> > > > >> > > > >> for key in Variable.keys(prefix="team_a_config_"): > > > >> val = Variable.get(key) > > > >> > > > >> Happy to review when a PR is ready. > > > >> > > > >> Thanks & Regards, > > > >> Amogh Desai > > > >> > > > >> > > > >> On Fri, May 1, 2026 at 3:02 AM 김준영 <[email protected]> wrote: > > > >> > > > >>> Hi Jens, > > > >>> > > > >>> That is a valid alternative, and using a single JSON variable is > > > >>> indeed a great pattern for centralized or static configurations to > > > >>> reduce I/O. > > > >>> > > > >>> However, for the use cases I'm targeting, the "Single JSON Variable" > > > >>> approach has a few significant drawbacks: > > > >>> > > > >>> 1.Concurrency & Atomic Updates: When multiple external systems or > > > >>> independent CI/CD pipelines need to update their own configurations, > > a > > > >>> single JSON variable creates a race condition. They would have to > > > >>> "Read-Modify-Write" the entire blob, which risks overwriting each > > > >>> other's changes. Separate variables allow for atomic, independent > > > >>> updates. > > > >>> > > > >>> 2.Integration Complexity: Many users integrate Airflow with external > > > >>> tools that independently push values to the Airflow Metadata DB or > > > >>> Secret Backend. Forcing these decoupled systems to coordinate and > > > >>> maintain a single shared JSON structure adds significant integration > > > >>> overhead. > > > >>> > > > >>> 3.Data Modeling Flexibility: While a JSON blob works for some, > > > >>> Variable.list(prefix=) allows Airflow to be unopinionated about how > > > >>> users model their data. It provides a standard "Key-Value store" > > > >>> experience (similar to AWS SSM or Redis) where prefix-based discovery > > > >>> is a first-class citizen. > > > >>> > > > >>> In short, while the JSON approach is a good workaround for specific > > > >>> cases, Variable.list() provides the necessary flexibility for highly > > > >>> dynamic and decoupled environments without forcing a specific data > > > >>> structure on the user. > > > >>> > > > >>> What do you think? > > > >>> > > > >>> Best regards, Jun Yeong > > > >>> > > > >>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <[email protected]>님이 작성: > > > >>>> > > > >>>> Hi Jun Yeong, > > > >>>> > > > >>>> one thought on this: We had similar. > > > >>>> > > > >>>> Our use case: Implemented a custom archiving (Dag) that needed to > > take > > > >>>> care of different retention times in different Dags (maybe bad > > example > > > >>>> because this archival itself accesses the database for archiving... > > > >>>> haha) and we wanted to have different retention times per Dag. > > > >>>> > > > >>>> What we did is we made a JSON structure with all parameters into a > > > >>>> single Variable. Then we did not need to have many IO operations and > > > >>>> Variables but could store all in a single Variable. > > > >>>> > > > >>>> Would this be a viable solution for your case as well? > > > >>>> > > > >>>> Jens > > > >>>> > > > >>>> On 30.04.26 22:47, 김준영 wrote: > > > >>>>> Hi Jens, > > > >>>>> > > > >>>>> Thank you for the thoughtful feedback! > > > >>>>> > > > >>>>> The primary demand for this feature comes from workflows that > > require > > > >>>>> dynamic configuration discovery. A common pattern is grouping > > related > > > >>>>> variables under a shared prefix (e.g., team_a_config_ or > > > >>>>> pipeline_x_param_). In many cases, these keys are generated or > > updated > > > >>>>> dynamically by external systems, meaning the exact set of keys > > isn't > > > >>>>> known at DAG authoring time. > > > >>>>> > > > >>>>> While users in Airflow 2.x relied on session.query(Variable).all() > > as > > > >>>>> a workaround, Airflow 3’s move toward the Task SDK aims to abstract > > > >>>>> away direct ORM/DB access for better security and stability. > > > >>>>> Variable.list(prefix=) provides a supported, clean way to achieve > > this > > > >>>>> discovery within that new architecture. > > > >>>>> > > > >>>>> Regarding the secrets backend limitation, I completely agree. It’s > > > >>>>> important to manage expectations, so I will update the PyDoc to > > > >>>>> explicitly state that this method only lists variables stored in > > the > > > >>>>> metadata database. > > > >>>>> > > > >>>>> Best regards, Jun Yeong > > > >>>>> > > > >>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <[email protected]>님이 > > 작성: > > > >>>>>> Hi! > > > >>>>>> > > > >>>>>> thanks for the discussion. While I am not against this I would > > say in > > > >>>>>> Airflow 2 it was also not a "public API" but the DB connecton > > "just > > > >>>>>> used" to list and have a missing API compensated. > > > >>>>>> > > > >>>>>> Can you express what the demand for the missing feature is? What > > > >>>>>> business function did you implement based on listing all > > Variables? > > > >>>>>> > > > >>>>>> As you already stated and also highlighted in the PR the list() > > might > > > >>>>>> not tell about all Variables as the list is not provided from > > secret > > > >>>>>> managers. So it might (small risk thoug) lead to some confusion. > > > >>> Should > > > >>>>>> be explicitly documented in the PyDoc. But this is a nit. > > > >>>>>> > > > >>>>>> Jens > > > >>>>>> > > > >>>>>> On 30.04.26 11:47, 김준영 wrote: > > > >>>>>>> Hi all, > > > >>>>>>> > > > >>>>>>> I'd like to propose adding Variable.list() to the Task SDK to > > address > > > >>>>>>> the gap left by the removal of direct ORM access in Airflow 3. > > > >>>>>>> > > > >>>>>>> Background: > > > >>>>>>> In Airflow 2.x, users could list all variables via: > > > >>>>>>> > > > >>>>>>> from airflow.models import Variable > > > >>>>>>> from airflow.utils.session import create_session > > > >>>>>>> > > > >>>>>>> with create_session() as session: > > > >>>>>>> variables = session.query(Variable).all() > > > >>>>>>> > > > >>>>>>> In Airflow 3.x, this pattern raises: > > > >>>>>>> RuntimeError: Direct database access via the ORM is not > > > >>> allowed in Airflow > > > >>>>>>> 3.0 > > > >>>>>>> > > > >>>>>>> There is currently no supported way to discover variable keys > > > >>> dynamically > > > >>>>>>> when they are not known at DAG authoring time. > > > >>>>>>> > > > >>>>>>> Proposal: > > > >>>>>>> - Add Variable.list(prefix=None) to the Task SDK > > > >>>>>>> - Scope is limited to the metadata database only (same as the > > > >>> old ORM pattern) > > > >>>>>>> - Secrets backend support is intentionally out of scope, as it > > > >>> would > > > >>>>>>> require a broader interface contract change and separate > > > >>> community > > > >>>>>>> discussion > > > >>>>>>> > > > >>>>>>> Related issue: https://github.com/apache/airflow/issues/61166 > > > >>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022 > > > >>>>>>> > > > >>>>>>> I would appreciate any feedback or concerns from the community > > > >>> before > > > >>>>>>> this moves forward. > > > >>>>>>> > > > >>>>>>> Best Regards, > > > >>>>>>> Jun Yeong Kim > > > >>>>>>> > > > >>>>>>> > > --------------------------------------------------------------------- > > > >>>>>>> To unsubscribe, e-mail: [email protected] > > > >>>>>>> For additional commands, e-mail: [email protected] > > > >>>>>>> > > > >>>>>> > > --------------------------------------------------------------------- > > > >>>>>> To unsubscribe, e-mail: [email protected] > > > >>>>>> For additional commands, e-mail: [email protected] > > > >>>>>> > > > >>>>> > > --------------------------------------------------------------------- > > > >>>>> To unsubscribe, e-mail: [email protected] > > > >>>>> For additional commands, e-mail: [email protected] > > > >>>>> > > > >>>> > > > >>>> > > --------------------------------------------------------------------- > > > >>>> To unsubscribe, e-mail: [email protected] > > > >>>> For additional commands, e-mail: [email protected] > > > >>>> > > > >>> > > > >>> --------------------------------------------------------------------- > > > >>> To unsubscribe, e-mail: [email protected] > > > >>> For additional commands, e-mail: [email protected] > > > >>> > > > >>> > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
