Hello everyone, Reviving this thread with an apology. I merged the implementation of this feature (https://github.com/apache/airflow/pull/70681) last week because I thought this discussion had been settled. Re-reading it, it was not: it stopped at "here is what is needed", nobody summarised it, and Elad's concern never got an answer. Amogh rightly reverted the change (https://github.com/apache/airflow/pull/73891), both for that reason and for two real bugs he found. Sorry about that. Let me try to close the loop properly here.
Summary of the discussion so far (July 2025): - Ash: OK with making the order fully configurable, as long as we document it, and keeping in mind that the API server/scheduler and the workers resolve secrets differently. - Me: in favour. The main use case is putting the metadata DB first so that frequently used connections stop generating secrets-manager calls (cost and latency). Conditions: good docs, and the order shown in the UI. - Amogh: makes sense in theory, but needs clearer docs, a way to find out the order when debugging, and handling of the worker-side backends. - Elad: against as proposed. The setting creates a blind spot for Dag authors, who cannot see what order the deployment uses. - Anton asked what was needed to merge; the answer was docs plus a read-only display of the order on the Connections and Variables pages, never editable from the UI. What the implementation does now (https://github.com/apache/airflow/pull/74007, re-applying #70681 with fixes): - Two options: [secrets] backends_order for server components and [workers] backends_order for task workers, covering the worker angle Ash and Amogh raised. The defaults reproduce today's order exactly, so nothing changes unless a Deployment Manager opts in. - Docs describing each backend, what is required, and the trade-offs. - A misconfigured value no longer breaks every airflow CLI command. The backends are now loaded on first use, so "airflow config get-value" and "airflow version" still work and can be used to debug it (Amogh's finding after the merge). Elad, to answer your question about Dag authors directly: the configured [secrets] backends_order is shown read-only on both the Connections and the Variables pages, to every logged-in user. It is not limited to users who can see the configuration (expose_config). So anyone looking at connections or variables in the UI can see that the metadata DB may not be the source that wins, and in which order the sources are checked, without asking the Deployment Manager. One gap I am aware of: the UI shows the server-side order, not [workers] backends_order, which is what tasks actually use at run time. I think we should show both, and I am happy to add that to the same PR if you agree. Possibly also we should fail airflow of the two are different - though eventually they might be even configured in different config files when we complete the Dag processor/task ask separation. Are there any remaining concerns with the approach, or with https://github.com/apache/airflow/pull/74007? If not, I would love follow up with merging and a lazy consensus to make it on time for 3.4.0. J.
