Hey Sameer and everyone, +1 to the revised scope in #74915.
The need has come up several times: #61645, discussion #59799, #63928, and #71111 all asked for this in different forms. I think it’s a good time to add the feature. I think it can proceed without an AIP for the following reasons: 1. The revised scope adds no pluggable abstraction, config entry, API, or DB migration. Reloading uses the existing dag_bundle_config_list format, supplied through Airflow’s existing _cmd / _secret mechanism. The PR makes that mechanism available for this key by marking it as sensitive. 2. It’s opt-in. A value set through a plain environment variable or airflow.cfg keeps today’s startup-only behavior. _CMD / _SECRET were not available for this key before, so no existing deployment will start reloading without an explicit change. 3. The trust boundary does not change. Bundle definitions remain under the Deployment Manager’s control; users cannot change them through Airflow. Neither of the two previous threads has received a reply yet. Sameer, I suggest waiting few more days (let's say until Monday, Nov. 2nd) to give a chance for others to comment on this thread, and following up with a [LAZY CONSENSUS] thread on the revised scope if there are no responses by then, or there's consensus. Disclosure: I helped Sameer narrow the scope and I’m reviewing the PR, so please take that into account when weighing my comment. Shahar On Sun, Oct 25, 2026 at 12:33 PM Sameer Raj via dev <[email protected]> wrote: > Hi everyone! > > Following up on this discussion, I would like to share the revised scope. > The earlier proposal included a DagBundleProvider abstraction. I have > dropped that abstraction from the current work. It can be proposed > separately through a future AIP if needed. The new PR is limited to > reloading the existing [dag_processor] dag_bundle_config_list through > Airflow’s_CMD or_SECRET mechanism: > > https://github.com/apache/airflow/pull/74915 > > Thanks to Shahar for helping me narrow down the scope and for reviewing the > approach. > > In our use case, we need to manage more than 500 DAG bundles, with > additions being automated. Restarting or redeploying Airflow processes > whenever a bundle is added makes this difficult. We want to fetch the > bundle list from an internal API. A deployment managed command can call > that API and return the same JSON list already accepted by > dag_bundle_config_list. The command could also read a file, or the list > could come from a secrets backend. > > The DAG processor checks for changes at > bundle_refresh_check_interval. Users do not need to change their > configuration unless they want to enable reload. Users only need to > configure _CMD or _SECRET if they want reload. > > The expected behavior is: > > - When a bundle is added, the Dag processor discovers it and starts > processing its files. Existing bundle instances are reused. > - When a bundle is removed, the processor stops processing it. An > unfiltered processor (without --bundle-name) marks it inactive, and the > existing stale DAG scan marks its dags stale. A successful read > containing > [] removes all bundles. > - Failed commands, unavailable secrets, wrong configs, invalid new > classpaths, and blank reads keep the last accepted list in that process. > The source must return the complete list. Commands should return a non > zero > exit on failure, rather than returning []. > - A fresh process has no previous list to fall back to. Invalid or > unavailable configuration prevents startup. A valid [] starts with no > bundles. > - Processors started with --bundle-name stop processing removed bundles > locally, but never deactivate missing bundles globally (retaining > existing > behaviour). A processor that sees the complete list is needed for global > deactivation. > - Multiple Dag processors may see changes at different times (due to > variance in the refresh interval). Each unfiltered processor reconciles > bundle in the database even when its own list has not changed, so it can > correct a stale write from another processor. They eventually converge > when > surviving processors see the same list and continue reconciling. > > All components, i.e. the DAG processor and workers that resolve bundles > need access to the configuration source. Existing Dag runs should finish > before removing a bundle if those runs need to complete, since tasks and > callbacks can still need its definition. > > For v1, changing an existing bundle’s definition is rejected, and the last > accepted list is retained. This includes changing tracking_ref or classpath > etc. Such a change requires a new bundle name or restarting the affected > components. New commits on the already configured branch continue to be > picked up through the existing bundle refresh behavior. Supporting changes > to an existing bundle definition can be added later incase required! > > There is also an optional DagFileParseRequest.bundle_config field > containing the accepted importer settings and team ownership. It allows the > parsing process (to parse the DAG) to use the configuration accepted by its > parent, without reading the command or secrets backend again and > potentially seeing a different list. This does change the parser protocol > but the compatibility handling is included in the PR > > The dag_bundle_config_list option is marked sensitive: true to enable _CMD > and _SECRET. This also means the option is hidden in configuration views > that hide sensitive values. The Deployment Manager continues to control the > command or secrets backend and its credentials. No API for creating or > changing bundle definitions is added. > > Related discussions by various users: > > - Runtime bundle management: > https://github.com/apache/airflow/issues/61645 > - Automatic bundle configuration loading: > https://github.com/apache/airflow/discussions/59799 > - Earlier file-based reload PR: > https://github.com/apache/airflow/pull/71111 > - Earlier dev-list discussion on runtime reload: > https://lists.apache.org/thread/co0z6yxc1g6ctqqmt7rdszjhjsq0w2hx > > I would appreciate feedback on this revised scope, proposed flow and the > state v1 limitation. > > Thanks, > Sameer Raj > > On Wed, Sep 23, 2026 at 3:35 PM Sameer Raj <[email protected]> wrote: > > > Hi everyone! > > I would like to discuss adding support for dynamically discovering new > DAG > > bundle configurations without requiring Airflow restarts. (following up > on > > comment: > > https://github.com/apache/airflow/issues/61645#issuecomment-5017217627) > > > > There's also multiple PRs/discussion going over for the same feature: > > - Runtime CRUD proposal: > > https://github.com/apache/airflow/issues/61645 > > - Automatic bundle configuration loading: > > https://github.com/apache/airflow/discussions/59799 > > - File-based hot-reload PR: > > https://github.com/apache/airflow/pull/63928 > > - Add dynamic DAG bundle configuration from file path > > https://github.com/apache/airflow/pull/71111 > > > > *The Problem:* > > > > Today, Dag bundles are configured through [dag_processor] > > dag_bundle_config_list. DagBundlesManager loads this configuration when > > the process starts, and the Dag processor uses the same bundle list. > > > > This becomes operationally difficult for deployments that manage many > > bundles and regularly add (or remove) them. Each configuration change > > currently requires restarting or redeploying the affected Airflow > processes. > > > > *Proposal* > > > > The proposal has two parts: > > > > First, introduce a DagBundleProvider abstraction into DagBundlesManager. > > It would allow Airflow to discover the bundle configurations construct > > bundles accordingly. Thus, the responsibility to fetch bundles is passed > on > > to DagBundleProvider by DagBundleManager. > > > > The configured class could obtain bundle information from a file, blob > > storage, an internal service, or another deployment-specific source (and > > this will decouple Airflow being aware of the source). Airflow would > define > > the class contract without adding direct support for every possible > source. > > > > The existing dag_bundle_config_list would remain the default > > implementation, so current deployments would keep the same behavior. > > > > Second, allow the Dag processor to periodically ask the configured class > > for the latest bundle configurations from its existing processing loop. > > > > When a bundle is added, the Dag processor would discover it and begin > > processing its files. When a bundle is removed, the Dag processor would > > stop processing it and use the existing removed-file handling for its > Dags. > > > > Each Dag processor performs this reconciliation periodically. Processors > > may observe a change at slightly different times, but they will converge > on > > the configuration returned by the provider. > > > > *Why use a configurable class?* > > > > In this comment > > <https://github.com/apache/airflow/issues/61645#issuecomment-5017217627 > >, > > Jarek asked why a custom dynamic manager would be preferable to managing > > Airflow configuration dynamically. > > > > On this, having this abstraction can allow users to implement their way > by > > which they plan to reload the configs, for example, fetch from external > > system etc. Also changing the environment variable for bundle list wont > > automatically reload the bundles, and the capability need to be built for > > the same. > > > > *Security* > > > > The security implications of changing bundle definitions through the > > Airflow API were raised in issue #61645 > > <https://github.com/apache/airflow/issues/61645#issuecomment-4212433885 > >. > > > > This proposal keeps management of bundle definitions within the existing > > Deployment Manager trust boundary. > > > > - The Deployment Manager selects and deploys the DagBundleProvider > > implementation. > > - The Deployment Manager controls access to its configuration source. > > - Credentials and authorization for that source remain deployment > > managed. > > - The Airflow API does not expose operations for creating or changing > > bundle definitions. > > - No new Airflow user role is introduced for managing bundles. > > > > *Compatibility* > > > > The existing dag_bundle_config_list remains the default source and keeps > > its current behavior. > > > > *Value Addition* > > > > Multiple users have raised PR/discuss thread on Airflow Github for the > > same, and this feature will be very useful for the community who are > > managing multiple bundles. Reference for the threads is shared above. > > > > *Current work* > > > > I have split the work into two PRs so each part can be reviewed > separately: > > > > - Provider abstraction: > > https://github.com/apache/airflow/pull/73209 > > - Runtime configuration reload: > > https://github.com/apache/airflow/pull/73217 > > > > I would appreciate feedback on this! > > > > Thanks, > > Sameer Raj > > > > > > > > > > >
