Hi Sumit, Thanks for the feedback. I agree an interface would be better for the long term. For example, a deployment could read bundle definitions directly from S3 or an internal service, without wrapping the source in a command.
A new public extension point would be loaded by every component, so it needs an AIP (or an amendment to AIP-66) as discussed in #73209. AIP-85 for Dag importers is a precedent. I suggest delivering the _cmd/_secret reload now to main (maybe we could even backport to 3.4.1?). If you or Sameer want to pursue the interface, let's start a separate AIP. Shahar On Fri, Oct 30, 2026 at 7:40 PM Sumit Maheshwari <[email protected]> wrote: > Personally, I prefer the interface approach as it aligns better with > enterprise-grade software. That said, I am also fine with proceeding via > the _cmd and _secret route for the time being. > > Thanks, > Sumit > > On Fri, Oct 30, 2026 at 11:47 AM Shahar Epstein <[email protected]> wrote: > > > Hey Sameer and everyone, > > > > +1 to the revised scope in #74915. > > > > The need has come up several times: #61645, discussion #59799, #63928, > and > > #71111 all asked for this in different forms. I think it’s a good time to > > add the feature. > > > > I think it can proceed without an AIP for the following reasons: > > > > 1. The revised scope adds no pluggable abstraction, config entry, API, or > > DB migration. Reloading uses the existing dag_bundle_config_list format, > > supplied through Airflow’s existing _cmd / _secret mechanism. The PR > makes > > that mechanism available for this key by marking it as sensitive. > > 2. It’s opt-in. A value set through a plain environment variable or > > airflow.cfg keeps today’s startup-only behavior. _CMD / _SECRET were not > > available for this key before, so no existing deployment will start > > reloading without an explicit change. > > 3. The trust boundary does not change. Bundle definitions remain under > the > > Deployment Manager’s control; users cannot change them through Airflow. > > > > Neither of the two previous threads has received a reply yet. Sameer, I > > suggest waiting few more days (let's say until Monday, Nov. 2nd) to give > a > > chance for others to comment on this thread, and following up with a > [LAZY > > CONSENSUS] thread on the revised scope if there are no responses by then, > > or there's consensus. > > > > Disclosure: I helped Sameer narrow the scope and I’m reviewing the PR, so > > please take that into account when weighing my comment. > > > > > > Shahar > > > > On Sun, Oct 25, 2026 at 12:33 PM Sameer Raj via dev < > > [email protected]> > > wrote: > > > > > Hi everyone! > > > > > > Following up on this discussion, I would like to share the revised > scope. > > > The earlier proposal included a DagBundleProvider abstraction. I have > > > dropped that abstraction from the current work. It can be proposed > > > separately through a future AIP if needed. The new PR is limited to > > > reloading the existing [dag_processor] dag_bundle_config_list through > > > Airflow’s_CMD or_SECRET mechanism: > > > > > > https://github.com/apache/airflow/pull/74915 > > > > > > Thanks to Shahar for helping me narrow down the scope and for reviewing > > the > > > approach. > > > > > > In our use case, we need to manage more than 500 DAG bundles, with > > > additions being automated. Restarting or redeploying Airflow processes > > > whenever a bundle is added makes this difficult. We want to fetch the > > > bundle list from an internal API. A deployment managed command can call > > > that API and return the same JSON list already accepted by > > > dag_bundle_config_list. The command could also read a file, or the list > > > could come from a secrets backend. > > > > > > The DAG processor checks for changes at > > > bundle_refresh_check_interval. Users do not need to change their > > > configuration unless they want to enable reload. Users only need to > > > configure _CMD or _SECRET if they want reload. > > > > > > The expected behavior is: > > > > > > - When a bundle is added, the Dag processor discovers it and starts > > > processing its files. Existing bundle instances are reused. > > > - When a bundle is removed, the processor stops processing it. An > > > unfiltered processor (without --bundle-name) marks it inactive, and > > the > > > existing stale DAG scan marks its dags stale. A successful read > > > containing > > > [] removes all bundles. > > > - Failed commands, unavailable secrets, wrong configs, invalid new > > > classpaths, and blank reads keep the last accepted list in that > > process. > > > The source must return the complete list. Commands should return a > non > > > zero > > > exit on failure, rather than returning []. > > > - A fresh process has no previous list to fall back to. Invalid or > > > unavailable configuration prevents startup. A valid [] starts with > no > > > bundles. > > > - Processors started with --bundle-name stop processing removed > > bundles > > > locally, but never deactivate missing bundles globally (retaining > > > existing > > > behaviour). A processor that sees the complete list is needed for > > global > > > deactivation. > > > - Multiple Dag processors may see changes at different times (due to > > > variance in the refresh interval). Each unfiltered processor > > reconciles > > > bundle in the database even when its own list has not changed, so it > > can > > > correct a stale write from another processor. They eventually > converge > > > when > > > surviving processors see the same list and continue reconciling. > > > > > > All components, i.e. the DAG processor and workers that resolve bundles > > > need access to the configuration source. Existing Dag runs should > finish > > > before removing a bundle if those runs need to complete, since tasks > and > > > callbacks can still need its definition. > > > > > > For v1, changing an existing bundle’s definition is rejected, and the > > last > > > accepted list is retained. This includes changing tracking_ref or > > classpath > > > etc. Such a change requires a new bundle name or restarting the > affected > > > components. New commits on the already configured branch continue to be > > > picked up through the existing bundle refresh behavior. Supporting > > changes > > > to an existing bundle definition can be added later incase required! > > > > > > There is also an optional DagFileParseRequest.bundle_config field > > > containing the accepted importer settings and team ownership. It allows > > the > > > parsing process (to parse the DAG) to use the configuration accepted by > > its > > > parent, without reading the command or secrets backend again and > > > potentially seeing a different list. This does change the parser > protocol > > > but the compatibility handling is included in the PR > > > > > > The dag_bundle_config_list option is marked sensitive: true to enable > > _CMD > > > and _SECRET. This also means the option is hidden in configuration > views > > > that hide sensitive values. The Deployment Manager continues to control > > the > > > command or secrets backend and its credentials. No API for creating or > > > changing bundle definitions is added. > > > > > > Related discussions by various users: > > > > > > - Runtime bundle management: > > > https://github.com/apache/airflow/issues/61645 > > > - Automatic bundle configuration loading: > > > https://github.com/apache/airflow/discussions/59799 > > > - Earlier file-based reload PR: > > > https://github.com/apache/airflow/pull/71111 > > > - Earlier dev-list discussion on runtime reload: > > > https://lists.apache.org/thread/co0z6yxc1g6ctqqmt7rdszjhjsq0w2hx > > > > > > I would appreciate feedback on this revised scope, proposed flow and > the > > > state v1 limitation. > > > > > > Thanks, > > > Sameer Raj > > > > > > On Wed, Sep 23, 2026 at 3:35 PM Sameer Raj <[email protected]> > wrote: > > > > > > > Hi everyone! > > > > I would like to discuss adding support for dynamically discovering > new > > > DAG > > > > bundle configurations without requiring Airflow restarts. (following > up > > > on > > > > comment: > > > > > https://github.com/apache/airflow/issues/61645#issuecomment-5017217627 > > ) > > > > > > > > There's also multiple PRs/discussion going over for the same feature: > > > > - Runtime CRUD proposal: > > > > https://github.com/apache/airflow/issues/61645 > > > > - Automatic bundle configuration loading: > > > > https://github.com/apache/airflow/discussions/59799 > > > > - File-based hot-reload PR: > > > > https://github.com/apache/airflow/pull/63928 > > > > - Add dynamic DAG bundle configuration from file path > > > > https://github.com/apache/airflow/pull/71111 > > > > > > > > *The Problem:* > > > > > > > > Today, Dag bundles are configured through [dag_processor] > > > > dag_bundle_config_list. DagBundlesManager loads this configuration > when > > > > the process starts, and the Dag processor uses the same bundle list. > > > > > > > > This becomes operationally difficult for deployments that manage many > > > > bundles and regularly add (or remove) them. Each configuration change > > > > currently requires restarting or redeploying the affected Airflow > > > processes. > > > > > > > > *Proposal* > > > > > > > > The proposal has two parts: > > > > > > > > First, introduce a DagBundleProvider abstraction into > > DagBundlesManager. > > > > It would allow Airflow to discover the bundle configurations > construct > > > > bundles accordingly. Thus, the responsibility to fetch bundles is > > passed > > > on > > > > to DagBundleProvider by DagBundleManager. > > > > > > > > The configured class could obtain bundle information from a file, > blob > > > > storage, an internal service, or another deployment-specific source > > (and > > > > this will decouple Airflow being aware of the source). Airflow would > > > define > > > > the class contract without adding direct support for every possible > > > source. > > > > > > > > The existing dag_bundle_config_list would remain the default > > > > implementation, so current deployments would keep the same behavior. > > > > > > > > Second, allow the Dag processor to periodically ask the configured > > class > > > > for the latest bundle configurations from its existing processing > loop. > > > > > > > > When a bundle is added, the Dag processor would discover it and begin > > > > processing its files. When a bundle is removed, the Dag processor > would > > > > stop processing it and use the existing removed-file handling for its > > > Dags. > > > > > > > > Each Dag processor performs this reconciliation periodically. > > Processors > > > > may observe a change at slightly different times, but they will > > converge > > > on > > > > the configuration returned by the provider. > > > > > > > > *Why use a configurable class?* > > > > > > > > In this comment > > > > < > > https://github.com/apache/airflow/issues/61645#issuecomment-5017217627 > > > >, > > > > Jarek asked why a custom dynamic manager would be preferable to > > managing > > > > Airflow configuration dynamically. > > > > > > > > On this, having this abstraction can allow users to implement their > way > > > by > > > > which they plan to reload the configs, for example, fetch from > external > > > > system etc. Also changing the environment variable for bundle list > wont > > > > automatically reload the bundles, and the capability need to be built > > for > > > > the same. > > > > > > > > *Security* > > > > > > > > The security implications of changing bundle definitions through the > > > > Airflow API were raised in issue #61645 > > > > < > > https://github.com/apache/airflow/issues/61645#issuecomment-4212433885 > > > >. > > > > > > > > This proposal keeps management of bundle definitions within the > > existing > > > > Deployment Manager trust boundary. > > > > > > > > - The Deployment Manager selects and deploys the DagBundleProvider > > > > implementation. > > > > - The Deployment Manager controls access to its configuration > > source. > > > > - Credentials and authorization for that source remain deployment > > > > managed. > > > > - The Airflow API does not expose operations for creating or > > changing > > > > bundle definitions. > > > > - No new Airflow user role is introduced for managing bundles. > > > > > > > > *Compatibility* > > > > > > > > The existing dag_bundle_config_list remains the default source and > > keeps > > > > its current behavior. > > > > > > > > *Value Addition* > > > > > > > > Multiple users have raised PR/discuss thread on Airflow Github for > the > > > > same, and this feature will be very useful for the community who are > > > > managing multiple bundles. Reference for the threads is shared above. > > > > > > > > *Current work* > > > > > > > > I have split the work into two PRs so each part can be reviewed > > > separately: > > > > > > > > - Provider abstraction: > > > > https://github.com/apache/airflow/pull/73209 > > > > - Runtime configuration reload: > > > > https://github.com/apache/airflow/pull/73217 > > > > > > > > I would appreciate feedback on this! > > > > > > > > Thanks, > > > > Sameer Raj > > > > > > > > > > > > > > > > > > > > > > > > > >
