Hi Sahar, Yeah, let's proceed with the _cmd/_secret approach for now, as it would be able to solve the use-case mentioned by Sameer and also opens up a way for the community to see whether the feature is really useful or not for them. We can start an AIP after that, and I believe there would be more participation from the community on this.
Thanks, Sumit On Sat, Oct 31, 2026 at 1:31 PM Shahar Epstein via dev < [email protected]> wrote: > Hi Sumit, > > Thanks for the feedback. I agree an interface would be better for the long > term. For example, a deployment could read bundle definitions directly from > S3 or an internal service, without wrapping the source in a command. > > A new public extension point would be loaded by every component, so it > needs an AIP (or an amendment to AIP-66) as discussed in #73209. AIP-85 for > Dag importers is a precedent. > > I suggest delivering the _cmd/_secret reload now to main (maybe we could > even backport to 3.4.1?). If you or Sameer want to pursue the interface, > let's start a separate AIP. > > > Shahar > > On Fri, Oct 30, 2026 at 7:40 PM Sumit Maheshwari <[email protected]> > wrote: > > > Personally, I prefer the interface approach as it aligns better with > > enterprise-grade software. That said, I am also fine with proceeding via > > the _cmd and _secret route for the time being. > > > > Thanks, > > Sumit > > > > On Fri, Oct 30, 2026 at 11:47 AM Shahar Epstein <[email protected]> > wrote: > > > > > Hey Sameer and everyone, > > > > > > +1 to the revised scope in #74915. > > > > > > The need has come up several times: #61645, discussion #59799, #63928, > > and > > > #71111 all asked for this in different forms. I think it’s a good time > to > > > add the feature. > > > > > > I think it can proceed without an AIP for the following reasons: > > > > > > 1. The revised scope adds no pluggable abstraction, config entry, API, > or > > > DB migration. Reloading uses the existing dag_bundle_config_list > format, > > > supplied through Airflow’s existing _cmd / _secret mechanism. The PR > > makes > > > that mechanism available for this key by marking it as sensitive. > > > 2. It’s opt-in. A value set through a plain environment variable or > > > airflow.cfg keeps today’s startup-only behavior. _CMD / _SECRET were > not > > > available for this key before, so no existing deployment will start > > > reloading without an explicit change. > > > 3. The trust boundary does not change. Bundle definitions remain under > > the > > > Deployment Manager’s control; users cannot change them through Airflow. > > > > > > Neither of the two previous threads has received a reply yet. Sameer, I > > > suggest waiting few more days (let's say until Monday, Nov. 2nd) to > give > > a > > > chance for others to comment on this thread, and following up with a > > [LAZY > > > CONSENSUS] thread on the revised scope if there are no responses by > then, > > > or there's consensus. > > > > > > Disclosure: I helped Sameer narrow the scope and I’m reviewing the PR, > so > > > please take that into account when weighing my comment. > > > > > > > > > Shahar > > > > > > On Sun, Oct 25, 2026 at 12:33 PM Sameer Raj via dev < > > > [email protected]> > > > wrote: > > > > > > > Hi everyone! > > > > > > > > Following up on this discussion, I would like to share the revised > > scope. > > > > The earlier proposal included a DagBundleProvider abstraction. I have > > > > dropped that abstraction from the current work. It can be proposed > > > > separately through a future AIP if needed. The new PR is limited to > > > > reloading the existing [dag_processor] dag_bundle_config_list through > > > > Airflow’s_CMD or_SECRET mechanism: > > > > > > > > https://github.com/apache/airflow/pull/74915 > > > > > > > > Thanks to Shahar for helping me narrow down the scope and for > reviewing > > > the > > > > approach. > > > > > > > > In our use case, we need to manage more than 500 DAG bundles, with > > > > additions being automated. Restarting or redeploying Airflow > processes > > > > whenever a bundle is added makes this difficult. We want to fetch the > > > > bundle list from an internal API. A deployment managed command can > call > > > > that API and return the same JSON list already accepted by > > > > dag_bundle_config_list. The command could also read a file, or the > list > > > > could come from a secrets backend. > > > > > > > > The DAG processor checks for changes at > > > > bundle_refresh_check_interval. Users do not need to change their > > > > configuration unless they want to enable reload. Users only need to > > > > configure _CMD or _SECRET if they want reload. > > > > > > > > The expected behavior is: > > > > > > > > - When a bundle is added, the Dag processor discovers it and > starts > > > > processing its files. Existing bundle instances are reused. > > > > - When a bundle is removed, the processor stops processing it. An > > > > unfiltered processor (without --bundle-name) marks it inactive, > and > > > the > > > > existing stale DAG scan marks its dags stale. A successful read > > > > containing > > > > [] removes all bundles. > > > > - Failed commands, unavailable secrets, wrong configs, invalid new > > > > classpaths, and blank reads keep the last accepted list in that > > > process. > > > > The source must return the complete list. Commands should return a > > non > > > > zero > > > > exit on failure, rather than returning []. > > > > - A fresh process has no previous list to fall back to. Invalid or > > > > unavailable configuration prevents startup. A valid [] starts with > > no > > > > bundles. > > > > - Processors started with --bundle-name stop processing removed > > > bundles > > > > locally, but never deactivate missing bundles globally (retaining > > > > existing > > > > behaviour). A processor that sees the complete list is needed for > > > global > > > > deactivation. > > > > - Multiple Dag processors may see changes at different times (due > to > > > > variance in the refresh interval). Each unfiltered processor > > > reconciles > > > > bundle in the database even when its own list has not changed, so > it > > > can > > > > correct a stale write from another processor. They eventually > > converge > > > > when > > > > surviving processors see the same list and continue reconciling. > > > > > > > > All components, i.e. the DAG processor and workers that resolve > bundles > > > > need access to the configuration source. Existing Dag runs should > > finish > > > > before removing a bundle if those runs need to complete, since tasks > > and > > > > callbacks can still need its definition. > > > > > > > > For v1, changing an existing bundle’s definition is rejected, and the > > > last > > > > accepted list is retained. This includes changing tracking_ref or > > > classpath > > > > etc. Such a change requires a new bundle name or restarting the > > affected > > > > components. New commits on the already configured branch continue to > be > > > > picked up through the existing bundle refresh behavior. Supporting > > > changes > > > > to an existing bundle definition can be added later incase required! > > > > > > > > There is also an optional DagFileParseRequest.bundle_config field > > > > containing the accepted importer settings and team ownership. It > allows > > > the > > > > parsing process (to parse the DAG) to use the configuration accepted > by > > > its > > > > parent, without reading the command or secrets backend again and > > > > potentially seeing a different list. This does change the parser > > protocol > > > > but the compatibility handling is included in the PR > > > > > > > > The dag_bundle_config_list option is marked sensitive: true to enable > > > _CMD > > > > and _SECRET. This also means the option is hidden in configuration > > views > > > > that hide sensitive values. The Deployment Manager continues to > control > > > the > > > > command or secrets backend and its credentials. No API for creating > or > > > > changing bundle definitions is added. > > > > > > > > Related discussions by various users: > > > > > > > > - Runtime bundle management: > > > > https://github.com/apache/airflow/issues/61645 > > > > - Automatic bundle configuration loading: > > > > https://github.com/apache/airflow/discussions/59799 > > > > - Earlier file-based reload PR: > > > > https://github.com/apache/airflow/pull/71111 > > > > - Earlier dev-list discussion on runtime reload: > > > > https://lists.apache.org/thread/co0z6yxc1g6ctqqmt7rdszjhjsq0w2hx > > > > > > > > I would appreciate feedback on this revised scope, proposed flow and > > the > > > > state v1 limitation. > > > > > > > > Thanks, > > > > Sameer Raj > > > > > > > > On Wed, Sep 23, 2026 at 3:35 PM Sameer Raj <[email protected]> > > wrote: > > > > > > > > > Hi everyone! > > > > > I would like to discuss adding support for dynamically discovering > > new > > > > DAG > > > > > bundle configurations without requiring Airflow restarts. > (following > > up > > > > on > > > > > comment: > > > > > > > https://github.com/apache/airflow/issues/61645#issuecomment-5017217627 > > > ) > > > > > > > > > > There's also multiple PRs/discussion going over for the same > feature: > > > > > - Runtime CRUD proposal: > > > > > https://github.com/apache/airflow/issues/61645 > > > > > - Automatic bundle configuration loading: > > > > > https://github.com/apache/airflow/discussions/59799 > > > > > - File-based hot-reload PR: > > > > > https://github.com/apache/airflow/pull/63928 > > > > > - Add dynamic DAG bundle configuration from file path > > > > > https://github.com/apache/airflow/pull/71111 > > > > > > > > > > *The Problem:* > > > > > > > > > > Today, Dag bundles are configured through [dag_processor] > > > > > dag_bundle_config_list. DagBundlesManager loads this configuration > > when > > > > > the process starts, and the Dag processor uses the same bundle > list. > > > > > > > > > > This becomes operationally difficult for deployments that manage > many > > > > > bundles and regularly add (or remove) them. Each configuration > change > > > > > currently requires restarting or redeploying the affected Airflow > > > > processes. > > > > > > > > > > *Proposal* > > > > > > > > > > The proposal has two parts: > > > > > > > > > > First, introduce a DagBundleProvider abstraction into > > > DagBundlesManager. > > > > > It would allow Airflow to discover the bundle configurations > > construct > > > > > bundles accordingly. Thus, the responsibility to fetch bundles is > > > passed > > > > on > > > > > to DagBundleProvider by DagBundleManager. > > > > > > > > > > The configured class could obtain bundle information from a file, > > blob > > > > > storage, an internal service, or another deployment-specific source > > > (and > > > > > this will decouple Airflow being aware of the source). Airflow > would > > > > define > > > > > the class contract without adding direct support for every possible > > > > source. > > > > > > > > > > The existing dag_bundle_config_list would remain the default > > > > > implementation, so current deployments would keep the same > behavior. > > > > > > > > > > Second, allow the Dag processor to periodically ask the configured > > > class > > > > > for the latest bundle configurations from its existing processing > > loop. > > > > > > > > > > When a bundle is added, the Dag processor would discover it and > begin > > > > > processing its files. When a bundle is removed, the Dag processor > > would > > > > > stop processing it and use the existing removed-file handling for > its > > > > Dags. > > > > > > > > > > Each Dag processor performs this reconciliation periodically. > > > Processors > > > > > may observe a change at slightly different times, but they will > > > converge > > > > on > > > > > the configuration returned by the provider. > > > > > > > > > > *Why use a configurable class?* > > > > > > > > > > In this comment > > > > > < > > > https://github.com/apache/airflow/issues/61645#issuecomment-5017217627 > > > > >, > > > > > Jarek asked why a custom dynamic manager would be preferable to > > > managing > > > > > Airflow configuration dynamically. > > > > > > > > > > On this, having this abstraction can allow users to implement their > > way > > > > by > > > > > which they plan to reload the configs, for example, fetch from > > external > > > > > system etc. Also changing the environment variable for bundle list > > wont > > > > > automatically reload the bundles, and the capability need to be > built > > > for > > > > > the same. > > > > > > > > > > *Security* > > > > > > > > > > The security implications of changing bundle definitions through > the > > > > > Airflow API were raised in issue #61645 > > > > > < > > > https://github.com/apache/airflow/issues/61645#issuecomment-4212433885 > > > > >. > > > > > > > > > > This proposal keeps management of bundle definitions within the > > > existing > > > > > Deployment Manager trust boundary. > > > > > > > > > > - The Deployment Manager selects and deploys the > DagBundleProvider > > > > > implementation. > > > > > - The Deployment Manager controls access to its configuration > > > source. > > > > > - Credentials and authorization for that source remain > deployment > > > > > managed. > > > > > - The Airflow API does not expose operations for creating or > > > changing > > > > > bundle definitions. > > > > > - No new Airflow user role is introduced for managing bundles. > > > > > > > > > > *Compatibility* > > > > > > > > > > The existing dag_bundle_config_list remains the default source and > > > keeps > > > > > its current behavior. > > > > > > > > > > *Value Addition* > > > > > > > > > > Multiple users have raised PR/discuss thread on Airflow Github for > > the > > > > > same, and this feature will be very useful for the community who > are > > > > > managing multiple bundles. Reference for the threads is shared > above. > > > > > > > > > > *Current work* > > > > > > > > > > I have split the work into two PRs so each part can be reviewed > > > > separately: > > > > > > > > > > - Provider abstraction: > > > > > https://github.com/apache/airflow/pull/73209 > > > > > - Runtime configuration reload: > > > > > https://github.com/apache/airflow/pull/73217 > > > > > > > > > > I would appreciate feedback on this! > > > > > > > > > > Thanks, > > > > > Sameer Raj > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
