Hi everyone! I would like to discuss adding support for dynamically discovering new DAG bundle configurations without requiring Airflow restarts. (following up on comment: https://github.com/apache/airflow/issues/61645#issuecomment-5017217627)
There's also multiple PRs/discussion going over for the same feature: - Runtime CRUD proposal: https://github.com/apache/airflow/issues/61645 - Automatic bundle configuration loading: https://github.com/apache/airflow/discussions/59799 - File-based hot-reload PR: https://github.com/apache/airflow/pull/63928 - Add dynamic DAG bundle configuration from file path https://github.com/apache/airflow/pull/71111 *The Problem:* Today, Dag bundles are configured through [dag_processor] dag_bundle_config_list. DagBundlesManager loads this configuration when the process starts, and the Dag processor uses the same bundle list. This becomes operationally difficult for deployments that manage many bundles and regularly add (or remove) them. Each configuration change currently requires restarting or redeploying the affected Airflow processes. *Proposal* The proposal has two parts: First, introduce a DagBundleProvider abstraction into DagBundlesManager. It would allow Airflow to discover the bundle configurations construct bundles accordingly. Thus, the responsibility to fetch bundles is passed on to DagBundleProvider by DagBundleManager. The configured class could obtain bundle information from a file, blob storage, an internal service, or another deployment-specific source (and this will decouple Airflow being aware of the source). Airflow would define the class contract without adding direct support for every possible source. The existing dag_bundle_config_list would remain the default implementation, so current deployments would keep the same behavior. Second, allow the Dag processor to periodically ask the configured class for the latest bundle configurations from its existing processing loop. When a bundle is added, the Dag processor would discover it and begin processing its files. When a bundle is removed, the Dag processor would stop processing it and use the existing removed-file handling for its Dags. Each Dag processor performs this reconciliation periodically. Processors may observe a change at slightly different times, but they will converge on the configuration returned by the provider. *Why use a configurable class?* In this comment <https://github.com/apache/airflow/issues/61645#issuecomment-5017217627>, Jarek asked why a custom dynamic manager would be preferable to managing Airflow configuration dynamically. On this, having this abstraction can allow users to implement their way by which they plan to reload the configs, for example, fetch from external system etc. Also changing the environment variable for bundle list wont automatically reload the bundles, and the capability need to be built for the same. *Security* The security implications of changing bundle definitions through the Airflow API were raised in issue #61645 <https://github.com/apache/airflow/issues/61645#issuecomment-4212433885>. This proposal keeps management of bundle definitions within the existing Deployment Manager trust boundary. - The Deployment Manager selects and deploys the DagBundleProvider implementation. - The Deployment Manager controls access to its configuration source. - Credentials and authorization for that source remain deployment managed. - The Airflow API does not expose operations for creating or changing bundle definitions. - No new Airflow user role is introduced for managing bundles. *Compatibility* The existing dag_bundle_config_list remains the default source and keeps its current behavior. *Value Addition* Multiple users have raised PR/discuss thread on Airflow Github for the same, and this feature will be very useful for the community who are managing multiple bundles. Reference for the threads is shared above. *Current work* I have split the work into two PRs so each part can be reviewed separately: - Provider abstraction: https://github.com/apache/airflow/pull/73209 - Runtime configuration reload: https://github.com/apache/airflow/pull/73217 I would appreciate feedback on this! Thanks, Sameer Raj
