Hi all, I'd like to propose adding `sets` to the triggerer and dag processor in the Helm chart, following the existing `workers.celery.sets` pattern, and get direction before opening PRs.
Why --- With `core.multi_team` enabled, each team needs a triggerer started with `--team-name`, otherwise deferred tasks of that team's Dags never resume (multi-team.rst, "Triggerer" section). The chart can only render one triggerer today, so a multi-team deployment on the official chart is not fully functional without patching templates. Running one dag processor per team or bundle (`--bundle-name`) is the matching need on the parsing side, for isolation and sizing. We run this in production (Airflow 3.3.2, several teams in one instance) through a wrapper chart that copies the upstream triggerer and dag processor templates in a loop. I'd rather contribute to it than keep that copy in sync. Proposal -------- triggerer: enableDefault: true # render the unsuffixed triggerer teamName: ~ # -> --team-name queues: ~ # -> --queues extraArgs: [] # appended to the default args sets: [] # each item overrides any triggerer.* key dagProcessor: enableDefault: true bundleNames: [] # -> repeated --bundle-name extraArgs: [] sets: [] - A set `<n>` renders `<fullname>-<component>-<n>` with its own ServiceAccount, PDB and ScaledObject, and a `<component>-set: <n>` label. Values merge exactly like worker sets (shared helper). - Sets reference bundles and teams by name only. Render-time validation checks them against dagBundleConfigList, so there is no second list to keep in sync (the concern raised on #61039). Users set small scope fields instead of a full CLI in `args` (the concern raised on #61963). - With every new field empty, the output is identical to today's. Planned as small PRs: shared set helpers (generalising workersMergeValues), triggerer.sets, dagProcessor.sets, then optional multi-team conveniences (deriving bundles from team_name, team-aware KEDA query). Related: #26494, #61037, #61039, #61963, #64037. Questions --------- 1. For the dag processor, is a separate `sets` list (referencing bundles by name) acceptable, or would you prefer per-bundle overrides on the dagBundleConfigList entries (as discussed on #61039)? The triggerer is scoped by team, not bundle, so `sets` seems the natural fit there either way. 2. Should this target 2.0.0? 3. The default instance's selectors also match set pods (same as worker sets today). Worth adding a `<component>-set: default` label to all three components while the 2.0 window allows breaking changes? Thanks, Mathieu Monet -- Drafted-by: Claude Code (AI assistant) and reviewed by me before sending. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
