Yep. It's a valid point that we should re-evaluate things now after Airflow 3 is out - the reason why we delayed it was that we wanted to get more clarity on what implementation and scope of the Airflow 3 changes will be and see how it fits.
I wonder what others - especially those who run Airflow at scale and hear the users asking for different forms of multi-team - would say to the expectations they have and how they map to Airflow 3 - maybe indeed we might come up with a simpler way of achieving those expectations. Definitely worth discussing it. J. On Thu, Jun 12, 2025 at 2:15 PM Ash Berlin-Taylor <a...@apache.org> wrote: > Hi everyone, > > One thing I’ve been struggling with while reading the other thread about > multi-team DB changes[0] is what is the end-user problem we are trying to > address with it. > > The main impetus for opening this discussion is that a lot has changed in > Airflow since this AIP was created in early 2024 and voted on mid-2024, and > I'm wondering if those changes are big enough to invalidate the design and > assumptions made at the time. > > Reading the DB changes thread I see that that changes are far reaching and > necessarily have to touch most of the Airflow object models, and this got > me thinking about what value do we actually get with the change, since as > stated in the AIP some of the non-goals are[1] (slightly edited here for > brevity with the “[…]"): > > > > • Sharing broker/backend for celery executors between teams. This MAY be > covered by future AIPs > > • Implementation of FAB-based multi-team Auth Manager. […] > > • Per-team concurrency and prioritization of tasks. […]. > > • Resource allocation per-executor. In the current proposal, executors > are run as sub-processes of Scheduler and we have very little control over > their individual resource usage. […] > > • Turn-key multi-team Deployment of Airflow (for example via Helm > chart). This is unlikely to happen.[…] > > • team management tools (creation, removal, rename etc.). […] > > • Combining "global" execution with "team" execution. While it should be > possible in the proposed architecture to have a "team" execution and > "global" execution in a single instance of Airflow, this has it's own > unique set of challenges and assumption is that Airflow Deployment is > either "global" (today) or "multi-team" (After this AIP is implemented) - > but it cannot be combined (yet). This is possible to be implemented in the > future. > > • Running multiple schedulers - one-per team. While it should be > possible if we add support to select DAGs "per team" per scheduler, this is > not implemented in this AIP and left for the future > > And also Design Non-goals from the AIP [2]: > > > • It’s not a primary goal of this proposal to significantly decrease > resource consumption for Airflow installation compared to the current ways > of achieving “multi-tenant” setup. […] > > • It’s not a goal of the proposal to provide a one-stop installation > mechanism for “Multi-team” Airflow. […] > > • It’s not a goal to decrease the overall maintenance effort involved in > responding to needs of different teams, […] > > The main pain point that we seem to be addressing with this AIP is this[3]: > > > The main reason for having multi-team deployment of Airflow is achieving > security and isolation between the teams, coupled with ability of the > isolated teams to collaborate via shared Datasets. > > > So what’s changed since we collectively (myself included) voted on and > accepted this AIP? Well, we now have AIP-82 — External event driven dags. > That could be used to achieve this goal right now in 3.0 with no changes to > Airflow itself, and is perhaps a more robust mechanism of doing it too. > > So my main question, given the wide reaching code changes need for AIP-67, > and (IMO) the imperfect/limited scope of team completion I wonder if using > AIP-82 would not be a better solution to the problem. > > 1. It’s much simpler from a code level, as nothing need to change > 2. It’s not _that_ much more complex from an operational point of view > (you have to run an extra scheduler and web server, but those would likely > need scaling up.) > 3. We won’t disappoint people by not implementing the part of multi-team > that they want (Someone being part of multiple teams, sharing > connections/vars between teams) > > And using this mechanism (of external dataset/asset polling) also negates > one of the biggest cons of the AIP-67, that of the tight coupling of > Airflow versions between the teams. In larger companies this is a _huge_ > problem already, and this would only make it worse. > > So what’s my idea (and at this stage is it only an idea for discussion) is > that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now and > decide if it’s still worth the added complexity of DB, code and operational > overhead, and decided if we still want it. > > Please, please, please point out if there are other benefits that I have > missed, I'm not trying to be selective and get my way, I'm trying to make > sure Airflow continues to meet the need of users, and can also continue to > evolve (where I worry that complexity of code/datamodel materially hurts > that final point) > > Thoughts? > > [0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c > [1]: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope > ? > [2]: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals > [3]: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded > ? > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
