Ash, you've raised some good points on the need to re-evaluate AIP-67, although I'm a bit confused on how AIP-82 factors into a multi-team solution. It's fair to have the discussion on how Airflow has changed and perhaps either redefining what AIP-67 means...or a set of new AIP's solving a subset of a larger need. We have seen talks at previous ( and even at the upcoming summit) where users have demonstrated their implementation of multi-team. I can't help but feel that they are being creative with some of those solutions (not that there's anything wrong with that), but because one doesn't exist in Airflow. Other than the feature being a consistent request on our Airflow surveys, we have a number of users that have asked and continue to ask when a multi-team solution would be available in Airflow. I think the next dev call (06/26) is great time to dive into this further.
-- Rajesh On 2025-06-12, 9:34 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Yep. It's a valid point that we should re-evaluate things now after Airflow 3 is out - the reason why we delayed it was that we wanted to get more clarity on what implementation and scope of the Airflow 3 changes will be and see how it fits. I wonder what others - especially those who run Airflow at scale and hear the users asking for different forms of multi-team - would say to the expectations they have and how they map to Airflow 3 - maybe indeed we might come up with a simpler way of achieving those expectations. Definitely worth discussing it. J. On Thu, Jun 12, 2025 at 2:15 PM Ash Berlin-Taylor <a...@apache.org <mailto:a...@apache.org>> wrote: > Hi everyone, > > One thing I’ve been struggling with while reading the other thread about > multi-team DB changes[0] is what is the end-user problem we are trying to > address with it. > > The main impetus for opening this discussion is that a lot has changed in > Airflow since this AIP was created in early 2024 and voted on mid-2024, and > I'm wondering if those changes are big enough to invalidate the design and > assumptions made at the time. > > Reading the DB changes thread I see that that changes are far reaching and > necessarily have to touch most of the Airflow object models, and this got > me thinking about what value do we actually get with the change, since as > stated in the AIP some of the non-goals are[1] (slightly edited here for > brevity with the “[…]"): > > > > • Sharing broker/backend for celery executors between teams. This MAY be > covered by future AIPs > > • Implementation of FAB-based multi-team Auth Manager. […] > > • Per-team concurrency and prioritization of tasks. […]. > > • Resource allocation per-executor. In the current proposal, executors > are run as sub-processes of Scheduler and we have very little control over > their individual resource usage. […] > > • Turn-key multi-team Deployment of Airflow (for example via Helm > chart). This is unlikely to happen.[…] > > • team management tools (creation, removal, rename etc.). […] > > • Combining "global" execution with "team" execution. While it should be > possible in the proposed architecture to have a "team" execution and > "global" execution in a single instance of Airflow, this has it's own > unique set of challenges and assumption is that Airflow Deployment is > either "global" (today) or "multi-team" (After this AIP is implemented) - > but it cannot be combined (yet). This is possible to be implemented in the > future. > > • Running multiple schedulers - one-per team. While it should be > possible if we add support to select DAGs "per team" per scheduler, this is > not implemented in this AIP and left for the future > > And also Design Non-goals from the AIP [2]: > > > • It’s not a primary goal of this proposal to significantly decrease > resource consumption for Airflow installation compared to the current ways > of achieving “multi-tenant” setup. […] > > • It’s not a goal of the proposal to provide a one-stop installation > mechanism for “Multi-team” Airflow. […] > > • It’s not a goal to decrease the overall maintenance effort involved in > responding to needs of different teams, […] > > The main pain point that we seem to be addressing with this AIP is this[3]: > > > The main reason for having multi-team deployment of Airflow is achieving > security and isolation between the teams, coupled with ability of the > isolated teams to collaborate via shared Datasets. > > > So what’s changed since we collectively (myself included) voted on and > accepted this AIP? Well, we now have AIP-82 — External event driven dags. > That could be used to achieve this goal right now in 3.0 with no changes to > Airflow itself, and is perhaps a more robust mechanism of doing it too. > > So my main question, given the wide reaching code changes need for AIP-67, > and (IMO) the imperfect/limited scope of team completion I wonder if using > AIP-82 would not be a better solution to the problem. > > 1. It’s much simpler from a code level, as nothing need to change > 2. It’s not _that_ much more complex from an operational point of view > (you have to run an extra scheduler and web server, but those would likely > need scaling up.) > 3. We won’t disappoint people by not implementing the part of multi-team > that they want (Someone being part of multiple teams, sharing > connections/vars between teams) > > And using this mechanism (of external dataset/asset polling) also negates > one of the biggest cons of the AIP-67, that of the tight coupling of > Airflow versions between the teams. In larger companies this is a _huge_ > problem already, and this would only make it worse. > > So what’s my idea (and at this stage is it only an idea for discussion) is > that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now and > decide if it’s still worth the added complexity of DB, code and operational > overhead, and decided if we still want it. > > Please, please, please point out if there are other benefits that I have > missed, I'm not trying to be selective and get my way, I'm trying to make > sure Airflow continues to meet the need of users, and can also continue to > evolve (where I worry that complexity of code/datamodel materially hurts > that final point) > > Thoughts? > > [0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c > <https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c> > [1]: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope > > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope> > ? > [2]: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals > > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals> > [3]: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded > > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded> > ? > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > <mailto:dev-unsubscr...@airflow.apache.org> > For additional commands, e-mail: dev-h...@airflow.apache.org > <mailto:dev-h...@airflow.apache.org> > >
