Great idea Vikram, I love the idea of making this a provider/pluggable. In some ways, we already have a pluggable mechanism for Authentication with Auth Backends *[1]*. Where we will need lot more work I think is:
1. Replacing Access Control provided by FAB with a base/core security model (that is still resource-based) *[2]* 2. Extend this to the other Airflow components (scheduler, workers, triggered, cli) or make them all driven by a single API that takes care of Auth. This will also reduce a lot of duplication of code across many of the components 3. For backwards compact, we could ship with FAB-provider that still uses Flask-app builder in addition to our recommended provider that will have more features and users/companies/stabkeholders can build on top of that provider to extend it further. References: [1]: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#auth-backends [2]: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/access-control.html On Tue, 14 Feb 2023 at 02:06, Mehta, Shubham <[email protected]> wrote: > Hi Vikram, > Thank you for taking the time to review the proposal. I appreciate your > insights — I will make sure to reach out to you directly in the future for > feedback as that would've undoubtedly saved us some time and effort. > > In regards to the separation of user management, I understand your > concerns and, on a high-level, I agree with you. However, I think it would > be beneficial to have more details on how it will work. Here are a few > questions that come to mind: > 1. How will the user-id/group-id interface interact with Airflow > resource-level permissions? What parts of "John can-edit dag1 and can-view > dag2" be part of Airflow core? What will be exposed to the external > system? > 2. Who will be responsible for managing the resource-level permissions? > Will it be the external system? > 3. What are the limitations of this new pluggable model compared to FAB? > Will there be restrictions on the granularity of resource access that > Airflow admins can provide to their users? > 4. As Jarek pointed out, with this change we want to make authorization > externally driven. Will this have a significant impact on Airflow > performance as authorization will be required for fetching variables, > executing tasks, etc.? > 5. What will the migration process look like for existing users to this > non-FAB pluggable model? > > In addition, are there any other user scenarios, beyond multi-tenancy, > that Airflow users are looking to enable and that require this > pluggability? Asking as I haven't come across them. Overall, I believe we > need more information on your proposal before seeking feedback from the > community. Could we work together during February to develop a concrete > proposal? > > Beside this, I would like to propose that we define the scope and > long-term vision of "Airflow core". To achieve this, it may be helpful to > first outline the perspectives of the Airflow PMCs. Recently, there have > been discussions regarding the separation of executors into a separate > package, the implementation of pluggable schedulers, and other related > topics. Currently, these decisions and discussions are somewhat ad hoc and > are made through the mailing list. I would be happy to collaborate and > invest time in this effort. > > Regards > Shubham > > On 2023-02-13, 11:04 AM, "Jarek Potiuk" <[email protected]> wrote: > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender and > know the content is safe. > > > > Hey Vikram, > > I think it's brilliant and I wonder how it happened that had not > occurred to us earlier. And I believe that is due to the natural > tendency of "following as we always did" rather than thinking > completely out-of-the-box. Thanks Vikram for bringing it up. > > The funny thing is that when I see this: > > > However, I don't agree that this level of user management belongs in > "Core Airflow". > > I almost immediately think - NOOOOO, why, it's always been here, how > can we remove it? > > But then if you look a bit closer: > > > think this is a time to consider the concept of a "user management > provider" with a simple built-in implementation being the current Airflow > functionality, enabling alternate more complex (but separate) > implementations such as your proposal here as alternate user management > providers. > > Then it starts to make way more sense. Way more. > > And when you look further: > > > Maybe, this also enables us to get rid of the Fab security manager > from core Airflow? > > My heart jumps and I am immediately sold on the idea. > > When I was commenting on the doc initially, something was not right. > I had a feeling It is probably the 5th time I am looking and > commenting on a similar document. And, well, I did, actually. Most of > the things we discussed there are already implemented out there. We > just need to make sure we expose enough of the API to use them. For > example we have Keycloak that is an open source implementation of > Identity and Access Management. With everything out there already > integrated. and I've been part of the project that integrated just the > authentication part. Now if we rethink the authorization and make it > simpler and "externally driven", this will not only be faster IMHO, > but also will allow enterprise users to integrate much better. > > I believe following the path that Vikram outlined will be a good > direction for everyone in the community - including all the Manage > Service providers, who will have a far easier job on integrating > Airflow into their authentication models. > > J. > > > > On Mon, Feb 13, 2023 at 6:24 PM Vikram Koka > <[email protected]> wrote: > > > > Shubham and Vincent, > > > > Let me start by saying that I apologize for my delayed response to > your original email. > > > > I appreciate the detailed write-up and the thought behind it. I > completely agree with your use case and understand how this is applicable > to enterprises with multiple data teams using Airflow. > > > > However, I don't agree that this level of user management belongs in > "Core Airflow". > > > > I strongly believe that the core Airflow mission is for the > community at large and for data practitioners either individuals or teams > within enterprises. And therefore, I don't disagree with the intent of > making it easier for enterprise teams to adopt Airflow. But, I think there > is a never ending list of user management features which are needed to > support Enterprise needs. We have already struggled with this over time and > faced challenges with the Fab security manager and its integration in > Airflow. > > > > I think we should use this opportunity and your use case to > "separate the user management" from Core Airflow outside of the absolute > basics. I think this is a time to consider the concept of a "user > management provider" with a simple built-in implementation being the > current Airflow functionality, enabling alternate more complex (but > separate) implementations such as your proposal here as alternate user > management providers. Maybe, this also enables us to get rid of the Fab > security manager from core Airflow? > > > > Best regards, > > Vikram > > > > > > On Fri, Feb 3, 2023 at 8:22 AM Beck, Vincent > <[email protected]> wrote: > >> > >> Thanks __ > >> > >> On 2023-02-03, 10:55 AM, "Jarek Potiuk" <[email protected]> wrote: > >> > >> CAUTION: This email originated from outside of the > organization. Do not click links or open attachments unless you can confirm > the sender and know the content is safe. > >> > >> > >> > >> Added. > >> > >> On Fri, Feb 3, 2023 at 3:53 PM Beck, Vincent > >> <[email protected]> wrote: > >> > > >> > Thank you! > https://cwiki.apache.org/confluence/display/~vin100.beck > >> > > >> > On 2023-02-02, 5:38 PM, "Jarek Potiuk" <[email protected]> > wrote: > >> > > >> > CAUTION: This email originated from outside of the > organization. Do not click links or open attachments unless you can confirm > the sender and know the content is safe. > >> > > >> > > >> > > >> > What's your cwiki ID, Vincent (I'll add you without going > into details yet) > >> > > >> > >
