+1 non-binding. We have always believed that a single airflow makes it easier for our users. Unfortunately that leaves much to be desired on isolation perspective. For that reason our most requested functionality remains; 'can I get a Variable or Connection only accessible to my team?'. Our users, and especially security team, will be delighted with this improvement.
On Mon, 5 Aug 2024, 05:19 Guangyang Li, <lig...@gmail.com> wrote: > +1 non-binding > > Very happy to see this support! I've seen Airflow in several companies with > a long long requirements.txt, in order to support the needs from multiple > teams. I'm sure those maintainers are super excited to have this feature. > > On Fri, Aug 2, 2024 at 7:00 PM Vikram Koka <vik...@astronomer.io.invalid> > wrote: > > > +1 binding > > > > Thank you Jarek for being very patient and working through all the twists > > and turns on this AIP. > > > > I do believe that the changes are all positive and I really like the > > evolution towards a more focused multi-team target. > > This has become an uber-AIP, which is now building on top of AIPs such as > > the AIP-72 Task Execution Interface, and > > AIP-73 Data Awareness (only referencing other in-flight AIPs at this > time, > > rather than already completed ones). > > That's fantastic. > > > > I think this may require the "Auth manager" work to be done as well, > which > > you brought up in a different thread. > > I noticed that you kept it out of scope of this AIP, which I think is > > right, but may be needed for the overall use case. > > > > Based on the dependencies, I also agree that this would be an Airflow 3.1 > > target, but having a future target use case in mind > > is also a good thing for the developers of the underlying AIPs to not > > preclude those scenarios and to flag additional work > > needed for this use case during implementation. > > > > > > > > > > > > > > > > > > On Fri, Aug 2, 2024 at 1:13 PM Vincent Beck <vincb...@apache.org> wrote: > > > > > +1 binding. Very good demonstration of perseverance on this one :) > > > > > > On 2024/08/02 20:02:00 "Bishundeo, Rajeshwar" wrote: > > > > +1 non-binding. > > > > > > > > Great job to all the folks getting this feature 1-step closer to > > reality! > > > > > > > > -- Rajesh > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2024-08-02, 3:57 PM, "Mehta, Shubham" <shu...@amazon.com.inva > > > <mailto:shu...@amazon.com.inva>LID> wrote: > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do > not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > > certain que le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 non-binding. > > > > > > > > > > > > Thanks, Jarek, for continuing to evolve this piece of work based on > the > > > hundreds of comments you received from everyone. While there are > > > dependencies on other AIPs, I'd love to see this implemented in Airflow > > > 3.0. I hope other AIP authors also consider this as a reality and plan > > for > > > a multi-team setup during implementation. Ideally, users migrating to > > > Airflow 3 should be able to plan for an optimal team structure and > access > > > permissions from the outset. The AWS team will definitely help in any > way > > > possible to ship this on time. Looking forward to the execution! > > > > > > > > > > > > Shubham > > > > > > > > > > > > On 2024-08-02, 12:46 PM, "Kaxil Naik" <kaxiln...@gmail.com <mailto: > > > kaxiln...@gmail.com> <mailto:kaxiln...@gmail.com <mailto: > > > kaxiln...@gmail.com>>> wrote: > > > > > > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do > not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > > certain que le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 binding > > > > > > > > > > > > > > > > > > > > Thank you for persisting with the questions & changes, great job. > > > > > > > > > > > > > > > > > > > > Realistically, though, I think this would happen in 3.1 due to > > > > dependencies, as you mentioned in the email. > > > > > > > > > > > > > > > > > > > > Looking forward. > > > > > > > > > > > > > > > > > > > > Regards, > > > > Kaxil > > > > > > > > > > > > > > > > > > > > On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T) > > > > <jens.scheff...@de.bosch.com.inva <mailto: > > > jens.scheff...@de.bosch.com.inva> <mailto: > > jens.scheff...@de.bosch.com.inva > > > <mailto:jens.scheff...@de.bosch.com.inva>>lid> wrote: > > > > > > > > > > > > > > > > > > > > > +1 binding. > > > > > > > > > > Many thougths and discussions went into this AIP. Also User survery > > > showed > > > > > a strong demand for this - same we have also in our house. A multi > > team > > > > > setup is eagerly needed. > > > > > Nevertheless the complexity and missing turn-key delivery will be a > > > > > challenge. I assume it needs to evolve. > > > > > Besides a bit sceptic words the possibilities rule-out the > additional > > > > > complexity. I think we should commit making it - especially not > with > > > > > options to have some more breaking changes in AF3 is the right > point > > to > > > > > plan all meede structural changes in! Thanks Jarek for driving > this! > > > > > > > > > > Sent from Outlook for iOS<https://aka.ms/o0ukef> < > > > https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> < > > > https://aka.ms/o0ukef&gt;>> > > > > > ________________________________ > > > > > From: Jarek Potiuk <ja...@potiuk.com <mailto:ja...@potiuk.com> > > > <mailto:ja...@potiuk.com <mailto:ja...@potiuk.com>>> > > > > > Sent: Thursday, August 1, 2024 10:40:16 PM > > > > > To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > <mailto: > > > dev@airflow.apache.org <mailto:dev@airflow.apache.org>> < > > > dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > > > dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> > > > > > Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow > > Components > > > > > > > > > > Voting ends a week from now - i.e Thursday 8th of August 2023 > > > (midnight). > > > > > > > > > > On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <ja...@potiuk.com > > > <mailto:ja...@potiuk.com> <mailto:ja...@potiuk.com <mailto: > > > ja...@potiuk.com>>> wrote: > > > > > > > > > > > Hello here, > > > > > > > > > > > > I reviewed and - I hope - addressed all the comments so far. One > > most > > > > > > recent change in the doc was to allow a connection/variable to > > > belong to > > > > > > multiple teams. Also I mentioned a potential future change that > > will > > > > > allow > > > > > > us to run "per-team" schedulers - leaving the UI/DB schema as the > > > main > > > > > > components that are shared between teams. > > > > > > > > > > > > I am also fine if this one's delivery will slip to 3.1 - because > it > > > has > > > > > > many dependencies on other AIPs - and also makes some assumptions > > on > > > > > > the other AIP-s scope when they are not yet fully fleshed out. So > > > some > > > > > > implementation details **might** change. the general approach is > > > going to > > > > > > stay: > > > > > > > > > > > > * not targeting manageable "mutliple tenants" - but more of a > > > small-ish > > > > > > set of teams within the same organisation that want to maintain > > > separate > > > > > > environments and isolated execution environment > > > > > > > > > > > > * common UI and scheduler (potentially per-team schedule) as the > > main > > > > > > "common" piece > > > > > > > > > > > > * each team being able to manage their team separately - to the > > > extent of > > > > > > parsing DAGs and executing tasks > > > > > > > > > > > > The AIP proposal with updates: > > > > > > > > > > > > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0 > > > < > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0 > > > > > > < > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0 > > > > > > < > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&amp;reserved=0> > > > ;> > > > > > < > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components > > > < > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components > > > > > > < > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components > > > > > > < > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components> > > > ;> > > > > > > > > > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Т���������������������������������������������������������������������ХF�?V�7V'67&�&R�?R��?�â?FWb�V�7V'67&�&T??�&f��r�???6�R��&pФf�"??FF�F���?�?6���?�G2�?R��?�â?FWbֆV�???�&f��r�???6�R��&p > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
