I like AIP-68. I have been thinking about the exact same thing. I think it can fit into either 2.x or 3.0. Before we take it out of draft I need to play with plugins more to get a good sense of how to actually implement it.
On Tue, Apr 23, 2024 at 12:38 AM Amogh Desai <amoghdesai....@gmail.com> wrote: > Good work on the AIPs @Scheffler Jens (XC-DX/ETV5) > <jens.scheff...@de.bosch.com>! > > I spent some time understanding your thoughts and I like the ideas and the > direction you are heading too. > > I also agree with Jarek, that we might need to start thinking in terms of > Airflow 3, especially now that you > brought up AIP 68. > > Thanks & Regards, > Amogh Desai > > > On Sat, Apr 20, 2024 at 3:55 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Hey Jens, > > > > I looked at the AIPs when you created them and I very much like the > > directions put there - but it also got me into a lot of thinking on the > > future of Airflow and AIPs. See the thread I started > > https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2 - > about > > Airflow 3. > > > > I think in both cases (especially about AIP-689 but also AIP-69) - it > would > > make a wealth of difference if we treat them in the context of Airflow 2, > > or (maybe) in the context of Airflow 3 which might start from taking the > > best of Airflow 2 and get rid of all the unnecessary baggage it has. In > the > > past many similar efforts like AIP-69 had stalled in general because they > > were far too complex to implement taking into account backwards > > compatibility expectations of Airflow 2. > > > > And I think it's the right time we should get to terms with the future of > > Airflow - whether/to what extent we want Airflow 3 to come, what level of > > compatibility it should have, which assumptions should be dropped. I > > personally have a feeling that AIP-69 would have been way easier to bring > > as one of Airflow 3 "foundational" AIP that could define the "new" remote > > architecture of Airflow rather than "plugin" to existing one. Dropping > > Celery & K8s Options, leaving the Remote + Local variant of it as the > only > > ones, without the direct DB communication channel we have now and > replacing > > it with smth else. > > > > That would be my first comment on it and a question - should we get a bit > > more clarity on what "future" of Airflow is before we discuss details and > > approach there. > > > > J. > > > > On Fri, Apr 19, 2024 at 10:06 PM Scheffler Jens (XC-AS/EAE-ADA-T) > > <jens.scheff...@de.bosch.com.invalid> wrote: > > > > > Dear Dev-Community, > > > > > > mainly triggered by the deadline for CFP for Summit I dropped two > “brand > > > new” AIP’s as ideas that are running in my head for a longer time. Note > > > these are DRAFT versions as a first write-up as solution concept and > are > > > lagging technical design and implementation yet. > > > > > > I’d kindly ask for feedback and review in Confluence cWiki (via > Comments) > > > – and also am seeking for people who like to join forces. > > > > > > AIP-68: Extended Plugin Interface for Custom Grid View Panels > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels > > > > > > Main motivation is that the UI has developed a lot in the recent time > and > > > AIP-38 is near completion. But it is focusing on technical details and > > logs > > > – and for most business users it is hard to read, missing business > > > perspective. I propose to extend the Plugin interface allowing > > > customizations on various new levels such that customer specific > business > > > information can be embedded, > > > > > > AIP-69: Remote Executor > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor > > > > > > Airflow can be deployed in cloud or on-prem and various options of > > > deployment are possible. But it lags the option to easily form a secure > > and > > > lean distributed setup for cases where individual nodes are far far > away > > > from the core of deployment. This imposes problems in opening firewalls > > and > > > might raise risk of security. Therefore I propose to add a “Remote > > > Executor” such that workload can easily distributed to remote > locations – > > > also with a chance that it is easier for cases where people want to > > > distribute workload to Windows (yeah there are really people around who > > > still have this). > > > > > > Looking forward for (constructive) feedback, discussion and opinions. > > > > > > Again, note: DRAFT means open to discussion, nothing fixed, nothing > coded > > > yet. Many implementation options possible – and in case of interest > > please > > > join forces with me 😃 > > > Once the discussion is calming I’d call for a vote separately like > > usually. > > > > > > Mit freundlichen Grüßen / Best regards > > > > > > Jens Scheffler > > > > > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > > > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > > > GERMANY | www.bosch.com > > > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > > jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> > > > > > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > > > Forschner, > > > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > > > > > > > > >
