Agree with Ash. I also have a feeling that this is a very generic description and some POC describing the approach we would like to take here is needed to be able to vote on it. Feels a bit too early to vote on it.
A lot of internals of the current (Airflow 2) way Airflow handling of tasks running in executor are about database communication and if you look closely, decoupling those internals to make it works with the current executor API might be either very difficult or complex if we stick to the current task <> airflow communication patterns. In some ways, you already get "Remote Executor" by simply completing AIP-44 (which precisely provides an HTTP-based. interface between tasks and scheduler in a very similar fashion as AIP-69 proposal, but without changing the communication patterns. As I see it it generally looks like AIP-69 is either the same as AIP-44 or the same as the future "Airflow 3" task isolation we've been discussing about and Ash is working on. Both AIP-44 and the future Task Isolation aim to solve pretty much the same problem but AIP-44 in a very "backwards compatible" way and limited change on how to achieve "remote execution" without actually making a lot of ripple effect on all other components. So I would really love to understand if AIP-69 is really something in-between and how - but the "lightweitness" of description makes it difficult to understand how different AIP-69 is from those two (of course we should see the future task isolation AIP as well to understand it better and know what kind of back-compatibilities it will involve in Airflow 3). I think at the very minimum we should see the proposal of how the API will look like between the task and executor (including the whole life-cycle of tasks), the way how we are going to implement all the complexity involved with task adoption, edge cases of scheduling, execution semantic promises the API will hold (exactly-once, at-most-once, at least once) - something that comes as given for celery, the queuing mechanism and technologies used (how do we handle distributed case where you have to manage multiple workers running and how the tasks will be distributed etc. etc. For me the AIP currently is mostly documenting a "wishlist" but the implementation details on which of those wishes we implement, and which not, and how is very much absent. J. On Sat, May 18, 2024 at 10:41 AM Ash Berlin-Taylor <a...@apache.org> wrote: > Can we have a link to the pr please? The AIP doc itself is still light on > what changes are actually needed > > On 18 May 2024 14:56:57 BST, Aritra Basu <aritrabasu1...@gmail.com> wrote: > >+1 (non-binding) > >The proposal was a good read, would love to see it come up and would love > >to help out if you need a helping hand. > > > >-- > >Regards, > >Aritra Basu > > > >On Sat, May 18, 2024, 7:15 PM Christian Schilling > ><christian.lellm...@googlemail.com.invalid> wrote: > > > >> Hi Jens, > >> > >> Thank you very much for the proposal! > >> This would be cool to have such a feature in Airflow. > >> > >> +1 non-binding > >> > >> Best, > >> > >> Chris > >> > >> Scheffler Jens (XC-AS/EAE-ADA-T) <jens.scheff...@de.bosch.com.invalid> > >> schrieb am Sa., 18. Mai 2024, 15:40: > >> > >> > Hi all, > >> > > >> > > >> > > >> > Discussion thread: > >> > > >> > https://lists.apache.org/thread/8hlltm9brdxqyf8jyw1syrfb11hl52k5 > >> > > >> > > >> > > >> > I would like to officially call for a vote to add a Remote Executor > >> > feature to Airflow – via a provider package. All details are > documented > >> in: > >> > > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor > >> > > >> > > >> > > >> > Since the first draft was raised and the discussion thread went > along, I > >> > integrated a bit of feedback and now also added some technical > details as > >> > proposed development. After successful vote I’d raise a draft PR as > PoC. > >> > As a big wave of emails was posted by Jarek after I dropped this AIP > I’d > >> > like to highlight that I propose to make this a tactical > implementation > >> > which might be a base for some discussion how to distribute work in a > >> > future Airflow 3.0. I would assume if interfaces and structures > change, > >> > rework will be needed and it is fully accepted that breaking changes > need > >> > to be applied if moving to Airflow 3. > >> > > >> > > >> > > >> > Looking forward to releasing this for Airflow 2.10 (but depending how > >> fast > >> > I can make it and also depending on if somebody wants to join forces). > >> > > >> > > >> > > >> > Consider this my +1 binding vote. > >> > > >> > > >> > > >> > The vote will last until 6:00 PM GMT/UTC on May 23, 2024, and until at > >> > least 3 binding votes have been cast. I have it a bit longer as usual > >> > because of a public holiday in some countries. > >> > > >> > > >> > > >> > Please vote accordingly: > >> > > >> > > >> > > >> > [ ] + 1 approve > >> > > >> > [ ] + 0 no opinion > >> > > >> > [ ] - 1 disapprove with the reason > >> > > >> > > >> > > >> > Only votes from PMC members and committers are binding, but other > members > >> > of the community are encouraged to check the AIP and vote with > >> > "(non-binding)". > >> > > >> > Mit freundlichen Grüßen / Best regards > >> > > >> > Jens Scheffler > >> > > >> > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > >> > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > >> > GERMANY | www.bosch.com > >> > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > >> > jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> > >> > > >> > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > >> > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > >> > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. > Markus > >> > Forschner, > >> > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > >> > > >> > > >> >
