Hi Vikram, Jarek, Ash, Thanks for your fast feedback. So I understand you ask for more details. Then I will cancel the vote if you think the AIP is not ready.
I would have expected with my earlier call for discussion to receive more feedback on what you expect missing. Please add comments to confluence or post a message to me. That would help making the AIP to the level of detail you expect. A PR is not ready helping to discover, it is rather a skeleton but a lot of pieces are not decided. I would have planned to tackle this during implementation. And then - once code is mature - there are PRs planned of course where also serious code feedback will be needed. But my intent was not to spent 100's of hours in a detailed concept then to get a push-back later. That is why I called for vote. If you can also please express the "do this" or "don't this" in the AIP feedback that will help that I am not wasting my time hitting the wall later. Also I was seeking for contribution, so if you have ideas or recommendation, please keep them coming. Immediate feedback to some of your questions: - @ash: What do you expect in a level of detail in a PR to review before accepting the AIP? Are you looking for certain patterns, modules which are used? - @jarek: PoC can be made but I fear that is already 50% of effort to a MVP. That is why I was seeking for feedback if this would be the right way before spending efforts. - @jarek: AIP-44 would certainly be leveraged. I have no plan to replicate existing API and I know that it will be a challenge to de-couple the task execution. Do you expect me to evaluate all details before a vote? I would have planned a pragmatic approach, once some dependency missing I would maybe have contributed to AIP-44 efforts to close gaps. But AIP-44 does not make a remote scheduling/execution possible as it does not include the remote worker and executor component. That is to be added in AIP-69. - @jarek/@ash: You briefly talked about the "task isolation" but are there any concepts you can share which will help? I understood you are _thinking_ about it and there will be papers _soon_? If there is anything that contributes, please share. I have no glassball. @jarek: Lifecyle of task and adoption etc.: Yes this is something to be addressed and is a must to be included. Mainly it would be around a heartbeat logic. I am sure there will be something to be done and resiliency will be covered. If you have requirements or ideas please contribute. Do you see this as a must have in the AIP before going to vote? Or is it just important it is covered? @jarek: API promises on exactly once: My plan is to use and rely on DB locks and transactions from the API. Assuming that something can go wrong also between API and remote side I would add an additional confirmation in the heartbeat when the task was accepted and is starting on remote (to cover the use case that a task is assigned and DB transaction committed and response is not reaching the remote because of... line breaks down). Als certainly if a remote worker is "lost" and does not heartbeat for a (to be configured) timeout then a task must be re-assigned or assumed to be failed. P.S.: I'd like to take your feedback serious, the AIP process description in Confluence just tells: "Once you or someone else feels like there’s a rough consensus on the idea and there’s no strong opposition, you can move your proposal to the Vote phase." - neither this nor the structure template mention that a technical spec or PR must be provided prior vote. If you feel that an AIP should include this then I assume the contribution docs need to be adjusted. Mit freundlichen Grüßen / Best regards Jens Scheffler Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | www.bosch.com Tel. +49 711 811-91508 | Mobil +49 160 90417410 | jens.scheff...@de.bosch.com Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus Forschner, Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert -----Original Message----- From: Vikram Koka <vik...@astronomer.io.INVALID> Sent: Saturday, May 18, 2024 6:57 PM To: dev@airflow.apache.org Subject: Re: [VOTE] AIP-69 Remote Executor I agree with Jarek and Ash on this. I believe that the AIP as written documents the “what” and the “why” very well, but is too light on the “how”. I very much would like to see this AIP become reality as well, but I believe that we need some foundational elements such as API-44 and the “task context” concept to take this AIP to fruition with enough functionality to be meaningful. It’s entire possible that you are proposing something in between that’s feasible in the short-term, but that’s not clear to me yet. Vikram On Sat, May 18, 2024 at 8:46 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Agree with Ash. I also have a feeling that this is a very generic > description and some POC describing the approach we would like to take > here is needed to be able to vote on it. Feels a bit too early to vote on it. > > A lot of internals of the current (Airflow 2) way Airflow handling of > tasks running in executor are about database communication and if you > look closely, decoupling those internals to make it works with the > current executor API might be either very difficult or complex if we > stick to the current task <> airflow communication patterns. In some > ways, you already get "Remote Executor" by simply completing AIP-44 > (which precisely provides an HTTP-based. interface between tasks and > scheduler in a very similar fashion as AIP-69 proposal, but without > changing the communication patterns. > > As I see it it generally looks like AIP-69 is either the same as > AIP-44 or the same as the future "Airflow 3" task isolation we've been > discussing about and Ash is working on. Both AIP-44 and the future > Task Isolation aim to solve pretty much the same problem but AIP-44 in > a very "backwards compatible" way and limited change on how to achieve > "remote execution" > without actually making a lot of ripple effect on all other components. > > So I would really love to understand if AIP-69 is really something > in-between and how - but the "lightweitness" of description makes it > difficult to understand how different AIP-69 is from those two (of > course we should see the future task isolation AIP as well to > understand it better and know what kind of back-compatibilities it will > involve in Airflow 3). > > I think at the very minimum we should see the proposal of how the API > will look like between the task and executor (including the whole > life-cycle of tasks), the way how we are going to implement all the > complexity involved with task adoption, edge cases of scheduling, > execution semantic promises the API will hold (exactly-once, > at-most-once, at least once) - something that comes as given for > celery, the queuing mechanism and technologies used (how do we handle > distributed case where you have to manage multiple workers running and how > the tasks will be distributed etc. etc. > > For me the AIP currently is mostly documenting a "wishlist" but the > implementation details on which of those wishes we implement, and > which not, and how is very much absent. > > J. > > On Sat, May 18, 2024 at 10:41 AM Ash Berlin-Taylor <a...@apache.org> wrote: > > > Can we have a link to the pr please? The AIP doc itself is still > > light on what changes are actually needed > > > > On 18 May 2024 14:56:57 BST, Aritra Basu <aritrabasu1...@gmail.com> > wrote: > > >+1 (non-binding) > > >The proposal was a good read, would love to see it come up and > > >would > love > > >to help out if you need a helping hand. > > > > > >-- > > >Regards, > > >Aritra Basu > > > > > >On Sat, May 18, 2024, 7:15 PM Christian Schilling > > ><christian.lellm...@googlemail.com.invalid> wrote: > > > > > >> Hi Jens, > > >> > > >> Thank you very much for the proposal! > > >> This would be cool to have such a feature in Airflow. > > >> > > >> +1 non-binding > > >> > > >> Best, > > >> > > >> Chris > > >> > > >> Scheffler Jens (XC-AS/EAE-ADA-T) <jens.scheff...@de.bosch.com > .invalid> > > >> schrieb am Sa., 18. Mai 2024, 15:40: > > >> > > >> > Hi all, > > >> > > > >> > > > >> > > > >> > Discussion thread: > > >> > > > >> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F > > >> > %2Flists.apache.org%2Fthread%2F8hlltm9brdxqyf8jyw1syrfb11hl52k5 > > >> > &data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C4300b2481a734dd > > >> > aadac08dc775b921e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C > > >> > 638516482373030237%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi > > >> > LCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sda > > >> > ta=E4blX6HbfvW73CkOWMxQoc5XSYDiIToAcLqbh9zVFE8%3D&reserved=0 > > >> > > > >> > > > >> > > > >> > I would like to officially call for a vote to add a Remote > > >> > Executor feature to Airflow – via a provider package. All > > >> > details are > > documented > > >> in: > > >> > > > >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik > i.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-69%2BRemote%2BExec > utor&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C4300b2481a734ddaad > ac08dc775b921e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C6385164823 > 73039443%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiL > CJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Hbvz5Ct1PE52pUDFDAeHO > SYncOZ%2BOFi9eZQilaxeu9E%3D&reserved=0 > > >> > > > >> > > > >> > > > >> > Since the first draft was raised and the discussion thread went > > along, I > > >> > integrated a bit of feedback and now also added some technical > > details as > > >> > proposed development. After successful vote I’d raise a draft > > >> > PR as > > PoC. > > >> > As a big wave of emails was posted by Jarek after I dropped > > >> > this AIP > > I’d > > >> > like to highlight that I propose to make this a tactical > > implementation > > >> > which might be a base for some discussion how to distribute > > >> > work in > a > > >> > future Airflow 3.0. I would assume if interfaces and structures > > change, > > >> > rework will be needed and it is fully accepted that breaking > > >> > changes > > need > > >> > to be applied if moving to Airflow 3. > > >> > > > >> > > > >> > > > >> > Looking forward to releasing this for Airflow 2.10 (but > > >> > depending > how > > >> fast > > >> > I can make it and also depending on if somebody wants to join > forces). > > >> > > > >> > > > >> > > > >> > Consider this my +1 binding vote. > > >> > > > >> > > > >> > > > >> > The vote will last until 6:00 PM GMT/UTC on May 23, 2024, and > > >> > until > at > > >> > least 3 binding votes have been cast. I have it a bit longer as > usual > > >> > because of a public holiday in some countries. > > >> > > > >> > > > >> > > > >> > Please vote accordingly: > > >> > > > >> > > > >> > > > >> > [ ] + 1 approve > > >> > > > >> > [ ] + 0 no opinion > > >> > > > >> > [ ] - 1 disapprove with the reason > > >> > > > >> > > > >> > > > >> > Only votes from PMC members and committers are binding, but > > >> > other > > members > > >> > of the community are encouraged to check the AIP and vote with > > >> > "(non-binding)". > > >> > > > >> > Mit freundlichen Grüßen / Best regards > > >> > > > >> > Jens Scheffler > > >> > > > >> > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert Bosch > > >> > GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen > | > > >> > GERMANY | www.bosch.com > > >> > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > >> > jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> > > >> > > > >> > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB > > >> > 14000; > > >> > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > >> > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. > > Markus > > >> > Forschner, > > >> > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja > > >> > Rückert > > >> > > > >> > > >
