I’m calling for a vote on this AIP https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK
Previous discussion thread was https://lists.apache.org/thread/drfmwbx5brtycjkmtf51owrmhvmlx2lk Based on the recent discussions I have added the section titled "Dag Parsing Changes”. This AIP (and the key point of removing direct DB access from tasks) will cause some breaking changes to custom operators, and it’s hard for us to judge how big an impact that change will have, but this AIP gives us so many benefits that it is my view that even if we can’t help migrate custom operators it’s worth it overall. A reminder of the main thing this AIP gives us: - No more direct DB access from workers — the number of active connections to the DB is commonly a scaling pain point with larger Airflow deployments - Improved security due to the above — No more can a task access any connection it likes, or go and affect the state of other DAGs - Gives us the AIP that enables Jens’ Remote Worker AIP to operate sensibly - It will let users _natively_ write tasks in languages — a nicer UX than just BashOperator. And for completness, the downsides as I see them: - It will (possibly irreparably) break a small number of custom operators - It is a lot of work (but achievable in the timeframe) The vote will run for 5 days and last till next Tuesday 16th of July 2024 11:00 UTC. Everyone is encouraged to vote, although only PMC members and Committer's votes are considered binding. This is my +1.
