Hey everyone, I updated our meeting notes document in the Airflow wiki to capture the notes from our dev call yesterday. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.10>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. I will follow-up with the agenda for the next meeting (scheduled for 24th Oct) in a few days. If you would like something to be added to the proposed agenda for that meeting, please add it here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-24October2024> or let me know. Best regards, Vikram --- Below is the summary from the call yesterday - Follow-up on action items from the last call: - Jens Scheffler walked through the draft diagram for splitting up the providers into Operators / Executors / Plugins, based on the long term (Airflow 3) view of where they would be used. Jens stated how a follow-up proposal could be to move the “Extra Links” to be stored in the Airflow meta-database, since they are shared across components. A similar issue is for the Extra Fields for Connections since they are also shared across components such as DAG Parser and Webserver. Vikram Koka added that this separation of providers was primarily driven from the packaging and distribution strategy as well as the release strategy to be used for the providers going forward because of the Core vs. Task SDK split coming as part of Airflow 3. The consensus from the group was a proposal needed to be written as a follow-up and shared with the dev list. Jens suggested that Vikram take a stab at the draft of the proposal and Vikram agreed to take it as an action item. - Vikram Koka walked through the monthly Development milestones for Airflow 3 <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Development+Milestones>, across all AIPs. The feedback from the group was positive and it was felt that it was helpful not only for the contributors, but the larger community. Kaxil Naik suggested that this should be a page within the top-level Airflow 3.0 page on the wiki. Jens Scheffler mentioned that he needed feedback on this PR on AIP-69 based on the AIP-72 work and Ash Berlin-Taylor responded he would review that async. Vikram asked the AIP owners to update the milestones with details and status as appropriate, but also to notify him of changes for potential ripple effects. - Presentations: - Rahul Vats walked through the high level Airflow 3 test plan <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Test+Plan> which included all the AIPs targeted for the release as well as some initial end to end scenarios. Rahul created these based on the AIP documents and asked for the AIP authors to review these and add additional end to end scenarios as appropriate, so that any breaking issues could be identified as part of the development process itself. Rahul recommended that these be used as the source of truth and be a foundation for the system tests. He also referenced the use of the testing slack channel #sig-airflow-3-qa-testing and asked for help from the community in testing this major release. Rahul took the action-item to articulate “what should be covered in the AIP unit tests vs. here in the E2E tests” with examples. Rajesh Bishundeo asked about performance testing AIP-59 in the context of Airflow 3. Michal Modras took the action-item to follow up with the original author of AIP-59 and see if the author or an alternate Google developer could give an updated proposal in time for Airflow 3 testing. - Brent Bovenzi shared how the team has been using the UI update opportunity (AIP-38) for a UI Design refresh and presented mockups including the new home and DAGs list page. This had also been shared as part of the Airflow Summit. Pierre Jeambrun presented a progress update on the FastAPI-based approach, including a quick demo (AIPs 84). Brent also presented a progress update on AIP-68, including a quick demo of the approach for custom react components (plug-ins) based on FastAPI. - Discussions: - Jens Scheffler brought up the topic of continued support for Python 3.9 in the upcoming Airflow 2.11 release. I.e. if it was required or if it could be dropped. The consensus was that there were no objections from the attendees, but the next step was for Jens Scheffler to send an email out to the dev list about this asking for a “lazy consensus”.
