Hey everyone, I updated our meeting notes document in the Airflow wiki to capture the notes from our dev call yesterday, the 7th of November. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.12>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. There's already an initial agenda for our next dev call, which is scheduled for 21th Nov. If you would like something to be added to the proposed agenda for that meeting, please add it here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-(Proposed)Agenda.2> or let me know. I am very excited by the progress on Airflow 3 to date and very much looking forward to the next milestone in a couple of weeks. Truly appreciate all the hard work being put into this by all the contributors! Best regards and talk to you all soon, Vikram -- Below is the summary from the call yesterday: - Follow-up on action items from the last call: - Update on AIP-79 with respect to what's left (Jed Cunningham, Vincent Beck, Jens Scheffler): Jed shared that it was unclear at this time if AIP-79 was blocked on AIP-38 (new UI). - Jed to report back by the next dev call if he could magic back the older Fab provider, so that the old Fab provider could exist independent of the new Fab provider, so that we are not blocked by the new UI work - The Connection form for the new UI is another item which needs to be done and is not blocked by anything else. Jens's intention is to wait and build the Connection form based on the Trigger form in the new UI. Though to be clear, no one actively owns the Connection form work at this time, but Brent and Jens believe that this will naturally follow as part of the AIP-38 work. - Clarity from Jed and Vincent that the new Fab provider is needed for the Auth manager and more and is not in conflict with the work that Jed is proposing above. - Breaking changes update (Vikram Koka): Vikram walked the group through his thoughts around capturing all changes that users need to worry about as part of upgrading to Airflow 3. Specifically to structure the newsfragments with a template. Vikram proposed an initial template with the following elements: - DAG code changes - Import changes - Other DAG code changes - e.g. SubDAGs - please note if it was already marked as “deprecated” or “experimental” - Stable API changes: both invocation and result - CLI changes: - Airflow configuration changes - Airflow.cfg changes - Integration changes - Behavior changes: Example Backfill or Execution date changes - Plugin changes - There was immediate agreement within the group for the above and the desire for a concrete template. Vikram to publish the above and then it could be added as a pre-commit. Vikram to also create an initial Upgrade / Change overview page, which would have example results based on some of the initial newsfragments (around 10). - Development updates and presentations: - Airflow 3 workstreams update (Vikram Koka): Vikram walked the group through the updates to the Airflow 3 workstreams <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+-+Workstreams> and Airflow 3 dev milestones <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Development+Milestones> pages to reflect the high level status of the Airflow 3 project as well as finer level monthly details. Positive feedback from Rajesh and the rest of the team on the overall summarization in the Workstreams page, with the more granular detail in the monthly dev milestones page. Vikram reiterated the ask for more help on the AIP-38 UI work. Brent said that we are now ready for others to pickup work now, since the back-end APIs are ready to use. Brent to create a Slack sub-channel for UI development specifically for AIP-38 work. - Discussion topics: - Update other candidates from the Workstreams list: - Remove SLAs and AIP-57 Refactor SLA Feature - obsoleted by AIP-86 Deadline alerts? (Dennis Ferruzzi) Dennis confirmed that this entry was now obsoleted by AIP-86 Deadline Alerts. - Remove poke, reschedule mode from Sensors? Kaxil Naik said that there was a discussion in one of the early dev calls and that he felt that this should be removed to really simplify the code and choices for the users. Daniel Standish felt that there was a difference between support in Operators vs. Airflow Core. He felt that the support in Operators was very confusing. Shubham Mehta felt that this capability was useful for very long running tasks (greater than 12 hours) in the past, but wasn't sure if it was still necessary after Airflow 2.8, and would get back to the team about the same. Jed Cunningham felt that it would be much simpler to take it out. There was a good discussion with multiple members of the team and both Igor Kholopov and Shubham Mehta confirmed that they didn't have a strict "no" at this time, but would need to confirm. After the discussion, the action item was for Kaxil Naik to start an email discussion around this in the dev list. - Inspect & Simplify Airflow Configurations: Shubham Mehta and Dennis Ferruzzi confirmed that we should just defer this to post Airflow 3.0. - OTEL integration into Airflow Scheduler (Daniel Standish): - There was a long discussion about the task log reading introduction into the very performance intensive Scheduler loop and the challenges therein. - Kaxil Naik to start an async discussion around what changes need to be done to either refactor or revert these changes for OTEL traces. - Action items for next dev call: - Jed Cunningham to confirm back regarding the older Fab provider changes for compatibility. - Vikram Koka to create a template for newsfragments for flagging upgrade issues. And to create an initial Airflow 3 upgrade considerations page based on some of the existing PRs. - Brent Bovenzi to create an Airflow 3 UI slack channel to facilitate collaboration on Airflow 3 UI dev ("airflow-3-ui") - DONE - Kaxil Naik to start email discussion regarding removal of Poke and reschedule mode from sensors. - Kaxil Naik to send an email about the issues with OTEL integration for Traces into the Scheduler.
