Hey everyone, Thank you to everyone who made the dev call earlier today. I updated our meeting notes document in the Airflow 3.x wiki <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.x>to capture the notes. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.6>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for the 14th of August at the same time. The agenda is focused on development and test updates for Airflow 3.1. Please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.7> . Best regards, Vikram -- Below is the summary from the call: - Catch-up on action items from last call: - Airflow 3.1 Release plan and wiki update (Vikram) - Vikram walked the team through the Airflow 3.x wiki page <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.x>. - He said that this was primarily focused on Airflow 3.1 at this time, but intended to be expanded for 3.2 and more in the future - Vikram outlined the target dates for release branch, release candidates, with a goal of cutting the GA release the week of September 15th, i.e. about 2 weeks before Airflow Summit. - He said that all the work streams showed good progress, with several getting close to completion. He outlined that there was still a lot of work still to be done on the Task SDK split, but it was making good progress. - Vikram said that he was concerned about the UI filters stream of work and requested community help on that front. - Jens asked about the Example DAGs work and Vikram confirmed that he had deliberately not included that in the 3.1 scope because of timing and was targeting that work for the 3.2 release. - Development Updates - AIP-90 Human in the loop <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-90+Human+in+the+loop> update (Wei) - Wei had sent a recorded demo of the working functionality from a dev branch. - It was amazingly comprehensive, covering the entry operator, the approval operator, and the branch operator. - The demo covered both the UI implementation and the API endpoints which could be used for programmatic interaction. - Jens confirmed that he had reviewed the PRs and suggested that the AIP was ready for broader UX feedback. - Vikram mentioned that he had spoken with a large Airflow user who was currently using Airflow → Temporal → Airflow as a workaround and was excited for this functionality to eliminate the external dependency. - AIP-72 Task Execution Interface aka Task SDK <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK> update (Ash and Amogh) - Ash covered how the team was working on making the TaskSDK distribution independent of Airflow Core, as well as fixing upgrade blockers such as edge cases and virtual environment connection access. - He mentioned that the key goals for 3.1 were: - TaskSDK installable without Airflow Core, and - Go SDK complete and ready for use - Amogh spoke about the test framework which has now been merged. He said that this now spins up a real Airflow instance to test SDK interface integration with the API server. - Benchmark (AIP-59 Performance tests framework <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework>) update (Augusto Hidalgo) - Eugene said that Augusto was on vacation, but that the testing framework was running on both Airflow 2.x and Airflow 3, producing results. - Vikram asked if Augusto could present at the next dev call or alternatively to share the results async before the next dev call. - AIP-81 Enhanced security in CLI update <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-81+Enhanced+Security+in+CLI+via+Integration+of+API> - airflowctl release planning (Bugra) - Bugra covered the detailed test cases created for validating compatibility of airflowctl with Airflow Core 3.1 - Bugra said that most bugs were fixed, with one or two open issues. - The proposed release timelines was to start RCs by mid-August, with a target release date of September 1st. - Ash asked the question about compatibility with Airflow 3.0.3 vs. Airflow 3.1, at which point Jarek suggested integrating tests into CI similar to Amogh's work above. - Bugra said the that intention was to deprecate some of the existing CLI commands in the future, to avoid duplication with those commands in airflowctl. - AIP-86 Deadline Alerts <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488182> update (Dennis, Rajesh, and Ramit) - Rajesh said that the team expected to be code complete (for async callbacks) this week. - Dennis confirmed that no UI requirements existed for this AIP, since it was purely a code invocation with notifications. - Dennis confirmed that async callbacks running in the Triggerer were working now and attempted a demo, but the demo gods interfered. - Ramit said that Sync callbacks to be run in the workers would be coming next week. This is similar to the 2.x SLA callback functionality and would be the easiest upgrade path. - Discussion Topics - Migration blocker issues (Elad) - Elad added a label for user-reported upgrade blockers, so that we could easily identify issues preventing 2.x → 3.x upgrades. - Jarek asked the community to help review and give inputs, even if they were unable to code a fix. - Action Items - Pierre: Request help from the community to work on UI filters (on the dev list and on slack) - Augusto Hidalgo: Share performance test results async when back from vacation - Dennis / Ramit: Demo the sync callback functionality in the next dev call - Community: Review Human-in-the-loop UX and provide feedback - Community: Contribute to UI filters work and upgrade blocking issues -- Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
