Hey everyone, Thank you for attending the dev call on Thursday. I updated our meeting notes document in the Airflow 3.x wiki <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.x>to capture the notes. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.12>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for the 4th of December at the same time. Please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.13> . Best regards, Vikram -- Below is the summary from the call: - Catch-up on action items from last call: - Swim lane issue / PR tracking update (Vikram Koka) - Vikram shared the output from the automated script he had created for tracking all open issues and PRs. - Using the 3.0 release date as a starting point, there are 308 open issues and 325 open PRs. - Core Airflow had about 200 open issues out of which the UI and API swim lane had a third of all open issues and PRs. - Helm Chart had 18 open issues and 14 open PRs. - Dev tools (including CI) had a surprisingly high number (34) of open PRs. - Providers had about a quarter of the overall, but not yet classified in detail, because of a script bug. - Vikram also added that he would be updating the 3.x page with the updated swim lane information including metrics, and the swim lane leads. - Vikram asked for volunteers to fill out the swim lane leads for areas which are backlogged. - Jens said that he would take the lead on Helm Chart, after chatting with Jed since Jed was now back from leave. - Jarek suggested that "draft PRs" be excluded from the counts. - Shahar asked about more detailed labels and leveraging the issue bot, but Vikram said that the issue bot was currently often wrong, requiring manual re-tagging. - Development Updates and Presentations: - Airflow 3.1.x patch release update (Ephraim Anierobi) - Ephraim said that Core Airflow 3.1.3 was released last week - Ephraim added that 3.1.4 was planned for the first week of December, adding that it was originally targeted for a week earlier, but wanted to avoid Good Friday week. - Jarek added that this was likely the last release before Christmas / New Year. - Bugra added that airflowctl patch release dependencies were being resolved and that a new release candidate was being created. - UI performance issues (Pierre / Brent) - Pierre said that the N+1 query issues were resolved and that the fixes were released as part of 3.1.3 - Pierre went on to say that the next priorities included adding missing indexes for sorting/filtering and that these needed to be configurable since the criteria could be different across user organizations. - Brent added that additional work would be needed for View specific enhancements, such as Grid View performance enhancements. - There was also a question raised about documenting guidance for the deployment of API servers for scale. - AI-generated PR management strategy - Vikram asked the question for the UI team about how many of the generated PRs (since the UI had the largest number of PR contributions) were AI-generated and if there were any quality issues - Brent responded that the PRs ranged from AI-assisted good PRs to very low-effort PRs which hadn't had enough thought behind them. - Pierre said that his approach was to match the review effort to the contribution effort i.e. spend very little time reviewing low-effort PRs and just give high level feedback for obviously AI-generated PRs, so that time wasn't wasted. - Overall the team felt that they were spending a disproportionate amount of time reviewing poor quality PRs. - Jarek said that ASF suggests (but does not mandate) disclosure of LLM usage in contributions - This led to a discussion on needing to update the contributing guide to reflect expectations on AI usage in the PR creation process. Ash took that action item to update the contributing guide. - Discussion topics: - Why more providers needed ? (Jarek Potiuk) - Jarek raised the challenge of increasing provider breadth, by saying that despite incoming requests for additional providers and people willing to contribute providers, we were limited by maintainer capacity. Jarek said that he had collaborated on a proposal with Vikram and others and would like to see this proposal move forward. - Vikram described the proposal for new providers as follows: - An Incubation step for new providers, then proceeding to "mainstream / production, and then to "attic / deprecation" if no longer being used - Contributors would need to take ownership responsibility for maintenance. - Similar to the issue / PR metrics for core Airflow above, we would track and publish metrics around open issues, PR reviews, CVE fixes per provider - Special case handling for mature, stable providers such as Slack. - This process would be similar to how we handle translations with Shahar leading translations as a committer, but with individuals taking ownership for the actual creation and maintenance. - We would continue to have only Apache Airflow committers as people who can merge, similar to translations. - Radu (from Vespa) said that they wished to contribute a Provider to Airflow. Radu went on to say that the shared ownership model described earlier would be mutually beneficial and that they would be incentivized to maintain their contribution, so that their users have a good experience - Vikram to update the AIP and send out a follow-up to the dev list - Multi-team to be aimed for 3.2 or 3.3 release (Vincent Beck / Niko) - Vincent wasn't at the meeting, but Niko chimed to say that they would like to target portions of the multi-team AIP for 3.2, with other elements following on in 3.3. - Vikram asked about the status of the Deadline Alerts AIP, and Niko said that the remainder of the Notifier improvements were planned for 3.2 - There will be a significant portion of the next dev call (on Dec 4th) dedicated to 3.2 scope planning. -- Vikram Koka Chief Strategy Officer Email: [email protected] <https://www.astronomer.io/>
