Hey everyone, Thank you for attending the dev call earlier today. I updated our meeting notes on the Airflow wiki and the link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.36>
To everyone who attended the meeting, please check the summary and add anything I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please ask questions if something is unclear. Our next meeting is scheduled for the 7th of May at the same time. It is scheduled for 8 a.m. Pacific Time. If you would like to discuss a particular topic, please let me know if you want to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.38> . Best regards, Vikram -- Below is the summary from the call: - Note: The call only got through roughly half the agenda. AIP-99, AIP-103, MCP (AIP-91), and AI Assistant (AIP-101) are deferred to the next dev call - Catch-up on action items from last call: - *Airflow 3.3 wiki update (Vikram)* - Vikram shared that the wiki update is in progress but not yet complete. He walked through the current state of the Airflow 3.3 scope page. - Items currently tracked for 3.3 include: - Continued Asset Partitions work (segment-based partitions and remaining date-based use cases), - AIP-103 Task State Management, - Pluggable retry policies (a positive vote is expected), - Deadline Alerts and Multi-team follow-on work, - AIP-99 agentic work targeting a 1.0 release, J - ava and Golang Task SDK work, and - AIP-94 CLI decoupling via Airflow CTL. - Vikram noted that Jens has a couple of follow-on AIPs from the 3.2 cycle that he hadn't yet captured, and asked Jens to add or tag them. Jens confirmed he still needs to write the AIP, which relates to pooling, deferring, and task-level limits, so no number has been assigned yet. - Bugra noted in chat that AIP-94 was missing from the wiki and would add it after the call. He also confirmed in the call that tickets for AIP-94 are now being created with a 3.3 milestone, following the 3.2 release. - Vikram said he will update the wiki to reflect AIP-94. - Vikram stated the target for 3.3 is to release by mid-June at the latest, ideally early June. He expressed a desire to move toward a roughly every-two-months release cadence going forward. - Jens suggested the approach should be timeboxed, with any features not ready being deferred to a later release rather than delaying the whole milestone. Vikram agreed. - Rajesh asked about what would be showcased at Airflow Summit in late August. Vikram noted that the agentic AI work and task state management were his personal focus areas from a Summit relevance perspective, and added that a 3.4 release in August before Summit is possible if the cadence holds. - Ash endorsed more frequent, smaller releases, noting they are less painful to produce and easier to upgrade to from the user side. He also noted that the client-server separation should help users upgrade more frequently going forward. - Ash and Jarek agreed to follow up async on the possibility of completing the airflow to airflow-core rename and related import cleanup for 3.3, with Amogh likely involved. Jarek noted this work is very close to his heart and he would like to dedicate more time to it. Jens added that cleanly separating the API server deployment from the DAG parser, and clarifying the provider split between plugin code and operators, should also be documented, ideally following the documentation-first approach discussed in the last retro. Jarek agreed and noted there is an unpublished internal document from Ash that could serve as a starting point. - Pierre shared in chat that there was a successful Airflow community meetup in Paris the evening before, with around 80 attendees, which was notably larger than usual for tech meetups in the city. - Airflow 3.3 Development Updates: - *AIP-72 Java Task SDK demo (Jason Liu / TP Chung)* - TP and Jason were unable to join due to time zone constraints. A pre-recorded demo by Jason Liu was shared by Phani Kumar during the call. The recording is also available here <https://drive.google.com/file/d/1oNzIEyTEM-2YHO-DaSuRAqgR0P78fFnF/view> . - Ash noted there were a couple of last-minute naming changes in the code that the demo had not yet been re-recorded to reflect, including a change to the stub(language=) parameter name. - The demo showcased the Java Task SDK, covering three key interfaces: Task (with an execute method), access to Airflow features including XCom, connections, and variables, and a DAG bundle interface for exposing DAGs to the DAG processor. - Two approaches for defining DAG structure were demonstrated. The first uses a Python stub DAG with @task.stub to reference Java tasks, which was the approach supported in the earlier Golang SDK work. The second is a pure Java DAG, where the DAG structure is defined entirely in Java and metadata from the JAR file is used by the DAG processor. - A key highlight of the demo was a true multi-language DAG showing Python and Java tasks running within the same DAG, with XCom passing data bidirectionally between the two languages. Task dependencies in the pure Java DAG are defined via a simple addTask API with explicit dependency lists. - Error handling was also demonstrated, with Java-side stack traces forwarded through the Task SDK to Airflow logs. - The overall reaction from the team was very enthusiastic, with Jarek describing it as "really cool" and Jens noting how impressive the Java support was. - Ash explained that architecturally, a Python coordinator process picks up the task and hands it off to Java for execution, handling log capture and secrets mediation so that the secrets backend does not need to be reimplemented in each language. - Several open questions were raised and will be shared with Jason and TP for async follow-up, including: how Java DAG bundles handle additional JAR dependencies (fat jar vs. classpath), whether the DAG processor needs the JAR present for pure Java DAGs or whether serialized DAG payloads are used instead, and whether the same approach applies to Go. Ash noted a PR or dev list post from Jason and TP is expected soon. Vikram noted he will make sure the questions are shared with Jason and TP for async response. - Discussion Topics: - *Agentic PR Triage skill (Jarek Potiuk)* - Jarek gave a demo of a skill-based agentic approach he has developed for PR triage, built on Claude Code. The demo experienced some slowness due to GitHub rate limits during the live session, but the overall approach and results were explained. - The triage skill is implemented entirely as English-language Markdown files describing classification criteria, conditions, response templates, tone rules, and batching/caching behavior. GraphQL queries are embedded inline to pull PR data. The skill was itself generated by Claude based on an earlier code implementation. - The workflow is conversational: a maintainer opens Claude Code in the Airflow repo and issues natural language commands such as "triage oldest PRs" or "triage PRs I am a reviewer of." The agent prefetches batches of PRs in the background while the maintainer reviews the current batch, making it fast enough to process around 300 PRs in a couple of hours. - A key design principle is that the agent never sends a comment without explicit maintainer confirmation. Jarek emphasized this as a golden rule. - Results from Jarek's first full triage run: out of 200 PRs triaged, 130 were closed (over 60% where contributors did not respond), 70 were merged, 75 received responses, and 149 were marked as ready for review. Around 40% of the remaining 300 contributor PRs are now in draft status awaiting action. - Brent raised concerns about the impersonal nature of AI-generated comments sent from a personal account, and the risk of discouraging genuine contributors. He noted that some automated reviews add noise rather than value, particularly when they simply restate pending comments already left by a human reviewer. Aritra suggested that adding an explicit banner or disclosure indicating AI generation would be a good middle ground and would be easy to implement as a single prompt change. Jarek agreed and said he would add this. - Jens thanked Jarek for tackling the backlog but raised a specific case where a colleague's PR for a critical production triggerer issue was converted to draft by the triage agent despite having active maintainer attention. Jarek acknowledged this and asked for feedback to be posted in the dedicated Slack channel he created for triage issues, so rules can be refined. - Kaxil shared his complementary approach, which uses GitHub Copilot for opt-in PR reviews and a more deterministic skill setup with hooks, bash scripts, and evals to prevent the agent from acting without confirmation. He noted that agents are probabilistic and that hooks are important for enforcing determinism in critical steps. He mentioned a blog post about this approach is expected soon. The key distinction between the two approaches is that Kaxil's is focused on pre-review AI assistance with manual control before posting, while Jarek's is focused on post-submission automated triage with human oversight at confirmation time. - Vikram noted that entire companies are being built to solve this problem and encouraged the team to remain open to other open source or commercial alternatives in addition to Jarek's skill. - Kaxil shared a link to the existing code review instructions file in the Airflow repo: .github/instructions/code-review.instructions.md <https://github.com/apache/airflow/blob/main/.github/instructions/code-review.instructions.md> . -- Vikram Koka Chief Strategy Officer Email: [email protected] <https://www.astronomer.io/>
