Hey everyone, First of all, thank you to everyone who has commented here and on the AIP!
I’ve come to realize that the original version of the AIP was too lengthy and overly detailed. To address this, I’ve revised it to focus on the problem and constraints, following the default AIP template. The implementation details have been moved to a separate subpage, which is explicitly non-binding and intended to support discussion. Locking in an implementation too early can limit useful feedback, which may explain the relative quiet in this thread. My hope is that this revised version will make it easier to review and encourage more input from the community. Since the AIP was reworked, some previous comments were automatically resolved. Please still feel free to review them, and continue the discussion either in this thread or directly on the AIP. I’d really appreciate it if you could take another look at the updated AIP and share your thoughts here. Thank you! Shahar On Sun, Apr 5, 2026 at 7:52 AM Bhavani Ravi <[email protected]> wrote: > You won't believe, I was thinking about something like this just last week. > So happy to see the community also moving in the same direction > > > On Sat, Apr 4, 2026 at 6:57 PM Aritra Basu <[email protected]> > wrote: > > > Overall a big +1 to this, I've added a few comments on it. But good read, > > would love to see this implemented and always open to helping get it to > the > > finish line! Thanks for the proposal! > > Thanks and Regards, > > Aritra Basu > > > > > > On Fri, Apr 3, 2026 at 6:33 PM Shahar Epstein <[email protected]> wrote: > > > > > Hello everyone, > > > > > > Since I first opened the thread discussing the MCP server, I've been > > > thinking for a long while about how we can practically bring > AI-assisted > > > debugging and operational insights directly into Airflow. As > > orchestration > > > environments grow more complex, the cost of troubleshooting, such as > > > navigating across Dag code, task instances, scheduler logs, and > > > configurations, translates directly to lost on-call time and delayed > > > pipelines. > > > > > > Today, organizations that want AI-assisted debugging are forced to > build > > > custom, ad-hoc integrations or rely on external paid solutions. This > > leads > > > to fragmented user experiences, duplicated effort, and most critically, > > > inconsistent security controls that risk exposing sensitive metadata or > > > bypassing Airflow's native Role-Based Access Control (RBAC). > > > > > > I think we can do better, by proposing AIP-101: Airflow AI Assistant - > > > Phase 1 (Read-only assistance). This AIP introduces an official, opt-in > > > plugin that provides a conversational UI directly within Airflow to > > answer > > > user questions about their instances, explain errors, and help > > troubleshoot > > > failures. > > > > > > To ensure this is done safely and securely, Phase 1 is strictly > > read-only. > > > The assistant does not modify Airflow state, nor does it operate > > > autonomously. Instead, it relies on the newly proposed Airflow MCP > Server > > > (AIP-91) as its data-retrieval engine. By leveraging the MCP standard, > > the > > > assistant guarantees that its answers are grounded in live system state > > > while strictly enforcing the authenticated user's RBAC permissions so > the > > > AI never accesses data the user cannot see. > > > > > > tl;dr of the proposed implementation: > > > > > > Packaging: Delivered as an opt-in, standalone plugin package within the > > > apache/airflow monorepo (with an independent release cycle). > > > Frontend: A conversational UI embedded directly in the Airflow web > > > interface. > > > Backend: A FastAPI-based plugin backend utilizing pydantic-ai to safely > > > orchestrate external LLM calls. > > > Data Access: Relies entirely on the Airflow MCP Server (AIP-91) to > fetch > > > read-only state. > > > > > > --- > > > > > > Because the assistant is heavily coupled with the secure tool-calling > > > execution provided by the MCP server, which is covered in a separate > AIP > > > (AIP-91) - please note the ongoing discussion here, as well as AIP-91 > > > itself: > https://lists.apache.org/thread/xgd66v6s7zf0xkvy3c7ysqvn4csgmw06 > > > https://cwiki.apache.org/confluence/x/G4q3FQ > > > > > > --- > > > > > > AIP-101 is available here: > > > https://cwiki.apache.org/confluence/x/8Ic8G > > > > > > A quick warning before you read: the AIP is quite long! (sorry Jarek) > > > Because integrating AI into an orchestrator opens up a lot of potential > > > pitfalls, [ChatGPT and] I tried to be extremely thorough in covering > all > > > the possible stuff that could go wrong :) > > > If you find a specific section to be overly detailed or repetitive, > > please > > > comment in the AIP and I'll try to handle it. > > > > > > I've managed to build a very inital POC, screenshots are available in > > this > > > section: > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620144#AIP101AirflowAIAssistantPhase1(Readonlyassistance)-BehavioralModel > > > > > > I would love to hear your thoughts. Please comment on the AIP and/or > > reply > > > to this thread. > > > > > > Thank you, > > > > > > Shahar > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
