Hi Shahar, I read through the updated AIP and something that went through my mind while reading this and AIP-102 is that no matter if it's a business user or an LLM, context is king. In your potential implementation appendix, you list six different types of mostly technical metadata to ground the LLM, but nothing specifically for an LLM .
Similar to doc_md for human users, what do you and others think about adding an agents_md param specifically for LLM context? Bryan On Sat, Apr 18, 2026, 11:03 AM Shahar Epstein <[email protected]> wrote: > Hey everyone, > > First of all, thank you to everyone who has commented here and on the AIP! > > I’ve come to realize that the original version of the AIP was too lengthy > and overly detailed. To address this, I’ve revised it to focus on the > problem and constraints, following the default AIP template. The > implementation details have been moved to a separate subpage, which is > explicitly non-binding and intended to support discussion. > > Locking in an implementation too early can limit useful feedback, which may > explain the relative quiet in this thread. My hope is that this revised > version will make it easier to review and encourage more input from the > community. > > Since the AIP was reworked, some previous comments were automatically > resolved. Please still feel free to review them, and continue the > discussion either in this thread or directly on the AIP. > > I’d really appreciate it if you could take another look at the updated AIP > and share your thoughts here. > > Thank you! > Shahar > > > On Sun, Apr 5, 2026 at 7:52 AM Bhavani Ravi <[email protected]> wrote: > > > You won't believe, I was thinking about something like this just last > week. > > So happy to see the community also moving in the same direction > > > > > > On Sat, Apr 4, 2026 at 6:57 PM Aritra Basu <[email protected]> > > wrote: > > > > > Overall a big +1 to this, I've added a few comments on it. But good > read, > > > would love to see this implemented and always open to helping get it to > > the > > > finish line! Thanks for the proposal! > > > Thanks and Regards, > > > Aritra Basu > > > > > > > > > On Fri, Apr 3, 2026 at 6:33 PM Shahar Epstein <[email protected]> > wrote: > > > > > > > Hello everyone, > > > > > > > > Since I first opened the thread discussing the MCP server, I've been > > > > thinking for a long while about how we can practically bring > > AI-assisted > > > > debugging and operational insights directly into Airflow. As > > > orchestration > > > > environments grow more complex, the cost of troubleshooting, such as > > > > navigating across Dag code, task instances, scheduler logs, and > > > > configurations, translates directly to lost on-call time and delayed > > > > pipelines. > > > > > > > > Today, organizations that want AI-assisted debugging are forced to > > build > > > > custom, ad-hoc integrations or rely on external paid solutions. This > > > leads > > > > to fragmented user experiences, duplicated effort, and most > critically, > > > > inconsistent security controls that risk exposing sensitive metadata > or > > > > bypassing Airflow's native Role-Based Access Control (RBAC). > > > > > > > > I think we can do better, by proposing AIP-101: Airflow AI Assistant > - > > > > Phase 1 (Read-only assistance). This AIP introduces an official, > opt-in > > > > plugin that provides a conversational UI directly within Airflow to > > > answer > > > > user questions about their instances, explain errors, and help > > > troubleshoot > > > > failures. > > > > > > > > To ensure this is done safely and securely, Phase 1 is strictly > > > read-only. > > > > The assistant does not modify Airflow state, nor does it operate > > > > autonomously. Instead, it relies on the newly proposed Airflow MCP > > Server > > > > (AIP-91) as its data-retrieval engine. By leveraging the MCP > standard, > > > the > > > > assistant guarantees that its answers are grounded in live system > state > > > > while strictly enforcing the authenticated user's RBAC permissions so > > the > > > > AI never accesses data the user cannot see. > > > > > > > > tl;dr of the proposed implementation: > > > > > > > > Packaging: Delivered as an opt-in, standalone plugin package within > the > > > > apache/airflow monorepo (with an independent release cycle). > > > > Frontend: A conversational UI embedded directly in the Airflow web > > > > interface. > > > > Backend: A FastAPI-based plugin backend utilizing pydantic-ai to > safely > > > > orchestrate external LLM calls. > > > > Data Access: Relies entirely on the Airflow MCP Server (AIP-91) to > > fetch > > > > read-only state. > > > > > > > > --- > > > > > > > > Because the assistant is heavily coupled with the secure tool-calling > > > > execution provided by the MCP server, which is covered in a separate > > AIP > > > > (AIP-91) - please note the ongoing discussion here, as well as AIP-91 > > > > itself: > > https://lists.apache.org/thread/xgd66v6s7zf0xkvy3c7ysqvn4csgmw06 > > > > https://cwiki.apache.org/confluence/x/G4q3FQ > > > > > > > > --- > > > > > > > > AIP-101 is available here: > > > > https://cwiki.apache.org/confluence/x/8Ic8G > > > > > > > > A quick warning before you read: the AIP is quite long! (sorry Jarek) > > > > Because integrating AI into an orchestrator opens up a lot of > potential > > > > pitfalls, [ChatGPT and] I tried to be extremely thorough in covering > > all > > > > the possible stuff that could go wrong :) > > > > If you find a specific section to be overly detailed or repetitive, > > > please > > > > comment in the AIP and I'll try to handle it. > > > > > > > > I've managed to build a very inital POC, screenshots are available in > > > this > > > > section: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620144#AIP101AirflowAIAssistantPhase1(Readonlyassistance)-BehavioralModel > > > > > > > > I would love to hear your thoughts. Please comment on the AIP and/or > > > reply > > > > to this thread. > > > > > > > > Thank you, > > > > > > > > Shahar > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > >
