Apologies for the delay, thanks for your great insights Jarek! I'll consider them when starting to work on the AIP and initial implementation.
Shahar On Tue, Jun 17, 2025 at 7:43 PM Jarek Potiuk <ja...@potiuk.com> wrote: > I wanted to pass my learnings from the AI Plumbers conference > https://lu.ma/vqx423ct that I was at just before Berlin Buzzwords. This is > where AI open source "creators" meet and discuss on the "low level AI > engineering" - and MCP was one of the subject of the unconference part we > had - and of course I jumped in and took part in it and listened (mostly) > and spoke (a bit) on our case (and listened to feedback) > > My take on it - and this summarises > 2 hour discussion we had.... > > * MCP is about "intents" of people (or AI imitating people) - not about > what you can do in the system, but what you want to achieve. MCP is really > a bridge between "engineering" driven API which describes low-level > capabilities of the system and "Product" driven view of what users want to > achieve (so far, so good - using use cases as starting point for MCP > running server is likely good idea) > * There are different users of your system and different expectations of > what they want to achieve. It's nearly impossible to figure out what > exactly every single user of Airflow would want from it - we know about > vastly different cases on how airflow is used, and we will likely never be > able to create an MCP server that would not be limited to a very small set > of use cases - which means that if we release an MCP server, it will be > usable by some but non-usable by others. This is a striking contrast with > the API approach where we deliberately create an API to be composed of a > small set of actions you can use and basically anyone can tailor it to > their expectations on how they are using it. > * There is also one important aspect of MCP server that is currently very > hotly debated - i.e. security. we should make sure that whatever we make > possible to do by the "agent" is not breaking the security assumptions and > model of those who run Airflow. > * Connected to that - whenever the MCP server has to get more permissions > to get more data that only more "privileged" user can access, there should > be human-in-the-loop involvement where the user can clearly give > permissions to agent to read specific data - or better, read the data and > hand it over to the agent. This is a crucial aspect in case we have some > parts of the system that require more permission than "default". > > In essence - no-one yet knows really how to do "good MCP" - we are very > early stage and things will get better over time in understanding what and > how to implement parts of it - and we should focus now on things that are > "known" leaving the "unknown" for our users to fill (and adapt and modify > in the future) > > Generally speaking I think summary of the conclusions I had was: > > * The term that I really liked is that we should likely not have our own > MCP server that people will be treating as a black-box, but that we should > be "MCP READY" -> i.e. we should give our users an option to have MCP > server that they will have to configure and adjust in the way that will be > good for them and that they will have to essentially build themselves, > building in prompts, capabilities they expect as "users" from the system, > using the building blocks, scaffolding and tooling that we will provide > them. > > * Open API and generating MCP server "base" from it is a good idea - it > provides a very good base in terms of security and we can turn our expert > knowledge of airflow internals into better describing the API. Ee should > focus indeed (and having a scaffolding) on improving description and > documentation of the Open API, use the generator to generate "scaffolding > MCP" from Airflow's openapi and tell the user "Here are next steps that > you have to do in order to have your MCP server" is likely most effective > way and safest. We should not pretend that we can implement the best > prompts for our users, most likely users should be able to write their > prompts that will better describe what they want from the system - we can > come with some examples, and guide the user on what they should do with > them and they will have to write, adapt and adjust their prompts - > especially that each of our users has different dag structure, functions, > criticality, etc. etc. Generally - Airflow is so generic that only the user > who knows their DAGs can properly descibe in a human language what they > might want from the system. Our OpenAPI should have a good-enough > description that it should be "easy" to describe the MCP prompts and such > that are good for them. > > * the security we have from making sure the MCP server can **only** do > whatever our OpenAPI allows - is already a very good start for security. > But we should also implement a layer of "Human In The Loop" that will allow > the MCP server to have access to more sensitive data - this is something > that we currently do not have, generally allowing the user to specify > access to only part of what they have access to - and exposing it to the > MCP layer in secure way. Without it, MCP server will be very limited in > many cases as it will not have access to necessary data to make it powerful > enough, but we also (by human in the loop) need to be able to expose more > data when needed (and control it by the human) > > * in essence - rather than having a single "MCP server to rule them all" - > we should have a mechanism that users should be able to build their own > server easily and use some of the building blocks we provide them to do so > (especially around security). But they should be ultimately in control of > what "intents" they might have when interacting with Airflow. > > Those are the thoughts and learnings I had. Maybe that will be helpful in > our > > J. > > > On Wed, Jun 4, 2025 at 1:09 PM Jason Sebastian Kusuma < > jsjasons...@gmail.com> > wrote: > > > I want to present a use case I have in mind. I think it would be useful > for > > task delay analysis for tasks in complex workflows spanning multiple DAGs > > or assets. > > > > We can ask questions like "why does task X finished later than > yestersay", > > where it could fetch upstream tasks and upstream DAGs execution durations > > from this day and yesterday, use them as context and conclude which > > upstream tasks causes the delay. > > > > On Wed, Jun 4, 2025, 5:53 PM Jason Sebastian Kusuma < > jsjasons...@gmail.com > > > > > wrote: > > > > > Hi all, > > > > > > Glad that my initial slack thread has been brought into an interesting > > > discussion. Would love to contribute to this effort. > > > > > > > > > Best regards, > > > > > > Jason Sebastian Kusuma > > > > > > On Wed, Jun 4, 2025, 5:07 PM Sumit Maheshwari <sumeet.ma...@gmail.com> > > > wrote: > > > > > >> At Uber, we've been using GenAI to debug task failures and suggest > > further > > >> mitigation steps to our users. We would love to collaborate on this > > >> project > > >> and share our learnings & contribute. > > >> > > >> On Mon, Jun 2, 2025 at 11:09 AM Amogh Desai <amoghdesai....@gmail.com > > > > >> wrote: > > >> > > >> > I agree that a use case driven approach is the way to go too. > > >> > > > >> > When we go full-blown, it is easy to lose track of the intention we > > >> started > > >> > with. > > >> > > > >> > Some of the use cases related to debuggability improvements is > > >> something we > > >> > already > > >> > had a north star for in: > > https://github.com/apache/airflow/issues/40975, > > >> > with a good amount of > > >> > analysis already done. Maybe you can get some data here that will be > > >> > useful. > > >> > > > >> > > > >> > Thanks & Regards, > > >> > Amogh Desai > > >> > > > >> > > > >> > On Fri, May 30, 2025 at 9:27 PM Avi <a...@astronomer.io.invalid> > > wrote: > > >> > > > >> > > Hi All, > > >> > > > > >> > > Had a chat with @Kaxil. And we discussed on use-case first > approach > > >> > rather > > >> > > than opening up to all possibilities, on building and maintaining > > the > > >> MCP > > >> > > server. > > >> > > > > >> > > A few use-cases which I had in mind were: > > >> > > - Debugging task failures > > >> > > - Schedule sparse across all dags for better resource utilization. > > >> > > - Recommend provider usage in place of native python codes. > > >> > > - Cross-DAG Dependency Analysis > > >> > > - Migration/Refactor Planning > > >> > > > > >> > > We would need an ongoing effort on this, as every other day > > something > > >> new > > >> > > comes up. Could be more optimization, or newer use-cases. We also > > >> need to > > >> > > maintain the security aspect of if. Like, where do we recommend > > >> running, > > >> > > auth and transport methods, etc. > > >> > > > > >> > > I made a new release today with FastMCP and discovery of tools > based > > >> on > > >> > > category. After seeing today that FastMCP mount and sending > refresh > > >> the > > >> > > tools list notification is working (*which it wasn't 3 months > > back*). > > >> > > > > >> > > Things I have on my roadmap: > > >> > > - HTTP based transport specifically for OAuth currently it is > stdio > > >> only > > >> > > for it to be installed in a plugin (*airflow-wingman*) as a > > >> dependency. > > >> > > - Waiting for search tools to be implemented. > > >> > > - Decide to bake prompts with the server, perhaps. > > >> > > - Very minimal Docs as a resource which tells about recent changes > > in > > >> > > Airflow behavior (Example: *every LLM till date still tries to > write > > >> > > schedule_interval instead of schedule*) > > >> > > > > >> > > @Shahar feel free to take a lead and see if there are things you > > would > > >> > like > > >> > > to cherry-pick from it. > > >> > > > > >> > > Thanks, > > >> > > Avi > > >> > > > > >> > > On Fri, May 30, 2025 at 9:57 AM Avi <a...@astronomer.io> wrote: > > >> > > > > >> > > > 💯 agreed > > >> > > > > > >> > > > On Fri, May 30, 2025 at 9:52 AM Kaxil Naik <kaxiln...@gmail.com > > > > >> > wrote: > > >> > > > > > >> > > >> Agreed > > >> > > >> > > >> > > >> On Fri, 30 May 2025 at 15:18, Jarek Potiuk <ja...@potiuk.com> > > >> wrote: > > >> > > >> > > >> > > >> > Yeah, I think none of us want to just publish the code and be > > >> "done" > > >> > > >> with > > >> > > >> > it ... There is a real work to be done to make MCP much more > > >> useful > > >> > > and > > >> > > >> "AI > > >> > > >> > friendly" and the examples you gave Brian are cool, because I > > >> think > > >> > we > > >> > > >> need > > >> > > >> > to (and this is mostly the ask from maintainers for the users > > to > > >> > come > > >> > > >> and > > >> > > >> > participate in the design phase of what the MCP can do). > > >> > > >> > It looks like that "backbone" of the MCP and the glue between > > the > > >> > REST > > >> > > >> API > > >> > > >> > and MCP can be done in a simple (and easy to maintain) way. > But > > >> the > > >> > > >> added > > >> > > >> > value is indeed in figuring out what would be useful missing > > >> things > > >> > - > > >> > > >> > starting from what broad use cases we want to address - and > > >> whether > > >> > > >> some > > >> > > >> > help for the agent can be described as prompts, better > > >> description > > >> > of > > >> > > >> the > > >> > > >> > API and examples or maybe we need more aggregated, new APIs > > >> (maybe > > >> > > >> simply > > >> > > >> > new REST API calls we need) that will allow the agents to > > reason > > >> > > better > > >> > > >> and > > >> > > >> > faster. All of that is possible. > > >> > > >> > > > >> > > >> > J > > >> > > >> > > > >> > > >> > On Fri, May 30, 2025 at 11:29 AM Kaxil Naik < > > kaxiln...@gmail.com > > >> > > > >> > > >> wrote: > > >> > > >> > > > >> > > >> > > You can easily add as many tools you want: > > >> > > >> > > https://gofastmcp.com/servers/tools > > >> > > >> > > > > >> > > >> > > I would be surprised if there is a thing you can't do with > > >> FastMCP > > >> > > >> 2.0+ > > >> > > >> > > that you can do with the MCP Python SDK. > > >> > > >> > > > > >> > > >> > > Like I said earlier: This is a simplistic example :) but > the > > >> gist > > >> > is > > >> > > >> we > > >> > > >> > > should be using the newer abstractions which I am happy to > > >> comment > > >> > > >> during > > >> > > >> > > the development phase too. Like everything else we need to > > >> ensure > > >> > > >> > > maintainability is worth the value we create. > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > On Fri, 30 May 2025 at 14:48, Kaxil Naik < > > kaxiln...@gmail.com> > > >> > > wrote: > > >> > > >> > > > > >> > > >> > > > Btw we don’t need to use FastMCP just for create MCP from > > >> > OpenApi > > >> > > >> spec. > > >> > > >> > > > Many of you mighht already be aware - FastMCP 1.0 was > > >> adopted in > > >> > > the > > >> > > >> > > > official mcp python sdk since 1.2 and is recommended > > >> high-level > > >> > > >> server > > >> > > >> > > > framework. > > >> > > >> > > > > > >> > > >> > > > Check: > > >> > > >> > > > > > >> > > >> > > >> > > > https://github.com/modelcontextprotocol/python-sdk/releases/tag/v1.2.0 > > >> > > >> > > > > > >> > > >> > > > @Bryan Coder: I will be surprised if you can’t do the > > >> use-case > > >> > you > > >> > > >> > > > mentioned with FastMCP - either the one donated to MCP > > Python > > >> > SDK > > >> > > or > > >> > > >> > > > FastMCP 2.0 - have you tried that? It isn’t just a > wrapper! > > >> > > >> > > > > > >> > > >> > > > On Fri, 30 May 2025 at 13:16, Avi > > <a...@astronomer.io.invalid > > >> > > > >> > > >> wrote: > > >> > > >> > > > > > >> > > >> > > >> Yeah FastMCP is nice, I didn't select fast mcp for this > > >> > specific > > >> > > >> > reason: > > >> > > >> > > >> - The sheer number of tools that are created using > OpenAPI > > >> spec > > >> > > >> > doesn't > > >> > > >> > > >> need to be passed to AI every single message. > > >> > > >> > > >> - Instead, we can do a hierarchical tool discovery based > > on > > >> > > >> > categories. > > >> > > >> > > >> And > > >> > > >> > > >> let AI select a particular category and then get tools > > only > > >> for > > >> > > >> that > > >> > > >> > > >> category. > > >> > > >> > > >> > > >> > > >> > > >> python3 -c " > > >> > > >> > > >> > import json > > >> > > >> > > >> > with open('path/to/openapi.json') as f: > > >> > > >> > > >> > spec = json.load(f) > > >> > > >> > > >> > > > >> > > >> > > >> > tags = {} > > >> > > >> > > >> > for path, methods in spec['paths'].items(): > > >> > > >> > > >> > for method, details in methods.items(): > > >> > > >> > > >> > if 'tags' in details: > > >> > > >> > > >> > for tag in details['tags']: > > >> > > >> > > >> > tags[tag] = tags.get(tag, 0) + 1 > > >> > > >> > > >> > > > >> > > >> > > >> > print('Tags and their counts:') > > >> > > >> > > >> > for tag, count in sorted(tags.items(), key=lambda x: > > x[1], > > >> > > >> > > >> reverse=True): > > >> > > >> > > >> > print(f'{tag}: {count}') > > >> > > >> > > >> > " > > >> > > >> > > >> Tags and their counts: > > >> > > >> > > >> Task Instance: 19 > > >> > > >> > > >> Asset: 13 > > >> > > >> > > >> Connection: 8 > > >> > > >> > > >> DagRun: 8 > > >> > > >> > > >> Backfill: 7 > > >> > > >> > > >> DAG: 7 > > >> > > >> > > >> Pool: 6 > > >> > > >> > > >> Variable: 6 > > >> > > >> > > >> XCom: 4 > > >> > > >> > > >> Config: 2 > > >> > > >> > > >> Event Log: 2 > > >> > > >> > > >> Import Error: 2 > > >> > > >> > > >> Plugin: 2 > > >> > > >> > > >> Task: 2 > > >> > > >> > > >> DagVersion: 2 > > >> > > >> > > >> Login: 2 > > >> > > >> > > >> DagSource: 1 > > >> > > >> > > >> DagStats: 1 > > >> > > >> > > >> DagReport: 1 > > >> > > >> > > >> DagWarning: 1 > > >> > > >> > > >> Extra Links: 1 > > >> > > >> > > >> Job: 1 > > >> > > >> > > >> Provider: 1 > > >> > > >> > > >> DAG Parsing: 1 > > >> > > >> > > >> Monitor: 1 > > >> > > >> > > >> Version: 1 > > >> > > >> > > >> > > >> > > >> > > >> My last attempt to do a hierarchical discovery with > > FastMCP > > >> > > didn't > > >> > > >> go > > >> > > >> > as > > >> > > >> > > >> expected. > > >> > > >> > > >> But this could be short term. There is something cooking > > in > > >> the > > >> > > >> model > > >> > > >> > > >> context protocol repo for search of a tool. Ref: > > >> > > >> > > >> > > >> > > >> > > >> https://github.com/modelcontextprotocol/modelcontextprotocol/pull/322 > > >> > > >> > > >> > > >> > > >> > > >> I'll give this a try with FastMCP to see if I can get > the > > >> > > >> > > >> hierarchical discovery working. > > >> > > >> > > >> > > >> > > >> > > >> - Avi > > >> > > >> > > >> > > >> > > >> > > >> On Fri, May 30, 2025 at 1:33 AM Bryan Corder < > > >> > > >> bryancor...@gmail.com> > > >> > > >> > > >> wrote: > > >> > > >> > > >> > > >> > > >> > > >> > In order to bring value, we might want to think beyond > > >> just > > >> > > >> wrapping > > >> > > >> > > the > > >> > > >> > > >> > API. As Kaxil just showed, it's easy to create > something > > >> with > > >> > > 10 > > >> > > >> > lines > > >> > > >> > > >> of > > >> > > >> > > >> > code and FastMCP. > > >> > > >> > > >> > > > >> > > >> > > >> > However, the Airflow API was made for Airflow > operators' > > >> > > >> > consumption, > > >> > > >> > > >> not > > >> > > >> > > >> > necessarily for LLM consumption. When you have an > > endpoint > > >> > > called > > >> > > >> > > >> "Delete > > >> > > >> > > >> > DAG" with a description "Delete a specific DAG" that's > > >> very > > >> > > easy > > >> > > >> for > > >> > > >> > > any > > >> > > >> > > >> > user who has already navigated to the Airflow API spec > > to > > >> > > >> > understand, > > >> > > >> > > >> but > > >> > > >> > > >> > maybe not the best tool description for an LLM. I > think > > >> we'd > > >> > > >> want to > > >> > > >> > > >> either > > >> > > >> > > >> > exclude that or add additional context for the LLM to > > know > > >> > it's > > >> > > >> > > >> > destructive. > > >> > > >> > > >> > > > >> > > >> > > >> > In addition, LLMs can struggle with tool selection > when > > >> you > > >> > > give > > >> > > >> it > > >> > > >> > 80 > > >> > > >> > > >> > tools to work with. Things in the middle sometimes get > > >> lost > > >> > in > > >> > > >> the > > >> > > >> > > >> context. > > >> > > >> > > >> > There are ways to customize the FastMCP ( > > >> > > >> > > >> > > https://gofastmcp.com/servers/openapi#custom-route-maps > > ) > > >> to > > >> > > cut > > >> > > >> > down > > >> > > >> > > >> the > > >> > > >> > > >> > list of options, should you choose. > > >> > > >> > > >> > > > >> > > >> > > >> > However, it may be better to create something more > > >> tailored > > >> > to > > >> > > >> LLMs. > > >> > > >> > > >> > Thinking about the use case of getting LLM assistance > > with > > >> > > >> > debugging a > > >> > > >> > > >> > failed run, one of the things my teams do is put the > > "run > > >> > book" > > >> > > >> for > > >> > > >> > > prod > > >> > > >> > > >> > support in the doc_md notes right with the DAG, so if > a > > >> file > > >> > > >> never > > >> > > >> > > >> shows up > > >> > > >> > > >> > they know exactly what to do in that situation > > >> (potentially, > > >> > do > > >> > > >> > > >> nothing). > > >> > > >> > > >> > We also include other information like, "xx task can > be > > >> > flaky. > > >> > > If > > >> > > >> > you > > >> > > >> > > >> get > > >> > > >> > > >> > this error, rerunning it will usually resolve it." The > > >> goal > > >> > is > > >> > > >> for > > >> > > >> > any > > >> > > >> > > >> > engineer armed with the stack trace and the run book > to > > be > > >> > able > > >> > > >> to > > >> > > >> > > solve > > >> > > >> > > >> > any error. My team has all that information right in > the > > >> UI. > > >> > To > > >> > > >> get > > >> > > >> > > that > > >> > > >> > > >> > information, the LLM would need to know to hit the DAG > > >> > Details > > >> > > >> > > endpoint > > >> > > >> > > >> for > > >> > > >> > > >> > one minor attribute amongst several for the doc_md and > > get > > >> > the > > >> > > >> > correct > > >> > > >> > > >> dag > > >> > > >> > > >> > id, run id, task id and try number to grab the stack > > trace > > >> > from > > >> > > >> the > > >> > > >> > > >> failed > > >> > > >> > > >> > run. It would then need to go elsewhere to find the > DAG > > >> code > > >> > to > > >> > > >> > > debug. I > > >> > > >> > > >> > think it would be better to just create a > > >> "debug_failed_task" > > >> > > >> tool > > >> > > >> > an > > >> > > >> > > >> LLM > > >> > > >> > > >> > could call from an MCP server that would string those > > >> calls > > >> > > >> together > > >> > > >> > > and > > >> > > >> > > >> > serve them up to the LLM on a silver platter. The LLM > > >> could > > >> > > focus > > >> > > >> > all > > >> > > >> > > >> its > > >> > > >> > > >> > "reasoning" efforts on solving the problem instead of > > >> > figuring > > >> > > >> out > > >> > > >> > how > > >> > > >> > > >> to > > >> > > >> > > >> > get the information it needs to even begin. > > >> > > >> > > >> > > > >> > > >> > > >> > Again, if we just want to wrap the API in FastMCP, we > > can > > >> > share > > >> > > >> > > Kaxil's > > >> > > >> > > >> 10 > > >> > > >> > > >> > lines of code in a Medium article and be done. I think > > the > > >> > real > > >> > > >> > value > > >> > > >> > > >> is in > > >> > > >> > > >> > providing an implementation of a limited set of more > > >> complex > > >> > > base > > >> > > >> > > tools > > >> > > >> > > >> > like debug_failed_task (described above), > > >> > pause_all_active_DAGs > > >> > > >> > > (because > > >> > > >> > > >> > I'm about to upgrade!), describe_DAG (grabs only the > > >> > > description, > > >> > > >> > > >> > dependencies, converts cron schedule to human readable > > if > > >> > > >> > applicable, > > >> > > >> > > >> etc) > > >> > > >> > > >> > and giving people a way to extend the server. > > >> > > >> > > >> > > > >> > > >> > > >> > The above is tool focused. As Avi pointed out, there > are > > >> also > > >> > > >> > > resources > > >> > > >> > > >> and > > >> > > >> > > >> > prompts, but I've only personally worked with tools > and > > >> have > > >> > > >> nothing > > >> > > >> > > to > > >> > > >> > > >> add > > >> > > >> > > >> > there. > > >> > > >> > > >> > > > >> > > >> > > >> > With all the LLM tools quickly advancing on the > > >> development > > >> > > side > > >> > > >> > (e.g. > > >> > > >> > > >> code > > >> > > >> > > >> > generation/review), it's great to see the community > > >> working > > >> > on > > >> > > >> > > building > > >> > > >> > > >> > tools to help with the operational side. > > >> > > >> > > >> > > > >> > > >> > > >> > Bryan > > >> > > >> > > >> > > > >> > > >> > > >> > > > >> > > >> > > >> > On Thu, May 29, 2025, 4:50 PM Kaxil Naik < > > >> > kaxiln...@gmail.com> > > >> > > >> > wrote: > > >> > > >> > > >> > > > >> > > >> > > >> > > One more comment: MCP SDKs have advanced quite a bit > > >> and I > > >> > > was > > >> > > >> > able > > >> > > >> > > to > > >> > > >> > > >> > get > > >> > > >> > > >> > > an Airflow MCP Server working with just the > following > > >> code > > >> > > >> block. > > >> > > >> > I > > >> > > >> > > >> was > > >> > > >> > > >> > > successfully able to pause/unpause a DAG from Claude > > and > > >> > > other > > >> > > >> MCP > > >> > > >> > > >> client > > >> > > >> > > >> > > as an example. So as much as possible we should > > utilize > > >> > > higher > > >> > > >> > level > > >> > > >> > > >> > > abstraction like FastMCP which allows creating > client > > >> from > > >> > > >> OpenAPI > > >> > > >> > > >> spec > > >> > > >> > > >> > > < > > >> https://gofastmcp.com/servers/openapi#openapi-integration > > >> > >: > > >> > > >> > > >> > > > > >> > > >> > > >> > > import os > > >> > > >> > > >> > > > > >> > > >> > > >> > > import httpx > > >> > > >> > > >> > > from fastmcp import FastMCP > > >> > > >> > > >> > > > > >> > > >> > > >> > > token = os.environ.get("AF_ACCESS_TOKEN") > > >> > > >> > > >> > > client = httpx.AsyncClient( > > >> > > >> > > >> > > base_url="http://localhost:28080";, > > >> > > >> > > >> > > headers={"Authorization": f"Bearer > {token}"}, > > >> > > >> > > >> > > ) > > >> > > >> > > >> > > > > >> > > >> > > >> > > openapi_spec = httpx.get(" > > >> > > >> http://localhost:28080/openapi.json > > >> > > >> > > >> > ").json() > > >> > > >> > > >> > > > > >> > > >> > > >> > > mcp = FastMCP.from_openapi( > > >> > > >> > > >> > > openapi_spec=openapi_spec, > > >> > > >> > > >> > > client=client, > > >> > > >> > > >> > > name="Airflow 3.0 API Server" > > >> > > >> > > >> > > ) > > >> > > >> > > >> > > > > >> > > >> > > >> > > if __name__ == "__main__": > > >> > > >> > > >> > > mcp.run() > > >> > > >> > > >> > > > > >> > > >> > > >> > > > > >> > > >> > > >> > > > > >> > > >> > > >> > > On Thu, 29 May 2025 at 20:32, Avi > > >> > <a...@astronomer.io.invalid > > >> > > > > > >> > > >> > > wrote: > > >> > > >> > > >> > > > > >> > > >> > > >> > > > @Shahar -- Yes. Definitely. Feel free to reachout > if > > >> you > > >> > > need > > >> > > >> > > >> anything. > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > I totally agree, it to live as a separate repo. > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > - Avi > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > On Thu, May 29, 2025 at 12:50 PM Kaxil Naik < > > >> > > >> > kaxiln...@gmail.com> > > >> > > >> > > >> > wrote: > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > > @Shahar -- Absolutely, I think you are driving > it > > >> with > > >> > > this > > >> > > >> > > email. > > >> > > >> > > >> > So I > > >> > > >> > > >> > > > > think you can lead it from here and whoever > wants > > to > > >> > join > > >> > > >> can > > >> > > >> > > >> co-lead > > >> > > >> > > >> > > or > > >> > > >> > > >> > > > > join in development. > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > Please feel free to drive :) > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > On Thu, 29 May 2025 at 17:07, Aaron Dantley < > > >> > > >> > > >> aarondant...@gmail.com> > > >> > > >> > > >> > > > > wrote: > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > > Hey All! > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > I’d be grateful to be included in the AIP > > >> discussions > > >> > > to > > >> > > >> > help > > >> > > >> > > if > > >> > > >> > > >> > > > possible > > >> > > >> > > >> > > > > > too! Like Shahar, I’ve never worked on any of > > >> these > > >> > > >> items so > > >> > > >> > > >> it’d > > >> > > >> > > >> > be > > >> > > >> > > >> > > > > great > > >> > > >> > > >> > > > > > to see how work gets assigned and goes > through a > > >> > whole > > >> > > >> > > >> development > > >> > > >> > > >> > > > cycle! > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > Looking forward to it! > > >> > > >> > > >> > > > > > Aaron > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > On Thu, May 29, 2025 at 7:32 AM Shahar > Epstein < > > >> > > >> > > >> sha...@apache.org> > > >> > > >> > > >> > > > > wrote: > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > > If it's ok, I would like to lead the AIP > > effort > > >> (or > > >> > > at > > >> > > >> > least > > >> > > >> > > >> > > > co-lead), > > >> > > >> > > >> > > > > as > > >> > > >> > > >> > > > > > > I've never written an AIP before. I could > > start > > >> > > >> drafting > > >> > > >> > it > > >> > > >> > > >> > during > > >> > > >> > > >> > > > the > > >> > > >> > > >> > > > > > next > > >> > > >> > > >> > > > > > > week. > > >> > > >> > > >> > > > > > > Avi - please let me know if it works for > you. > > >> > > >> > > >> > > > > > > > > >> > > >> > > >> > > > > > > > > >> > > >> > > >> > > > > > > Shahar > > >> > > >> > > >> > > > > > > > > >> > > >> > > >> > > > > > > > > >> > > >> > > >> > > > > > > On Thu, May 29, 2025, 13:09 Kaxil Naik < > > >> > > >> > kaxiln...@gmail.com > > >> > > >> > > > > > >> > > >> > > >> > > wrote: > > >> > > >> > > >> > > > > > > > > >> > > >> > > >> > > > > > > > Yes separate repo, please and we would > need > > >> > someone > > >> > > >> to > > >> > > >> > > lead > > >> > > >> > > >> > this > > >> > > >> > > >> > > > > effort > > >> > > >> > > >> > > > > > > on > > >> > > >> > > >> > > > > > > > the proposal & development too. Avi - you > > are > > >> > > >> probably > > >> > > >> > > well > > >> > > >> > > >> > > > equipped > > >> > > >> > > >> > > > > to > > >> > > >> > > >> > > > > > > > lead it and I am sure more folks like > Aaraon > > >> > would > > >> > > be > > >> > > >> > > eager > > >> > > >> > > >> to > > >> > > >> > > >> > > work > > >> > > >> > > >> > > > > on > > >> > > >> > > >> > > > > > > its > > >> > > >> > > >> > > > > > > > development and on-going maintenance. > > >> > > >> > > >> > > > > > > > > > >> > > >> > > >> > > > > > > > Regards, > > >> > > >> > > >> > > > > > > > Kaxil > > >> > > >> > > >> > > > > > > > > > >> > > >> > > >> > > > > > > > On Thu, 29 May 2025 at 15:25, Jarek > Potiuk < > > >> > > >> > > >> ja...@potiuk.com> > > >> > > >> > > >> > > > wrote: > > >> > > >> > > >> > > > > > > > > > >> > > >> > > >> > > > > > > > > Yep. Having MCP is cool and drawing our > > >> > > >> implementation > > >> > > >> > > >> from > > >> > > >> > > >> > > > > > experiences > > >> > > >> > > >> > > > > > > > and > > >> > > >> > > >> > > > > > > > > usage of other MCP servers out there is > > even > > >> > > cooler > > >> > > >> > > >> > (especially > > >> > > >> > > >> > > > > that > > >> > > >> > > >> > > > > > we > > >> > > >> > > >> > > > > > > > can > > >> > > >> > > >> > > > > > > > > have some insights how people already > use > > >> them > > >> > > with > > >> > > >> > > >> Airflow) > > >> > > >> > > >> > - > > >> > > >> > > >> > > if > > >> > > >> > > >> > > > > we > > >> > > >> > > >> > > > > > > can > > >> > > >> > > >> > > > > > > > > bring together a few of those, put some > > >> nice, > > >> > > >> relevant > > >> > > >> > > >> > Airflow > > >> > > >> > > >> > > > > > prompts. > > >> > > >> > > >> > > > > > > > > Ideally we could have some examples of > how > > >> MCP > > >> > > can > > >> > > >> be > > >> > > >> > > used > > >> > > >> > > >> > > taken > > >> > > >> > > >> > > > > from > > >> > > >> > > >> > > > > > > > those > > >> > > >> > > >> > > > > > > > > who are using airflow (the debugging > > >> example by > > >> > > >> Avi is > > >> > > >> > > >> cool) > > >> > > >> > > >> > > > > > > > > > > >> > > >> > > >> > > > > > > > > I am not sure implementing it as > provider > > is > > >> > > really > > >> > > >> > "the > > >> > > >> > > >> way" > > >> > > >> > > >> > > > > though > > >> > > >> > > >> > > > > > - > > >> > > >> > > >> > > > > > > I > > >> > > >> > > >> > > > > > > > > would rather see `apache-airflow-mcp" > > >> separate > > >> > > >> repo - > > >> > > >> > > >> it's so > > >> > > >> > > >> > > > > > different > > >> > > >> > > >> > > > > > > > and > > >> > > >> > > >> > > > > > > > > distinct from airflow it does not really > > >> > require > > >> > > >> any > > >> > > >> > of > > >> > > >> > > >> > Airflow > > >> > > >> > > >> > > > > > > internals > > >> > > >> > > >> > > > > > > > > and code to be implemented - it makes > very > > >> > little > > >> > > >> > sense > > >> > > >> > > >> to be > > >> > > >> > > >> > > the > > >> > > >> > > >> > > > > > part > > >> > > >> > > >> > > > > > > of > > >> > > >> > > >> > > > > > > > > airflow "workspace" where we would > develop > > >> it > > >> > > >> together > > >> > > >> > > >> with > > >> > > >> > > >> > > > > airflow - > > >> > > >> > > >> > > > > > > > > because if it will talk over the REST > api, > > >> all > > >> > we > > >> > > >> need > > >> > > >> > > is > > >> > > >> > > >> the > > >> > > >> > > >> > > > > > `client` > > >> > > >> > > >> > > > > > > > that > > >> > > >> > > >> > > > > > > > > might be just a dependency. And there is > > >> even > > >> > no > > >> > > >> > reason > > >> > > >> > > >> for > > >> > > >> > > >> > MCP > > >> > > >> > > >> > > > and > > >> > > >> > > >> > > > > > > > airflow > > >> > > >> > > >> > > > > > > > > to be installed and developed together > > >> (that's > > >> > > the > > >> > > >> > main > > >> > > >> > > >> > reason > > >> > > >> > > >> > > > why > > >> > > >> > > >> > > > > we > > >> > > >> > > >> > > > > > > > want > > >> > > >> > > >> > > > > > > > > providers to be kept in monorepo. > > >> > > >> > > >> > > > > > > > > > > >> > > >> > > >> > > > > > > > > J. > > >> > > >> > > >> > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > >> > > >> > > >> > > > > > > > > On Thu, May 29, 2025 at 8:37 AM Amogh > > Desai > > >> < > > >> > > >> > > >> > > > > > amoghdesai....@gmail.com> > > >> > > >> > > >> > > > > > > > > wrote: > > >> > > >> > > >> > > > > > > > > > > >> > > >> > > >> > > > > > > > > > Seems like a promising area to invest > in > > >> > given > > >> > > >> the > > >> > > >> > > >> benefits > > >> > > >> > > >> > > it > > >> > > >> > > >> > > > > can > > >> > > >> > > >> > > > > > > > > provide > > >> > > >> > > >> > > > > > > > > > to > > >> > > >> > > >> > > > > > > > > > the users as mentioned by Shahar and > > >> > Abhishek. > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > Abhishek also has a promising talk > > >> submitted > > >> > > >> which i > > >> > > >> > > am > > >> > > >> > > >> > > looking > > >> > > >> > > >> > > > > > > forward > > >> > > >> > > >> > > > > > > > > to > > >> > > >> > > >> > > > > > > > > > this year at the summit. > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > In any case, this seems to be one of > the > > >> > first > > >> > > of > > >> > > >> > the > > >> > > >> > > >> very > > >> > > >> > > >> > > few > > >> > > >> > > >> > > > > > > > > > implementations of trying > > >> > > >> > > >> > > > > > > > > > to integrate Airflow officially / > > >> > unofficially > > >> > > >> with > > >> > > >> > an > > >> > > >> > > >> MCP > > >> > > >> > > >> > > > > server. > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > Thanks & Regards, > > >> > > >> > > >> > > > > > > > > > Amogh Desai > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > On Thu, May 29, 2025 at 2:56 AM Aaron > > >> > Dantley < > > >> > > >> > > >> > > > > > > aarondant...@gmail.com> > > >> > > >> > > >> > > > > > > > > > wrote: > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > Hey! > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > I also think this is a great idea! > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > Would it be possible to be included > in > > >> the > > >> > > >> > > development > > >> > > >> > > >> > > > process? > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > Sorry I’m new to this group, but > would > > >> > > >> appreciate > > >> > > >> > > any > > >> > > >> > > >> > > > > suggestions > > >> > > >> > > >> > > > > > > on > > >> > > >> > > >> > > > > > > > > how > > >> > > >> > > >> > > > > > > > > > to > > >> > > >> > > >> > > > > > > > > > > contribute to the MCP server > > >> development! > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > Regards! > > >> > > >> > > >> > > > > > > > > > > Aaron > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > On Wed, May 28, 2025 at 2:57 PM Avi > > >> > > >> > > >> > > > <a...@astronomer.io.invalid > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > > > wrote: > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > Nice to see the idea to > incorporate > > an > > >> > > >> official > > >> > > >> > > MCP > > >> > > >> > > >> > > server > > >> > > >> > > >> > > > > for > > >> > > >> > > >> > > > > > > > > > > > Airflow. It's been really magical > to > > >> see > > >> > > >> what a > > >> > > >> > > >> simple > > >> > > >> > > >> > > LLM > > >> > > >> > > >> > > > > can > > >> > > >> > > >> > > > > > do > > >> > > >> > > >> > > > > > > > > with > > >> > > >> > > >> > > > > > > > > > an > > >> > > >> > > >> > > > > > > > > > > > Airflow MCP server built just from > > >> APIs. > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > A few things that I noticed in my > > >> > > experience: > > >> > > >> > > >> > > > > > > > > > > > - The number of tools that the > > OpenAPI > > >> > spec > > >> > > >> > > >> generates > > >> > > >> > > >> > is > > >> > > >> > > >> > > > > quite > > >> > > >> > > >> > > > > > > > huge. > > >> > > >> > > >> > > > > > > > > > Most > > >> > > >> > > >> > > > > > > > > > > > tools (*Claude, VS Code with > GitHub > > >> > > Copilot, > > >> > > >> > > Cursor, > > >> > > >> > > >> > > > > Windsurf*) > > >> > > >> > > >> > > > > > > > which > > >> > > >> > > >> > > > > > > > > > > uses > > >> > > >> > > >> > > > > > > > > > > > mcp-client limits it to a number > of > > >> 100 > > >> > > >> tools. > > >> > > >> > > (*The > > >> > > >> > > >> > > > > read-only > > >> > > >> > > >> > > > > > > mode > > >> > > >> > > >> > > > > > > > > > > creates > > >> > > >> > > >> > > > > > > > > > > > less tools in comparison*.) > > >> > > >> > > >> > > > > > > > > > > > - MCP server are just not tools. > > There > > >> > are > > >> > > >> other > > >> > > >> > > >> things > > >> > > >> > > >> > > as > > >> > > >> > > >> > > > > > well, > > >> > > >> > > >> > > > > > > > like > > >> > > >> > > >> > > > > > > > > > > > resources and prompts. Prompts are > > >> super > > >> > > >> helpful > > >> > > >> > > in > > >> > > >> > > >> > case > > >> > > >> > > >> > > of > > >> > > >> > > >> > > > > > > > debugging > > >> > > >> > > >> > > > > > > > > > for > > >> > > >> > > >> > > > > > > > > > > > example. It is a way of teaching > LLM > > >> > about > > >> > > >> > > Airflow. > > >> > > >> > > >> > Say I > > >> > > >> > > >> > > > > want > > >> > > >> > > >> > > > > > to > > >> > > >> > > >> > > > > > > > > have > > >> > > >> > > >> > > > > > > > > > a > > >> > > >> > > >> > > > > > > > > > > > failing task investigated. A > prompt > > >> can > > >> > be > > >> > > >> > helpful > > >> > > >> > > >> in > > >> > > >> > > >> > > > letting > > >> > > >> > > >> > > > > > LLM > > >> > > >> > > >> > > > > > > > > know > > >> > > >> > > >> > > > > > > > > > a > > >> > > >> > > >> > > > > > > > > > > > step-by-step process of carrying > out > > >> the > > >> > > >> > > >> investigation. > > >> > > >> > > >> > > > > > > > > > > > - Where do you run the MCP > server? I > > >> > > wouldn't > > >> > > >> > want > > >> > > >> > > >> my > > >> > > >> > > >> > > > laptop > > >> > > >> > > >> > > > > to > > >> > > >> > > >> > > > > > > do > > >> > > >> > > >> > > > > > > > > the > > >> > > >> > > >> > > > > > > > > > > > heavy processing, which would want > > us > > >> to > > >> > go > > >> > > >> for > > >> > > >> > > the > > >> > > >> > > >> SSE > > >> > > >> > > >> > > > > instead > > >> > > >> > > >> > > > > > > of > > >> > > >> > > >> > > > > > > > > > stdio. > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > This is why I chose two different > > >> path of > > >> > > >> using > > >> > > >> > > mcp > > >> > > >> > > >> > > server > > >> > > >> > > >> > > > > with > > >> > > >> > > >> > > > > > > > > > airflow, > > >> > > >> > > >> > > > > > > > > > > > which I intend to talk about at > the > > >> > summit. > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > 1. AI-Augmented Airflow - This > > helped > > >> me > > >> > > add > > >> > > >> a > > >> > > >> > > chat > > >> > > >> > > >> > > > interface > > >> > > >> > > >> > > > > > > > inside > > >> > > >> > > >> > > > > > > > > > > > Airflow using a plugin to talk to > an > > >> > > Airflow > > >> > > >> > > >> instance > > >> > > >> > > >> > > (read > > >> > > >> > > >> > > > > > only > > >> > > >> > > >> > > > > > > > > mode). > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > 2. Airflow-Powered AI - > > Experimenting > > >> > with > > >> > > >> this > > >> > > >> > > has > > >> > > >> > > >> > been > > >> > > >> > > >> > > > > > totally > > >> > > >> > > >> > > > > > > > > > magical, > > >> > > >> > > >> > > > > > > > > > > > how powerful AI can become when it > > has > > >> > > >> access to > > >> > > >> > > >> > airflow. > > >> > > >> > > >> > > > > > Also, a > > >> > > >> > > >> > > > > > > > > > > directory > > >> > > >> > > >> > > > > > > > > > > > structure to maintain the DAGs, > and > > it > > >> > can > > >> > > >> write > > >> > > >> > > >> DAGs > > >> > > >> > > >> > on > > >> > > >> > > >> > > > the > > >> > > >> > > >> > > > > > > fly. I > > >> > > >> > > >> > > > > > > > > > > totally > > >> > > >> > > >> > > > > > > > > > > > see a need where LLMs eventually > > will > > >> > need > > >> > > a > > >> > > >> > > >> scheduler, > > >> > > >> > > >> > > > > > although > > >> > > >> > > >> > > > > > > a > > >> > > >> > > >> > > > > > > > > > > complete > > >> > > >> > > >> > > > > > > > > > > > airflow just for an LLM might > seem a > > >> bit > > >> > > >> > overkill > > >> > > >> > > to > > >> > > >> > > >> > the > > >> > > >> > > >> > > > rest > > >> > > >> > > >> > > > > > of > > >> > > >> > > >> > > > > > > > the > > >> > > >> > > >> > > > > > > > > > > > community. > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > I chose to build this on top of > open > > >> API > > >> > is > > >> > > >> > > because > > >> > > >> > > >> > that > > >> > > >> > > >> > > > was > > >> > > >> > > >> > > > > > the > > >> > > >> > > >> > > > > > > > only > > >> > > >> > > >> > > > > > > > > > way > > >> > > >> > > >> > > > > > > > > > > > to get proper RBAC enabled. > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > I have so many points to discuss. > > >> Would > > >> > > love > > >> > > >> to > > >> > > >> > > hear > > >> > > >> > > >> > from > > >> > > >> > > >> > > > the > > >> > > >> > > >> > > > > > > > > community > > >> > > >> > > >> > > > > > > > > > > and > > >> > > >> > > >> > > > > > > > > > > > then take it forward. > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > Thanks, > > >> > > >> > > >> > > > > > > > > > > > Avi > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > On Wed, May 28, 2025 at 6:32 PM > > Aritra > > >> > > Basu < > > >> > > >> > > >> > > > > > > > > aritrabasu1...@gmail.com> > > >> > > >> > > >> > > > > > > > > > > > wrote: > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > I definitely think there's > > >> potential to > > >> > > >> > interact > > >> > > >> > > >> with > > >> > > >> > > >> > > an > > >> > > >> > > >> > > > > > > airflow > > >> > > >> > > >> > > > > > > > > MCP > > >> > > >> > > >> > > > > > > > > > > > > server. Though I think I'd be > > >> > interested > > >> > > to > > >> > > >> > see > > >> > > >> > > >> how > > >> > > >> > > >> > > many > > >> > > >> > > >> > > > > and > > >> > > >> > > >> > > > > > > how > > >> > > >> > > >> > > > > > > > > > > > frequently > > >> > > >> > > >> > > > > > > > > > > > > people are making use of MCP > > >> servers in > > >> > > the > > >> > > >> > wild > > >> > > >> > > >> > before > > >> > > >> > > >> > > > > > > investing > > >> > > >> > > >> > > > > > > > > > > effort > > >> > > >> > > >> > > > > > > > > > > > in > > >> > > >> > > >> > > > > > > > > > > > > building and maintaining one for > > >> > airflow. > > >> > > >> I'm > > >> > > >> > > sure > > >> > > >> > > >> > the > > >> > > >> > > >> > > > data > > >> > > >> > > >> > > > > > is > > >> > > >> > > >> > > > > > > > > > > available > > >> > > >> > > >> > > > > > > > > > > > > out there, just needs finding. > > >> > > >> > > >> > > > > > > > > > > > > -- > > >> > > >> > > >> > > > > > > > > > > > > Regards, > > >> > > >> > > >> > > > > > > > > > > > > Aritra Basu > > >> > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > On Wed, 28 May 2025, 11:18 pm > > Julian > > >> > > >> LaNeve, > > >> > > >> > > >> > > > > > > > > > > > <jul...@astronomer.io.invalid > > >> > > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > wrote: > > >> > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > I think this would be > > interesting > > >> now > > >> > > >> that > > >> > > >> > the > > >> > > >> > > >> > > > Streamable > > >> > > >> > > >> > > > > > > HTTP > > >> > > >> > > >> > > > > > > > > > spec < > > >> > > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > >> > > >> > > >> > > > > > > > > > >> > > >> > > >> > > > > > > > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > >> > > > >> > > > https://modelcontextprotocol.io/specification/2025-03-26/basic/transports > > >> > > >> > > >> > > > >> > > >> > > >> > > > > > > > > > > > > > is out. I think in theory we > > could > > >> > > >> publish > > >> > > >> > > this > > >> > > >> > > >> > first > > >> > > >> > > >> > > > as > > >> > > >> > > >> > > > > an > > >> > > >> > > >> > > > > > > > > Airflow > > >> > > >> > > >> > > > > > > > > > > > > > provider that installs a > plugin > > to > > >> > > >> expose an > > >> > > >> > > MCP > > >> > > >> > > >> > > > > endpoint, > > >> > > >> > > >> > > > > > > as a > > >> > > >> > > >> > > > > > > > > > PoC - > > >> > > >> > > >> > > > > > > > > > > > > this > > >> > > >> > > >> > > > > > > > > > > > > > becomes a much nicer > experience > > >> than > > >> > a > > >> > > >> local > > >> > > >> > > >> stdio > > >> > > >> > > >> > > one. > > >> > > >> > > >> > > > > > > > > > > > > > -- > > >> > > >> > > >> > > > > > > > > > > > > > Julian LaNeve > > >> > > >> > > >> > > > > > > > > > > > > > CTO > > >> > > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > Email: jul...@astronomer.io > > >> > > >> > > >> > > > > > > > > > > > > > <mailto:jul...@astronomer.io > > >> > >Mobile: > > >> > > >> 330 > > >> > > >> > 509 > > >> > > >> > > >> 5792 > > >> > > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > On May 28, 2025, at 1:25 PM, > > >> Shahar > > >> > > >> > Epstein > > >> > > >> > > < > > >> > > >> > > >> > > > > > > > sha...@apache.org > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > wrote: > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > Dear community, > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > Following the thread on > Slack > > >> [1], > > >> > > >> > initiated > > >> > > >> > > >> by > > >> > > >> > > >> > > Jason > > >> > > >> > > >> > > > > > > > Sebastian > > >> > > >> > > >> > > > > > > > > > > > Kusuma, > > >> > > >> > > >> > > > > > > > > > > > > > I'd > > >> > > >> > > >> > > > > > > > > > > > > > > like to start an effort to > > >> > officially > > >> > > >> > > support > > >> > > >> > > >> MCP > > >> > > >> > > >> > > in > > >> > > >> > > >> > > > > > > > Airflow's > > >> > > >> > > >> > > > > > > > > > > > > codebase. > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > *Some background * > > >> > > >> > > >> > > > > > > > > > > > > > > Model Context Protocol (MCP) > > is > > >> an > > >> > > open > > >> > > >> > > >> standard, > > >> > > >> > > >> > > > > > > open-source > > >> > > >> > > >> > > > > > > > > > > > framework > > >> > > >> > > >> > > > > > > > > > > > > > > that standardizes the way AI > > >> models > > >> > > >> like > > >> > > >> > LLM > > >> > > >> > > >> > > > integrate > > >> > > >> > > >> > > > > > and > > >> > > >> > > >> > > > > > > > > share > > >> > > >> > > >> > > > > > > > > > > data > > >> > > >> > > >> > > > > > > > > > > > > > with > > >> > > >> > > >> > > > > > > > > > > > > > > external tools, systems and > > data > > >> > > >> sources. > > >> > > >> > > >> Think > > >> > > >> > > >> > of > > >> > > >> > > >> > > it > > >> > > >> > > >> > > > > as > > >> > > >> > > >> > > > > > a > > >> > > >> > > >> > > > > > > > > "USB-C > > >> > > >> > > >> > > > > > > > > > > for > > >> > > >> > > >> > > > > > > > > > > > > > AI" - > > >> > > >> > > >> > > > > > > > > > > > > > > a universal connector that > > >> > simplifies > > >> > > >> and > > >> > > >> > > >> > > > standardizes > > >> > > >> > > >> > > > > AI > > >> > > >> > > >> > > > > > > > > > > > > integrations. A > > >> > > >> > > >> > > > > > > > > > > > > > > notable example of an MCP > > >> server is > > >> > > >> > GitHub's > > >> > > >> > > >> > > official > > >> > > >> > > >> > > > > > > > > > > implementation > > >> > > >> > > >> > > > > > > > > > > > > > [3], which > > >> > > >> > > >> > > > > > > > > > > > > > > allows LLMs such as Claude, > > >> > Copilot, > > >> > > >> and > > >> > > >> > > >> OpenAI > > >> > > >> > > >> > > (or: > > >> > > >> > > >> > > > > "MCP > > >> > > >> > > >> > > > > > > > > > clients") > > >> > > >> > > >> > > > > > > > > > > > to > > >> > > >> > > >> > > > > > > > > > > > > > > fetch pull request details, > > >> analyze > > >> > > >> code > > >> > > >> > > >> changes, > > >> > > >> > > >> > > and > > >> > > >> > > >> > > > > > > > generate > > >> > > >> > > >> > > > > > > > > > > review > > >> > > >> > > >> > > > > > > > > > > > > > > summaries. > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > *How could an MCP server be > > >> useful > > >> > in > > >> > > >> > > >> Airflow?* > > >> > > >> > > >> > > > > > > > > > > > > > > Imagine the possibilities > when > > >> LLMs > > >> > > can > > >> > > >> > > >> > seamlessly > > >> > > >> > > >> > > > > > interact > > >> > > >> > > >> > > > > > > > > with > > >> > > >> > > >> > > > > > > > > > > > > > Airflow’s > > >> > > >> > > >> > > > > > > > > > > > > > > API: triggering DAGs using > > >> natural > > >> > > >> > language, > > >> > > >> > > >> > > > retrieving > > >> > > >> > > >> > > > > > DAG > > >> > > >> > > >> > > > > > > > run > > >> > > >> > > >> > > > > > > > > > > > > history, > > >> > > >> > > >> > > > > > > > > > > > > > > enabling smart debugging, > and > > >> more. > > >> > > >> This > > >> > > >> > > kind > > >> > > >> > > >> of > > >> > > >> > > >> > > > > > > integration > > >> > > >> > > >> > > > > > > > > > opens > > >> > > >> > > >> > > > > > > > > > > > the > > >> > > >> > > >> > > > > > > > > > > > > > door > > >> > > >> > > >> > > > > > > > > > > > > > > to a more intuitive, > > >> conversational > > >> > > >> > > interface > > >> > > >> > > >> for > > >> > > >> > > >> > > > > > workflow > > >> > > >> > > >> > > > > > > > > > > > > orchestration. > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > *Why do we need to support > it > > >> > > >> officially?* > > >> > > >> > > >> > > > > > > > > > > > > > > Quid pro quo - LLMs become > an > > >> > > integral > > >> > > >> > part > > >> > > >> > > of > > >> > > >> > > >> > the > > >> > > >> > > >> > > > > modern > > >> > > >> > > >> > > > > > > > > > > development > > >> > > >> > > >> > > > > > > > > > > > > > > experience, while Airflow > > >> evolves > > >> > > into > > >> > > >> the > > >> > > >> > > >> go-to > > >> > > >> > > >> > > for > > >> > > >> > > >> > > > > > > > > > orchestrating > > >> > > >> > > >> > > > > > > > > > > AI > > >> > > >> > > >> > > > > > > > > > > > > > > workflows. By officially > > >> supporting > > >> > > it, > > >> > > >> > > we’ll > > >> > > >> > > >> > > enable > > >> > > >> > > >> > > > > > > multiple > > >> > > >> > > >> > > > > > > > > > users > > >> > > >> > > >> > > > > > > > > > > > to > > >> > > >> > > >> > > > > > > > > > > > > > > interact with Airflow > through > > >> their > > >> > > >> LLMs, > > >> > > >> > > >> > > > streamlining > > >> > > >> > > >> > > > > > > > > automation > > >> > > >> > > >> > > > > > > > > > > and > > >> > > >> > > >> > > > > > > > > > > > > > > improving accessibility > across > > >> > > diverse > > >> > > >> > > >> workflows. > > >> > > >> > > >> > > All > > >> > > >> > > >> > > > > of > > >> > > >> > > >> > > > > > > that > > >> > > >> > > >> > > > > > > > > is > > >> > > >> > > >> > > > > > > > > > > > viable > > >> > > >> > > >> > > > > > > > > > > > > > > with relatively small > > >> development > > >> > > >> effort > > >> > > >> > > (see > > >> > > >> > > >> > next > > >> > > >> > > >> > > > > > > > paragraph). > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > *How should it be > > implemented?* > > >> > > >> > > >> > > > > > > > > > > > > > > As of today, there have been > > >> > several > > >> > > >> > > >> > > implementations > > >> > > >> > > >> > > > of > > >> > > >> > > >> > > > > > MCP > > >> > > >> > > >> > > > > > > > > > servers > > >> > > >> > > >> > > > > > > > > > > > for > > >> > > >> > > >> > > > > > > > > > > > > > > Airflow API, the most > visible > > >> one > > >> > [4] > > >> > > >> made > > >> > > >> > > by > > >> > > >> > > >> > > > Abhishek > > >> > > >> > > >> > > > > > > Bhakat > > >> > > >> > > >> > > > > > > > > > from > > >> > > >> > > >> > > > > > > > > > > > > > > Astronomer. > > >> > > >> > > >> > > > > > > > > > > > > > > The efforts of implementing > it > > >> and > > >> > > >> > > >> maintaining it > > >> > > >> > > >> > > in > > >> > > >> > > >> > > > > our > > >> > > >> > > >> > > > > > > > > codebase > > >> > > >> > > >> > > > > > > > > > > > > > shouldn't > > >> > > >> > > >> > > > > > > > > > > > > > > be too cumbersome (at least > in > > >> > > >> theory), as > > >> > > >> > > we > > >> > > >> > > >> > could > > >> > > >> > > >> > > > > > utilize > > >> > > >> > > >> > > > > > > > > > > packages > > >> > > >> > > >> > > > > > > > > > > > > like > > >> > > >> > > >> > > > > > > > > > > > > > > fastmcp to auto-generate the > > >> server > > >> > > >> using > > >> > > >> > > the > > >> > > >> > > >> > > > existing > > >> > > >> > > >> > > > > > > > OpenAPI > > >> > > >> > > >> > > > > > > > > > > specs. > > >> > > >> > > >> > > > > > > > > > > > > I'd > > >> > > >> > > >> > > > > > > > > > > > > > > be very happy if Abhishek > > could > > >> > share > > >> > > >> his > > >> > > >> > > >> > > experience > > >> > > >> > > >> > > > in > > >> > > >> > > >> > > > > > > this > > >> > > >> > > >> > > > > > > > > > > thread. > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > *Where else could we utilize > > >> MCP?* > > >> > > >> > > >> > > > > > > > > > > > > > > Beyond the scope of the > public > > >> > API, I > > >> > > >> > could > > >> > > >> > > >> also > > >> > > >> > > >> > > > > imagine > > >> > > >> > > >> > > > > > > > using > > >> > > >> > > >> > > > > > > > > it > > >> > > >> > > >> > > > > > > > > > > to > > >> > > >> > > >> > > > > > > > > > > > > > > communicate with Breeze. > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > *How do we proceed from > here?* > > >> > > >> > > >> > > > > > > > > > > > > > > Feel free to share your > > thoughts > > >> > here > > >> > > >> in > > >> > > >> > > this > > >> > > >> > > >> > > > > discussion. > > >> > > >> > > >> > > > > > > > > > > > > > > If there are no objections, > > >> I'll be > > >> > > >> happy > > >> > > >> > to > > >> > > >> > > >> > start > > >> > > >> > > >> > > > > > working > > >> > > >> > > >> > > > > > > on > > >> > > >> > > >> > > > > > > > > an > > >> > > >> > > >> > > > > > > > > > > AIP. > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > Sincerely, > > >> > > >> > > >> > > > > > > > > > > > > > > Shahar Epstein > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > *References:* > > >> > > >> > > >> > > > > > > > > > > > > > > [1] Slack discussion, > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > >> > > >> > > >> > > > > > > > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > >> > > >> > > >> > > >> > > >> > > > >> > > > > >> > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746121916951569 > > >> > > >> > > >> > > > > > > > > > > > > > > [2] Introducing the model > > >> context > > >> > > >> > protocol, > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > https://www.anthropic.com/news/model-context-protocol > > >> > > >> > > >> > > > > > > > > > > > > > > [3] GitHub Official MCP > > server, > > >> > > >> > > >> > > > > > > > > > > > > > > > >> > > >> https://github.com/github/github-mcp-server > > >> > > >> > > >> > > > > > > > > > > > > > > [4] Unofficial MCP Server > made > > >> by > > >> > > >> Abhishek > > >> > > >> > > >> Hakat, > > >> > > >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > >> > > > https://github.com/abhishekbhakat/airflow-mcp-server > > >> > > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > > >> > > >> > > >> > > > > > > > > > > >> > > >> > > >> > > > > > > > > > >> > > >> > > >> > > > > > > > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > > >> > > > > >> > > > >> > > > > > >
