Hi, I joined airflow community around two months ago and I would also like to share some thoughts, probably more from a user perspective. I agree that LLM can streamline DAG creation, especially for users/teams that are new to Airflow. However, in production system, static DAG files may be more favorable by devops or platform teams that manage the Airflow service.
As Jarek mentioned, there are security concerns for authoring DAGs through UI. I also received feedback from people to avoid using dynamic DAG generation. Basically, avoid delegating DAG creation to DAG parsing on the server end. To deal with those concerns, we have to build a framework, similar what Amogh mentioned, to render explicit DAG code in CI and deploy those as static Python files. I would also not recommend this practice, and I think providing proper prompts to the existing AI tools are also very helpful to write proper DAG. Thanks, Kevin Sent from Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Amogh Desai <amoghdesai....@gmail.com> Sent: Tuesday, July 8, 2025 2:04:33 AM To: dev@airflow.apache.org <dev@airflow.apache.org> Subject: Re: [DISCUSS] - LLM-Powered DAG Generation Strongly concur with everything Jarek had to tell. I would be in favour of a strong NO for this. The use case of having an embedded UI to type in natural language is as good as opening your IDE locally with integrated LLM models and asking it to generate the dag using natural language and drop it in the DAGs folder. If you have airflow hosted somewhere, you can consider using the git sync model to achieve similar results. There also have been a couple of drag and drop plugins built on top of Airflow to construct dags in a more naive manner, for example: https://github.com/Joffreybvn/airflow-dag-builder is one I remember from a couple of years ago. Soon, there also will be multi language support for Airflow tasks and that is also something you can consider using if Python has a learning curve :) Thanks & Regards, Amogh Desai On Tue, Jul 8, 2025 at 1:10 AM Jens Scheffler <j_scheff...@gmx.de.invalid> wrote: > Hi, > > also sharing the opinion that no dedicated UI is needed for this - but > would be very welcoming to share experience and maybe expereince and > starting promt to get going. So if you want to post (somewhere, e.g. > medium) an article about how this is possible, that migth be worthwile > to share. > > CFP for the next Airflow summit is over but there might be also other > oppuortunities to share and spread the use w/o needing to integrate into > Airflow codebase. Maybe even on Monthly townhall hosted by Astronomer. > (The taks from 2 years ago might be a bit outdated though they share the > same ground idea - but a lot has evolved since then...) > > Jens > > On 07.07.25 12:53, Kaxil Naik wrote: > > Yup agreed with Jarek. A strong no from my side. We don't want to allow > > authoring DAGs from Airflow UI especially just to provide an LLM > interface. > > > > > > > > On Mon, 7 Jul 2025 at 12:27, Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> Also you might take a look at Airflow Summit videos > >> > https://www.youtube.com/playlist?list=PLGudixcDaxY2NIjMYT8t5zA9KJ47wTCkM > >> -> > >> and look back to 2023. There were at least several talks about using > LLMs > >> to generate Airflow Dags, and our users are doing it already - and I > guess > >> it's quite natural for people to generate the Dags with the help of LLMs > >> already. > >> > >> On Mon, Jul 7, 2025 at 8:52 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >>> FYI I added your email directly - because apparently you are not > >>> subscribed to devlist - please do subscribe following the "community" > tab > >>> on our website. > >>> > >>> I don't want to cut down your wings and excitement, but this is a > >>> deliberate choice that Airflow UI does not allow to author DAGs. This > is > >> a > >>> security feature. And our security model > >>> > >> > https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html > >>> is very clear that "UI users" do not have (and should not have) > >>> capabilities of authoring DAGs (not as Python code - that allows > >>> arbitrary code execution). Maybe (and that is something we might > consider > >>> in the future) if there is a declarative way of creating DAGs which > does > >>> not allow to provide arbitrary code, we could allow that, but we have > not > >>> even settled on the idea of having a single declarative way of creating > >>> Dags. > >>> > >>> Also Airflow DAGS are just Python Code placed in a folder. And there is > >>> absolutely nothing stopping you to open your IDE with Claude , Cursor, > >>> Copilot, use the prompt of your choice and ... generate DAGs with LLM. > >>> There is absolutely no need to have a UI for that.- all the IDEs out > >> there > >>> already have a fantastic LLM integration, with capability of adding > >> prompt, > >>> using MCP servers (there are even several MCP servers for Airflow > created > >>> by the community and we are discussing about creating our own MCP > server > >>> https://lists.apache.org/thread/xgd66v6s7zf0xkvy3c7ysqvn4csgmw0 - > those > >>> IDEs have code completion, syntax check, allow you to interact with the > >>> Agents and approve/reject proposals when you are using agents to create > >>> your DAGs. They even allow you to use your own models that can be > >> RAG-ified > >>> based - for example - on the private DAGs your company might have. This > >> all > >>> works **today**. > >>> > >>> I don't think personally there is any benefit of creating a similar > >>> feature in Airflow UI. I can't see any to be honest. Maybe others have > a > >>> different opinion or maybe you can explain what benefits you see by > >> adding > >>> such a "UI feature" to Airflow itself (and also the problem about > >> security > >>> is extremely important and a huge blocker for the whole idea - until > this > >>> is somewhat addressed the whole idea is basically impossible to be > >> accepted > >>> by the community. > >>> > >>> J. > >>> > >>> > >>> On Mon, Jul 7, 2025 at 8:37 AM Harikrishnan Girikumar < > >>> harikrishnangiriku...@gmail.com> wrote: > >>> > >>>> Hello Team, > >>>> > >>>> My name is Harikrishnan(Hari), I have an idea/improvement proposal for > >>>> Airflow. > >>>> LLM-powered feature within Apache Airflow to significantly enhance > the > >>>> DAG > >>>> authoring experience. Users would be able to provide natural language > >>>> descriptions or queries and leverage Large Language Models (LLMs) to > >>>> automatically generate and modify Airflow DAGs. This aims to > democratize > >>>> DAG creation, reduce the learning curve for new users, and accelerate > >> the > >>>> development of complex workflows. For example: We can have a UI tab > in > >>>> Airflow where users can add their respective authentication > credentials > >>>> for > >>>> the LLMs they want to use (OpenAI, Claude or their personal model > >> serving > >>>> link etc.) they can select their AI from drop down and a chat window > to > >>>> input queries like: Create a DAG to copy my data from S3 to Postgres > and > >>>> the code generated would be copied to DAG folder. We can restrict the > >>>> Prompts to be strictly for DAG generation for initial trial, further > >> down > >>>> the line a RAG feature could be added where a Vectorized version of > >>>> Airflow > >>>> documentation is used to improve the accuracy of DAG creation. > >>>> > >>>> I am really excited about this feature, this would reduce the learning > >>>> curve and improve the interaction for new users. Let me know your > >>>> thoughts, > >>>> looking forward to hearing from the team. > >>>> > >>>> Regards, > >>>> Hari > >>>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
