Hello Team, My name is Harikrishnan(Hari), I have an idea/improvement proposal for Airflow. LLM-powered feature within Apache Airflow to significantly enhance the DAG authoring experience. Users would be able to provide natural language descriptions or queries and leverage Large Language Models (LLMs) to automatically generate and modify Airflow DAGs. This aims to democratize DAG creation, reduce the learning curve for new users, and accelerate the development of complex workflows. For example: We can have a UI tab in Airflow where users can add their respective authentication credentials for the LLMs they want to use (OpenAI, Claude or their personal model serving link etc.) they can select their AI from drop down and a chat window to input queries like: Create a DAG to copy my data from S3 to Postgres and the code generated would be copied to DAG folder. We can restrict the Prompts to be strictly for DAG generation for initial trial, further down the line a RAG feature could be added where a Vectorized version of Airflow documentation is used to improve the accuracy of DAG creation.
I am really excited about this feature, this would reduce the learning curve and improve the interaction for new users. Let me know your thoughts, looking forward to hearing from the team. Regards, Hari
