Praveen, Great question! You cannot "add a new step" to a running workflow, but it is possible to dynamically decide which tasks to run based on the results of a previous (upstream) task.
In practice this means that you would want to maintain a set of tasks in your DAG that you can dynamically decide amongst based on the results of some "decide what to do" task. This can be done with a "BranchOperator". Airflow Concept doc: https://airflow.apache.org/docs/apache-airflow/1.10.6/concepts.html#branching Astronomer doc: https://www.astronomer.io/docs/learn/airflow-branch-operator Are you able to share more of your use case? If you do it will be easier for people to offer suggestions. Best, Kirk -----Original Message----- From: Tzu-ping Chung <t...@astronomer.io.INVALID> Sent: Wednesday, October 9, 2024 3:59 AM To: dev@airflow.apache.org Subject: [EXT] Re: Dynamic Workflows Queries I believe both are impossible. Airflow is designed to run statically defined workflows. The triggering of the workflow can be dynamic, and each step of the workflow can be dynamic, but the overall structure of the workflow must be static; you cannot add things or change their ordering on the fly. TP > On Oct 9, 2024, at 16:49, Praveen Sharma > <praveen.sha...@kickdrumtech.com.INVALID> wrote: > > Hello > > I just came across apache airflow recently and this looks like a very > promising tool. But I have a few questions which I'm not able to find > answers to at the moment. > > Can you guys please help me with this, Can we do these steps in apache > airflow. If yes please guide me to the correct documentation and process. > > 1. Add a New Step in a Running Workflow 2. Change Sequence of Steps > dynamically in a Running workflow > > Your quick help will be highly appreciated. As a new user choosing the > right path for the same is confusing. > > Thanks and Regards > Praveen Sharma --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org DISCLAIMER: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. BAM prohibits all personnel from having any business related communications over text message or other unapproved communication applications. Unless pre-approved, BAM employees are only permitted to communicate over email, Bloomberg and BAM telephone lines. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
