steve148 opened a new pull request, #43055: URL: https://github.com/apache/airflow/pull/43055
# Summary Adds a new sensor to the Google Cloud provider that waits on the status of a target in a workflow invocation. # Why might this useful? Right now I use Airflow to trigger a Dataform workflow on a schedule. I use the `async=True` argument on the `DataformCreateWorkflowInvocationOperator` and the `DataformWorkflowInvocationStateSensor` to wait until the workflow is complete before running subsequent steps. This approach is simple but comes with some tradeoffs due to its lack of granularity. 1. If any target in the workflow fails, the sensor will also fail. If I want to have subsequent tasks run for any target that does succeed, this approach will not work. 2. If I have multiple targets that my subsequent tasks depend on, there may be large gap between when Dataform completes the tasks. This approach has subsequent tasks run when the whole Dataform workflow is complete. The sensor added in this PR addresses the trade offs listed above by providing a more granular sensor. Instead of waiting for the workflow to complete, it waits for a target within the workflow to complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
