Hi folks, I’m writing to share my thought regarding the possibility of supporting “custom TI dependencies”.
Currently we maintain the dependency check rules under “airflow.ti_deps.deps". They cover the dependency checks like if there are available pool slot/if the concurrency allows/TI trigger rules/if the state is valid, etc., and play essential role in the scheduling process. One idea was brought up in our team's internal discussion: why shouldn’t we support custom TI dependencies? In details: just like the cluster policies (dag_policy/task_policy/task_instance_mutation_hook/pod_mutation_hook), if we support users add their own dependency checks as custom classes (and also put under airflow_local_settings.py), it will allow users to have much higher flexibility in the TI scheduling. These custom TI deps should be added as additions to the existing default deps (not replacing or removing any of them). For example: similar to check for pool availability/concurrency, the job may need to check for user’s infra-specific conditions, like if a GPU is available right now (instead of competing with other jobs randomly), or if an external system API is ready to be called (otherwise wait a bit ). And a lot more other possibilities. Why cluster policies won’t help here? task_instance_mutation_hook is executed in a “worker”, not in the DAG file processor, just before the TI is executed. What we are trying to gain some control here, though, is in the scheduling process (based on custom rules, to decide if the TI state should be updated so it can be scheduled for execution). I would love to know how community finds this idea, before we start to implement anything. Any quesiton/suggestion would be greatly appreciated. Many thanks! XD
