Hi Kevin, I think compared to the current "Pool Slots" this proposal is good but opens a very large door for complexity.
A good example would be similar like Kubernetes schedules jobs and how Memory, CPU, GPUs, Storage is reserved and committed making a job working. Whereas this is great the scheduling complexity to reserve and decide where to run a job will be really really worth an AIP to discuss a design and complexity. I would propose (w/o AIP) not to dramatically change this. In our current environment we push this "problem" down to use Kubernetes for this resource based scheduling (either PodOperator or KubernetesExecutor) where the POD template and scheduling of Kubernetes can be leveraged. All "simple" tasks where just a worker slot is needed are running on Celery and that is how we balance out. If an AIP is generated don't forget about the complexity of deferred operators running in Async where the list of active tasks need to be summed up for memory whereas the host of the triggerer is especially limited. Reservation must probably be different for sync vs. async tasks. Mit freundlichen Grüßen / Best regards Jens Scheffler Deterministik open Loop (XC-DX/ETV5) Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | http://www.bosch.com/ Tel. +49 711 811-91508 | Mobil +49 160 90417410 | jens.scheff...@de.bosch.com Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus Forschner, Stefan Grosch, Dr. Markus Heyn, Dr. Tanja Rückert -----Original Message----- From: kevin Klein <ke...@coinefits.com> Sent: Freitag, 15. September 2023 11:32 To: dev@airflow.apache.org Subject: [DISCUSS] Allow multiple pools for one task and require all corresponding pool slots to be available before a task can run Hi, I see there's a discussion with the same idea here already https://github.com/apache/airflow/issues/13975 But there's no AIP yet, I'm considering making a proposal so that people can seriously consider this make such an improvement. The idea is that one task may depend on multiple resources, and we may want to make sure all these different types of resources are available before the task can run. Each type of resource is limited and we may have a pool for this resource. Currently airflow only uses one pool for one task to decide if this task can run, then it won't cover the multi-resource case. For example, a task may depend on both resource A and B, where A can be the memory, and B can be the number of connections. We set up Pool_A for resource A and Pool_B for resource B. Pool_A can have 16 slots (e.g. 16GB memory), while Pool_B can have 10 slots (10 connections at most). We may have different types of tasks that use different amounts of memory(A) and connection(B) resources. For instance, a type of task T1 needs 2 slots from Pool_A, and 1 slot from Pool_B, while another type of task T2 needs 1 slot from Pool_A, and 2 slots from Pool_B. When we run a few T1 and T2 task instances together, we'll have a problem in finding out how many T1 and T2 tasks can run together within the resource limit. If we assume T1 only uses Pool_A, and T2 only uses Pool_B, then airflow can schedule 8 T1 tasks and 5 T2 tasks to run together, but then it can actually consume 8*2+5 = 21 amounts of resource A at run time. In this case, A is memory, and if the slot unit is GB in memory, it means now airflow allows the running tasks to occupy 21GB memory while there's only 16GB available. Then the tasks will fail at run time. We need to let airflow be aware that T1 and T2 need both resource A and B, and both resources' available slots have to be enough before one task can run. As suggested in the above github thread, one proposal is to allow the pool and pool slots to take list arguments, and the airflow scheduler needs to check that all the passed-in pools in the list have enough slots before it allows a task to run. Please let me know what you think or if there're any suggestions. Thank you very much! -- Best, Kevin --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
