Hi everyone, I would like to restart the vote on AIP-104, now titled Iterable Tasks (IT) and Task Spreading (TS):
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS The previous vote, under the name "Task Iteration and Dynamic Task Batching", ran during the summer and did not get much engagement: https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j The AIP has changed since then, so please vote on the current page rather than on what you may remember: - Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is already widely used for Task Instance (as remarked by Daniel). - Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The primitive is now .spread(across=N) instead of .batch(size=N), as "size" read like a chunk length while it is the number of Task Instances. Items are distributed round robin over those N Task Instances. The new name was discussed and voted on in separate threads (linked below). - Iterable Tasks now checkpoint progress per item through AIP-103 Task State. On a retry, or after a worker crash, items that already finished are not run again; only the items that failed or were in flight are. - The page now documents task_concurrency, the setting that controls how many items run at once inside one Task Instance. - The page now lists the current limits: deferrable operators, reschedule-mode sensors and TriggerDagRunOperator are not supported inside .iterate() yet (deferrable support is planned in a separate PR); execution_timeout applies per item only to async operators; multiple_outputs is ignored; and pools count the Task Instance rather than its items. The previous discussion threads can be found here: https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc The naming of the spread primitive: https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2 (discussion) https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote) Thank you again to everyone who reviewed the AIP and took part in the discussions. A special thank you to Kaxil for the many detailed reviews, and to TP, Jens, Jarek and Ash for their comments. Thanks as well to Daniel, Jedidiah and Amogh for their more recent reviews on the AIP page, and to everyone who voted on the naming of the spread primitive. The vote will remain open for 5 days and will close on Wednesday, 7 October 2026, at 00:00 GMT. Everyone is encouraged to vote, although only votes from PMC members and committers are considered binding. Please vote accordingly: [ ] +1 Approve [ ] +0 no opinion [ ] -1 disapprove with the reason Please consider this my +1 (binding). Regards, David General (Internal Property)
