dabla opened a new pull request, #62922:
URL: https://github.com/apache/airflow/pull/62922
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
*Description*
This PR is the initial implementation of Dynamic Task Iteration (DTI), as
discussed in the
[devlist](https://lists.apache.org/[email protected]) and
building upon the foundations of
[AIP-98](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-98%3A+Add+async+support+for+PythonOperator+in+Airflow+3).
For further context on the use cases and performance benefits of DTI, see
this [Medium
Article](https://medium.com/apache-airflow/scaling-with-airflow-3-2-when-to-defer-and-when-to-use-native-async-operators-78a1ba35237d).
The XCom Database Constraint Challenge
While porting our internal "monkey-patched" version of DTI (used since
Airflow 2.x) to the core, I've identified a significant technical hurdle
regarding XCom handling.
*The Issue*
Around Airflow 2.10/2.11, a change was introduced to the database
constraints for the XCom table. Specifically:
- Current State: The DB prevents creating indexed XComs (map_index >= 0)
unless a corresponding mapped TaskInstance exists in the task_instance table.
- The Conflict: DTI is designed to process multiple indexed XComs within a
single Task Instance. Because there is no 1-to-1 mapping of map_index to a
physical TI row, the DB constraint blocks the insertion of these results.
*Current Workaround in this PR*
To maintain functionality without immediate schema changes, I have
implemented a custom XComIterable. This appends the index directly to the XCom
key to bypass the constraint and manages the iteration logic internally.
Proposed Long-term Fix
I believe the cleanest path forward is to adjust the DB constraint to allow
indexed XComs even in the absence of an indexed TI. This would:
Simplify the DTI implementation.
Align the DB schema with the more flexible task patterns introduced in
Airflow 3.x.
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
