shivaam commented on code in PR #62561:
URL: https://github.com/apache/airflow/pull/62561#discussion_r2899993549
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1981,6 +1981,9 @@ def _mark_backfills_complete(self, session: Session =
NEW_SESSION) -> None:
# todo: AIP-78 simplify this function to an update statement
query = select(Backfill).where(
Backfill.completed_at.is_(None),
+ # Guard: backfill must have at least one association,
+ # otherwise it is still being set up (see #61375).
+ exists(select(BackfillDagRun.id).where(BackfillDagRun.backfill_id
== Backfill.id)),
Review Comment:
Good point about the edge case. The guard approach is preferred as the
flush approach might not work and creates another race condition.
Before creating a backfill, we check if there are any active backfills for
the same DAG and throw an error. Currently, we immediately commit the Backfill
row, so a concurrent request sees it and raises AlreadyRunningBackfill. If we
batch everything into one transaction with flush(), the Backfill row stays
invisible to other sessions until all DagRuns are created and the final commit
happens. That opens a window of seconds where a concurrent request sees zero
active backfills and can create a duplicate backfill.
[[Check for existing
backfills](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/backfill.py#L583-L589)](https://github.com/apache/airflow/blob/bae2c27e3ebf8a4368bd64cb7456fc360a2dd2f9/airflow-core/src/airflow/models/backfill.py#L577-L591)
For the guard approach, I think we can handle the orphan two different ways:
- Add an age-based cleanup in _mark_backfills_complete — backfills with zero
BackfillDagRun rows older than 10 minutes get marked complete instead of being
stuck forever.
- In _create_backfill, if the DagRun creation fails after the Backfill row
is already committed, catch the exception and mark the backfill complete
immediately so it doesn't permanently block future backfills for that DAG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
