Thank you Shivam and Elad for the AIP and for building a working prototype.

I think I understand the operational need. Like a couple of others, I have
a problem with the hold on future runs, and I think the

problem is architectural. I am -1 on that part as proposed.


Firstly, holding a task across all future runs changes the shape of the
DAG. It could change which assets get created, what

downstream asset-triggered DAGs see, and what the alerts watching this DAG
mean. The AIP mentions neither assets nor DAG

versions. It does reference deadline alerts somewhat, but doesn't reference
any asset behavior or asset-based dependencies.

A change to what a DAG does should logically be a new version, which
tremendously simplifies the user experience while retaining

consistent behavior across capabilities.


Secondly, there is a race condition with in-flight runs. The AIP gives a
table for each task state, but the outcome depends on

timing. With many runs are simultaneously active for the same DAG, some
could be past the task, some are queued and so on.

This behavior is undefined. The gradual deployment example has the same
problem.


Thirdly, the main stated motivation is an unavailable external service,
meaning one task in one DAG cannot run successfully.

In a real deployment many tasks across many DAGs call the same service. An
"systems operator" would have to find every one

and place a hold on each. The AIP does not address this holistically. It
dismisses pools, but a shared pool at zero slots is the one

existing control that works across DAGs, and the AIP itself says a task
hold is "the same as setting a pool to 0 slots today."

This is puzzling.


Fourthly, we already have patterns for this, and they belong in the
operator. Durable execution on the task state store

(AIP-103) means a task doesn't redo completed work after an interruption.
Retry policies (AIP-105) let the operator

decide, per failure, whether to retry, wait, fail, or escalate for a human
decision.


ManagedAgentToolset already handles provider outages with failover inside
the operator. This is cleaner for the DAG

author and for the DAG operator. It is rational to say that these only help
DAGs written for them. That is true, and it is why the

operator should carry the behavior, and why the gradual deployment case
belongs with versioning. The AIP doesn't mention AIP-103 or AIP-105.



Fifthly, synchronous DAG execution. DAGs that are invoked through the REST
API and waited on become undefined.

Users who trigger a DAG and wait for the result, with the experimental wait
endpoint or their own polling, will be stuck.

The AIP never mentions API-triggered runs. With a task on hold the run
stays Running and the caller blocks until the hold is released or

dagrun_timeout expires, and that has no default. Result XComs never arrive,
and the caller cannot tell a held run from a

slow one, because the AIP says tools reading task states cannot tell a held
task from a waiting one. Anyone who can clear

a task can place a task hold, so one person handling an incident can stall
every synchronous caller of that DAG without

their knowing. What should the wait endpoint return for a run stuck on a
hold, and should triggering be refused or flagged?


Vikram

On Tue, Nov 17, 2026 at 6:53 PM Elad Kalif via dev <[email protected]>
wrote:

> Hello everyone
> Thanks for the feedback. I edited the AIP with some clarifications and
> given the positive momentum I will start a vote thread later this week.
> I'd like to highlight that performance concerns can be addressed with code
> review and sharing benchmarks if requested. That is not something we can
> really address on AIP level.
> The feature scope stays as initially planned at both task level and TI
> level.
>
> On Sun, Nov 15, 2026 at 12:02 AM Sameer Mesiah <[email protected]>
> wrote:
>
> > Hi,
> >
> > So I have been through the AIP as well as this thread. Overall, I am in
> > favour of this AIP because:
> >
> > 1. There is value in being able to intervene without redesigning or
> > redeploying a DAG. Dedicated pools, asset dependencies and
> > human-in-the-loop tasks can address anticipated scenarios, but they
> require
> > the DAG to have been designed accordingly. An on-call engineer may need
> to
> > prevent one task from starting during an incident without stopping
> > unrelated work or waiting for a deployment. David raised a few
> alternatives
> > that effectively achieve what this task/ti hold feature does but I don’t
> > think any are as ergonomic.
> >
> > 2. I see Ash raised some objections on the basis that this may violate
> the
> > ‘pipeline as code’ philosophy. But I don’t think that conflict is
> inherent.
> > Don’t we already allow users to intervene through the UI and API, such as
> > pausing DAGs, clearing tasks and marking task states. A task-instance
> hold
> > is no different i.e. the code defines the workflow, while the hold
> > temporarily controls whether a particular task instance can start.
> >
> > I think the biggest sticking point (already raised by others) is the
> > potential performance implications. This of course depends on the final
> > agreed scope (task/tis or just tis) but it would be interesting to see
> what
> > the actual planned metrics for the benchmark would be. I have left a
> > comment on the confluence page.
> >
> > Thanks,
> > Sameer.
> >
> > On Thu, 5 Nov 2026 at 00:55, <[email protected]> wrote:
> >
> > > Hi all,
> > >
> > > Elad and I would like to start the discussion on AIP-128, which
> > > proposes introducing task holds in Airflow.
> > >
> > > Users sometimes need one task to wait while unrelated work
> > > continues—for example, while correcting bad data or waiting for a
> > > vendor API to recover. A hold lets them prevent that task from
> > > starting without pausing the whole Dag.
> > >
> > > AIP:
> > >
> >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold
> > >
> > > Demo (1.5 min):
> > >
> > >
> >
> https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2
> > >
> > > The proposal supports two kinds of hold:
> > >
> > > - Task instance hold: one task in one run, for example publish_report
> > > in today's run while bad data is fixed.
> > > - Task hold: one task in every run, for example while a vendor API is
> > > unavailable.
> > >
> > > The scheduler checks holds before picking tasks to start. An attempt
> > > it has already picked may still run. A hold doesn't change the task's
> > > state; other state changes, such as upstream failure, happen as usual.
> > > Once all holds end, normal scheduling can continue.
> > >
> > > Every hold has a reason, an optional end time and an audit log entry.
> > > We have a working prototype, shown in the video.
> > >
> > > The AIP explains the key design decisions and their trade-offs. We'd
> > > particularly welcome feedback on whether a hold should only block a
> > > task's start, and whether the two proposed scopes are sufficient for a
> > > first version.
> > >
> > > Please comment on the AIP doc, or reply here.
> > >
> > > Thanks,
> > > Shivam
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [email protected]
> > > For additional commands, e-mail: [email protected]
> > >
> > >
> >
>

Reply via email to