Hello everyone Thanks for the feedback. I edited the AIP with some clarifications and given the positive momentum I will start a vote thread later this week. I'd like to highlight that performance concerns can be addressed with code review and sharing benchmarks if requested. That is not something we can really address on AIP level. The feature scope stays as initially planned at both task level and TI level.
On Sun, Nov 15, 2026 at 12:02 AM Sameer Mesiah <[email protected]> wrote: > Hi, > > So I have been through the AIP as well as this thread. Overall, I am in > favour of this AIP because: > > 1. There is value in being able to intervene without redesigning or > redeploying a DAG. Dedicated pools, asset dependencies and > human-in-the-loop tasks can address anticipated scenarios, but they require > the DAG to have been designed accordingly. An on-call engineer may need to > prevent one task from starting during an incident without stopping > unrelated work or waiting for a deployment. David raised a few alternatives > that effectively achieve what this task/ti hold feature does but I don’t > think any are as ergonomic. > > 2. I see Ash raised some objections on the basis that this may violate the > ‘pipeline as code’ philosophy. But I don’t think that conflict is inherent. > Don’t we already allow users to intervene through the UI and API, such as > pausing DAGs, clearing tasks and marking task states. A task-instance hold > is no different i.e. the code defines the workflow, while the hold > temporarily controls whether a particular task instance can start. > > I think the biggest sticking point (already raised by others) is the > potential performance implications. This of course depends on the final > agreed scope (task/tis or just tis) but it would be interesting to see what > the actual planned metrics for the benchmark would be. I have left a > comment on the confluence page. > > Thanks, > Sameer. > > On Thu, 5 Nov 2026 at 00:55, <[email protected]> wrote: > > > Hi all, > > > > Elad and I would like to start the discussion on AIP-128, which > > proposes introducing task holds in Airflow. > > > > Users sometimes need one task to wait while unrelated work > > continues—for example, while correcting bad data or waiting for a > > vendor API to recover. A hold lets them prevent that task from > > starting without pausing the whole Dag. > > > > AIP: > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold > > > > Demo (1.5 min): > > > > > https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2 > > > > The proposal supports two kinds of hold: > > > > - Task instance hold: one task in one run, for example publish_report > > in today's run while bad data is fixed. > > - Task hold: one task in every run, for example while a vendor API is > > unavailable. > > > > The scheduler checks holds before picking tasks to start. An attempt > > it has already picked may still run. A hold doesn't change the task's > > state; other state changes, such as upstream failure, happen as usual. > > Once all holds end, normal scheduling can continue. > > > > Every hold has a reason, an optional end time and an audit log entry. > > We have a working prototype, shown in the video. > > > > The AIP explains the key design decisions and their trade-offs. We'd > > particularly welcome feedback on whether a hold should only block a > > task's start, and whether the two proposed scopes are sufficient for a > > first version. > > > > Please comment on the AIP doc, or reply here. > > > > Thanks, > > Shivam > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
