Hi Jack. I think both points are valid. Yes, a re-fire max_active_runs will add value in my opinion. We can't have multiple ones running. Furthermore, this is for outside airflow (I think), the logic (Task) running should also take race-condition into consideration to avoid collision and having a max_active_run will mitigate to a point.
I think the DB (Postgres) will not be a bottleneck. I have also made an initial PR for the suggested implementation, happy to hear your feedback. https://github.com/apache/airflow/pull/74583 On Wed, Oct 7, 2026 at 8:15 PM Griffith, Jack S via dev < [email protected]> wrote: > Hi Ramzi, all, > +1 on this feature. I am evaluating adopting airflow and intend to run > partitioned-asset Dags for use cases where restated data for > already-processed partitions is routine. > The proposed modes and opt-in seem right to me. I have no strong feelings > on the implicit default when a user opts-in. This is my first interaction > with the project but as there is no activity on this thread perhaps opening > a PR would get traction? This feature may be light enough to not require > one. > Some thoughts on implementation: > > 1. Concurrency. I had planned to propose this already but a > per-partition-key active-run limit, like the Dag-wide max_active_runs, > would be useful, e.g. with NewRun should a re-fire wait for an in-flight > run of the same partition to finish? > 2. Performance. Does the DB have the right indices to support the new > query patterns? > Thanks, > Jack > > On 2026/09/16 07:54:20 Ramzi Alashabi wrote: > > Bumping this. > > > > ________________________________ > > Your Personal Data: We may collect and process information about you that > may be subject to data protection laws. For more information about how we > use and disclose your personal data, how we protect your information, our > legal basis to use your information, your rights and who you can contact, > please refer to: www.gs.com/privacy-notices< > http://www.gs.com/privacy-notices> >
