Well, initially I was also going to argue we should remove
ignore_first_depends_on_past and just make a decision for users.  I was
going to say, any param that complicated must be viewed as suspect.  But
then I noticed it was already sorta deprecated in 1.x.

>From here
<https://github.com/apache/airflow/blob/1b4c6d5fdeac0e700d4202e109c7cc849c0e95c7/docs/apache-airflow/howto/upgrading-from-1-10/index.rst?plain=1#L1085-L1089>
:

**Deprecating ignore_first_depends_on_past on backfill command and default
> it to True**
>
> When doing backfill with ``depends_on_past`` dags, users will need to pass
> ``--ignore-first-depends-on-past``.
> We should default it as ``true`` to avoid confusion
>

But also, why is it a BaseOperator param?  I cannot find any sense in that
because.  With backfill you have an implied collection of dag runs and thus
a first is defined.  But not really so with a task.  If the implied
collection is all runs, well the first of them always ignores this since
there's no prior TI to examine.  So it seems this should def be removed
from base operator.

BUT now let us turn our attention to the bigger sibling,
`depends_on_past`.  This is a task param.

I think its primary reason for existing is in a backfill scenario, where
you want to backfill many runs concurrently, and some of the tasks can run
concurrently for multiple runs, but there's maybe one task that can't run
concurrently with another of the same task from a different dag run.  This
could allow some tasks from future exec dates to complete and then you
would just be waiting on the one depends_on_past one.

I guess I'm just asking the question, do we really need these params.  It
seems very edge case.  What if we moved depends_on_past to be dag scope?
That would be simpler.  Then essentially that means that either your dags
can run in parallel or they can't and not "they can kinda sorta run in
parallel".

If it's just about limiting concurrency for a task across all dags, there's
another param for that, `max_active_tis_per_dag`.

WDYT?

Reply via email to