> I strongly disagree with the proposal of changing the default for all DAGs. This requires every user that does not specify catchup to modify their DAGs.
I don't think that's accurate. Tamara's proposal is just to change the default value of the global configuration that controls this <https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#catchup-by-default>, not remove the option entirely or force people to specify it at the dag level. If teams rely on the current behaviour, they can simply update the global config to keep it as True. I do understand the concern about operational overhead in large organizations. It’s reasonable for platform teams to want to avoid requiring DAG authors to modify their code just to migrate to Airflow 3. But in this case, the solution is straightforward: Platform teams can set the config back to True as part of their migration process, all without touching DAG author code. The real question is whether DAG authors or platform teams should bear the "default" burden? Keeping catchup enabled by default might make migrations easier for platform teams, but it increases risk and unnecessary work for DAG authors, especially those who may not understand the relationship between start dates, logical dates, and catchup behaviour. Based on what we see on Astro, the majority of DAGs on our platform have catchup set to False. If we're prioritizing DAG authors, disabling catchup by default is the right move. On Wed, Mar 5, 2025 at 1:55 PM Michał Modras <michalmod...@google.com.invalid> wrote: > I strongly disagree with the proposal of changing the default for all DAGs. > This requires every user that does not specify catchup to modify their > DAGs. As pointed out in other similar threads about changes requiring DAG > code changes: > > >I am concerned simply because it is a physical code change, and one that > would affect a good percentage of DAGs out there. No matter how complex the > change is, it forces the users to modify their code, which is huge > operational overhead in bigger organizations. Imagine - there could be a > central platform team responsible for migrating to Airflow 3. But then, > before Airflow 3 deployments can be used, each team using particular > Airflow deployment would need to modify the code. At the same time, the > platform team might not be permitted to touch code of these teams - > these are different personas. It can easily become a very complex migration > procedure to expedite across the organization, even if the code change > itself is simple. > > Let's prioritise Airflow 3 ease of adoption. The fewer breaking changes, > the faster it will be adopted across the industry. > > > On Wed, Mar 5, 2025 at 7:43 PM Tamara Fingerlin > <tamara.finger...@astronomer.io.invalid> wrote: > > > Hey there, long time reader, first time poster here :) > > > > > > *tl;dr:* > > > > *As part of the 3.0 release, I would like to propose changing the default > > for `catchup_by_default` from True to False. * > > > > *This discussion asks for input and whether this can be a lazy consensus > or > > should be a vote.* > > > > Timethings are hard. Especially for new Airflow users. When I first > started > > using Airflow, it took me a while (and one or two napkin sketches) to > > understand how to set the start_date and trigger the DAG runs I wanted. > To > > this day, I still often just pick a date a couple days in the past and > just > > set catchup to False to not have to do the math on schedules that aren’t > > straightforward. > > > > As part of the Astronomer DevRel team, I teach users about Airflow. This > > “gotcha” is especially common for new users to run into. Imagine that > > you’re a new person writing a DAG with a start date of Jan 1st. You > unpause > > your DAG, and you unexpectedly see a large amount of DAG runs kicking > off. > > When we talk to practitioners in Airflow 101 webinars, many share that > have > > accidentally overflooded their Airflow deployment because they didn’t > > understand the relationship between the start_date and DAG runs, by not > > knowing about catchup, or by forgetting to add the line when writing new > > dags. > > > > This is why I propose changing the config catchup_by_default from True to > > False. > > > > Pro: > > > > - > > > > Less accidental DAG runs by beginners and people accidentally > forgetting > > catchup=False. Especially for beginners this is confusing. > > - > > > > One parameter less for beginners to learn when they write their first > > DAG, one line less to write for most DAGs in the future. > > > > > > Con: > > > > - > > > > Breaking change, but since it is a config value a minor one that users > > who want the old behavior can easily adjust. We can add something to > the > > config linter to highlight this change, and prompt users to set the > > value > > back to True if they prefer the current behaviour. > > > > > > > > Elad pointed out that there has been previous discussion on this > including: > > > > > > - > > > > The suggestion to move away from a binary option to an enum to have > more > > fine grained control on when to catch up (only when the DAG is first > > turned > > on, only when the DAG is not first turned on, always, never…) #35392 > > <https://github.com/apache/airflow/pull/35392#issuecomment-1792254428 > > > > > > > > This is a good idea, but there is more to figure out. As others have > > pointed out in the PR, if we go this route this means more > configurations. > > I don’t think changing the default blocks from going this route in the > > future. > > > > When the time comes, we could turn this into an enum. For migration > > purposes and to avoid DAG code changes, we could add more options > including > > “always” and “never”, and map True to “always” and False to “never”. For > > this feature, what we do at the global level should match what’s > available > > at the DAG level, meaning the DAG parameter will also need to be adjusted > > accordingly. Even in this new model, defaulting to "False"/“never” is the > > right way forward. > > > > > > - > > > > #38168 <https://github.com/apache/airflow/pull/38168> > > discussed/proposed > > the possibility for an option to disable the “catch up of the latest > DAG > > run” behavior when unpausing a DAG with catchup=False. > > > > While it is closely related I think this is a separate issue that > merits > > its own discussion. I.e. we’re not talking about changing a default > > value, > > we’re talking about fundamentally changing what catchup=False means. > > It’s a > > lot less alarming for users to accidentally trigger one DAG run > because > > they didn’t understand catchup behaviour, versus a large number of DAG > > runs. That is the confusing behaviour, and what I’m hoping to prevent > > with > > the default change. > > > > > > > > I started a PR here for the most basic option for this change, just > > changing the config variable from True to False: > > https://github.com/apache/airflow/pull/47354 > > > > If there is general alignment I’d try for a lazy consensus, otherwise a > > vote 🙂 > > >
