When there are differing opinions but seems that there is a favourable option someone (actually anyone) might call for a vote https://www.apache.org/foundation/voting.html#votes-on-code-modification
For such votes, committers have binding votes. -1 is a veto (usually needs to be justified) and kills the proposal unless the person who vetoed will change their mind. J. On Tue, Mar 22, 2022 at 4:53 PM Philippe Lanoe <pla...@cloudera.com.invalid> wrote: > I agree with Jarek in the sense that the DAG developer **should** know > when the DAG should start, however in practice for time-based scheduling it > can be cumbersome to maintain especially: > - everytime the jobs evolves and gets updated / new version > - when developers have to maintain hundreds/thousands of independent jobs, > keeping track of start_date for each of them can be difficult > > Not to mention that many companies do not have state-of-the-art CI/CD > processes which could allow them to dynamically change the start date. In > many cases when a change is made to a job, the developers simply want to > update the job and the next run will take it into account. > > I also agree with Collin and Constance that the "run last interval" is a > valid use case and therefore this parameter could accept three values to > handle all of these cases. > However I would suggest: > > Catchup=True : run all intervals > Catchup=False: do not run any past interval > Catchup="Last Interval" (or any better name :)) > > I know that the DAG authors who relied on Catchup=False to run the last > interval will need to adjust their DAG but if added a third option not to > trigger any run then the DAG authors who relied on catchup=False + set > start date will also need to update their DAG to have the proper value. And > in my opinion when I read Catchup=False the natural way of reading it is > "no catchup", therefore it would be better to fix it in the right direction. > > What is the next step here? Who can decide / approve such a new feature > request? > > Thanks, > Philippe > > On Mon, Mar 21, 2022 at 4:05 PM Constance Martineau > <consta...@astronomer.io.invalid> wrote: > >> I've had a variation of this debate a few times, and the behaviour you >> find intuitive in my opinion comes down to your background (software >> engineer vs data engineer vs BI developer vs data analyst), industry >> standards, and the scope of responsibility DAG authors have at your >> organization. My vote is to extend the catchup setting to either run all >> intervals (catchup=True today), run the most recent interval (catchup=False >> today) or schedule the next interval. I have seen organizations where both >> would be beneficial depending on the data pipeline in question. >> >> All of Alex's points are why I think we at least need the option. >> >> I came from an institutional investor, and we had plenty of DAGs that ran >> daily, weekly, monthly, quarterly and yearly. >> >> Many financial analysts - who were not DAG authors themselves - would >> have access to the Airflow Webserver in order to rerun tasks. They do not >> have the ability to adjust the start_date. During Audit season, it was >> common to see yearly dags being run for earlier years. To support this, >> means we needed to implement a start date for an earlier year. Saw DAG >> authors deal with this in two ways: Set the start_date to first day of >> prior year to get the DAG out and let it run, then modify the start_date to >> something earlier or set the start_date to something earlier, watch the DAG >> and quickly update the state of the dag to success (or fail). One is better >> than the other (no fun explaining to an executive why reports were >> accidentally sent externally), but neither are great. Option 3 - setting >> the start_date between the data interval period and leaving it - always >> caused confusion with other stakeholders. >> >> A global default, and DAG-level option would have been amazing. >> >> >> >> On Sun, Mar 20, 2022 at 5:15 PM Collin McNulty >> <col...@astronomer.io.invalid> wrote: >> >>> While that’s true, I think there are often stakeholders that expect a >>> DAG to run only on the day for which it is scheduled. It’s pretty >>> straightforward for me to explain to non-technical stakeholders that “aw >>> shucks we deployed just a little too late for this week’s run, we’ll run it >>> manually to fix it”. On the contrary, explaining why a DAG that I said >>> would run on Tuesdays sent out an alert on a Friday to a VP of Finance is … >>> rough. I understand that Airflow does not make guarantees about when tasks >>> will execute, but I try to scale such that when a task can start and when >>> it does start are close enough to not have to explain the difference to >>> other stakeholders. >>> >>> Editing start_date can also be tough in some conditions. If I’m baking a >>> DAG into an image, using build-once-deploy-to-many CI/CD, and testing in a >>> lower environment for longer than the interval between runs, I’m toast on >>> setting the start_date to avoid what I consider a spurious run. That’s a >>> lot of “ands” but I think it’s a fairly common set of circumstances we >>> should support. >>> >>> Collin McNulty >>> >>> >>> >>> On Sun, Mar 20, 2022 at 3:12 PM Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>>> Good. Love some mental stretching :). >>>> >>>> I believe you should **not** base the time of your run on the time it >>>> is released. Should not the DAG author know when there is a "start date" >>>> planned for the DAG? Should the decision on when the DAG interval start be >>>> made on combination of both start date in the dag **and** the time of not >>>> only when it's merged, but actually when airflow first **parses** the DAG. >>>> Not even mentioning the time zone issues. >>>> >>>> Imagine you case when DAG is merged 5 minutes between the midnight >>>> Mon/Tue and you have many DAGs. So many that parsing all the DAGs can take >>>> 20 minutes. Then the fact if your DAG runs this interval or that depends >>>> not even on the decision of when it is merged but also how long it takes >>>> Airflow to get to parse your DAG for the first time. >>>> >>>> Sounds pretty crazy :). >>>> >>>> J. >>>> >>>> >>>> On Sun, Mar 20, 2022 at 9:02 PM Collin McNulty >>>> <col...@astronomer.io.invalid> wrote: >>>> >>>>> Jarek, >>>>> >>>>> I tend to agree with you on this, but let me play devil’s advocate. If >>>>> I have a DAG that runs a report every Tuesday, I might want it to run >>>>> every >>>>> Tuesday starting whenever I am able to release the DAG. But if I release >>>>> on >>>>> a Friday, I don’t want it to try to run “for” last Tuesday. In this case, >>>>> the correct start_date for the dag is the day I release the DAG, but I >>>>> don’t know this date ahead of time and it differs per environment. Doing >>>>> this properly seems doable with a CD process that edits the DAG to insert >>>>> the start_date, but that’s fairly sophisticated tooling for a scenario >>>>> that >>>>> I imagine is quite common. >>>>> >>>>> Collin McNulty >>>>> >>>>> On Sun, Mar 20, 2022 at 1:55 PM Jarek Potiuk <ja...@potiuk.com> wrote: >>>>> >>>>>> Once again - why is it bad to set a start_date in the future, when - >>>>>> well - you **actually** want to run the first interval in the future ? >>>>>> What prevents you from setting the start-date to be a fixed time in >>>>>> the future, where the start date is within the interval you want to >>>>>> start first? Is it just "I do not want to specify conveniently >>>>>> whatever past date will be easy to type?" >>>>>> If this is the only reason, then it has a big drawback - because >>>>>> "start_date" is **actually** supposed to be the piece of metadata for >>>>>> the DAG that will tell you what was the intention of the DAG writer on >>>>>> when to start it. And precisely one that allows you to start things in >>>>>> the future. >>>>>> >>>>>> Am I missing something? >>>>>> >>>>>> On Sun, Mar 20, 2022 at 7:42 PM Larry Komenda >>>>>> <avoicelikerunningwa...@gmail.com> wrote: >>>>>> > >>>>>> > Alex, that's a good point regarding the need to run a DAG for the >>>>>> most recent schedule interval right away. I hadn't thought of that >>>>>> scenario >>>>>> as I haven't needed to build a DAG with that large of a scheduling gap. >>>>>> In >>>>>> that case I agree with you - it seems like it would make more sense to >>>>>> make >>>>>> this configurable. >>>>>> > >>>>>> > Perhaps there could be an additional DAG-level parameter that could >>>>>> be set alongside "catchup" to control this behavior. Or there could be a >>>>>> new parameter that could eventually replace "catchup" that supported 3 >>>>>> options - "catchup", "run most recent interval only", and "run next >>>>>> interval only". >>>>>> > >>>>>> > On Sat, Mar 19, 2022 at 1:02 PM Alex Begg <alex.b...@gmail.com> >>>>>> wrote: >>>>>> >> >>>>>> >> I would not consider it a bug to have the latest data interval run >>>>>> when you enable a DAG that is set to catchup=False. >>>>>> >> >>>>>> >> I have legitimate use for that feature by having my production >>>>>> environment have catchup_by_default=True but my lower environments are >>>>>> using catchup_by_default=False, meaning if I want to test the DAG >>>>>> behavior >>>>>> as scheduled in a lower environment I can just enable the DAG. >>>>>> >> >>>>>> >> For example, in a staging environment if I need to test out the >>>>>> functionality of a DAG that was scheduled for @monthly and there was no >>>>>> way >>>>>> to test the most recent data interval, than to test a true data interval >>>>>> of >>>>>> the DAG it could be many days, even weeks until they will occur. >>>>>> >> >>>>>> >> Triggering a DAG won’t run the latest data interval, it will use >>>>>> the current time as the logical_date, right? So that will won’t let me >>>>>> test >>>>>> a single as scheduled data interval. So in that @monthly senecio it will >>>>>> be >>>>>> impossible for me to test the functionality of a single data interval >>>>>> unless I wait multiple weeks. >>>>>> >> >>>>>> >> I see there could be a desire to not run the latest data interval >>>>>> and just start with whatever full interval follows the DAG being turned >>>>>> on. >>>>>> However I think that should be configurable, not fixed permanently. >>>>>> >> >>>>>> >> Alternatively it could be ideal to have a way to trigger a >>>>>> specific run for a catchup=False DAG that just got enabled by adding a 3d >>>>>> option to the trigger button drop down to trigger a past scheduled run. >>>>>> Then in that dialog the form can default to the most recent full data >>>>>> interval but then let you also specify a specific past interval based on >>>>>> the DAG's schedule. I often had to debug a DAG in production and I wanted >>>>>> to trigger a specific past data interval, not just the most recent. >>>>>> >> >>>>>> >> Alex Begg >>>>>> >> >>>>>> >> On Thu, Mar 17, 2022 at 4:58 PM Larry Komenda < >>>>>> avoicelikerunningwa...@gmail.com> wrote: >>>>>> >>> >>>>>> >>> I agree with this. I'd much rather have to trigger a single >>>>>> manual run the first time I enable a DAG than to either wait to enable >>>>>> until after I want it to run or by editing the start_date of the DAG >>>>>> itself. >>>>>> >>> >>>>>> >>> I'd be in favor of adjusting this behavior either permanently or >>>>>> by a configuration. >>>>>> >>> >>>>>> >>> On Fri, Mar 4, 2022 at 3:00 PM Philippe Lanoe >>>>>> <pla...@cloudera.com.invalid> wrote: >>>>>> >>>> >>>>>> >>>> Hello Daniel, >>>>>> >>>> >>>>>> >>>> Thank you for your answer. In your example, as I experienced, >>>>>> the first run would not be 2010-01-01 but 2022-03-03, 00:00:00 (it is >>>>>> currently March 4 - 21:00 here), which is the execution date >>>>>> corresponding >>>>>> to the start of the previous data interval, but the result is the same: >>>>>> an >>>>>> undesired dag run. (For instance, in case of cron schedule '00 22 * * *', >>>>>> one dagrun would be started immediately with execution date of >>>>>> 2022-03-02, >>>>>> 22:00:00) >>>>>> >>>> >>>>>> >>>> I also agree with you that it could be categorized as a bug and >>>>>> I would also vote for a fix. >>>>>> >>>> >>>>>> >>>> Would be great to have the feedback of others on this. >>>>>> >>>> >>>>>> >>>> On Fri, Mar 4, 2022 at 6:17 PM Daniel Standish >>>>>> <daniel.stand...@astronomer.io.invalid> wrote: >>>>>> >>>>> >>>>>> >>>>> You are saying, when you turn on for the first time a dag with >>>>>> e.g. @daily schedule, and catchup = False, if start date is 2010-01-01, >>>>>> then it would run first the 2010-01-01 run, then the current run >>>>>> (whatever >>>>>> yesterday is)? That sounds familiar. >>>>>> >>>>> >>>>>> >>>>> Yeah I don't like that behavior. I agree that, as you say, >>>>>> it's not the intuitive behavior. Seems it could reasonably be >>>>>> categorized >>>>>> as a bug. I'd prefer we just "fix" it rather than making it >>>>>> configurable. >>>>>> But some might have concerns re backcompat. >>>>>> >>>>> >>>>>> >>>>> What do others think? >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> -- >>>>> >>>>> Collin McNulty >>>>> Lead Airflow Engineer >>>>> >>>>> Email: col...@astronomer.io <john....@astronomer.io> >>>>> Time zone: US Central (CST UTC-6 / CDT UTC-5) >>>>> >>>>> >>>>> <https://www.astronomer.io/> >>>>> >>>> -- >>> >>> Collin McNulty >>> Lead Airflow Engineer >>> >>> Email: col...@astronomer.io <john....@astronomer.io> >>> Time zone: US Central (CST UTC-6 / CDT UTC-5) >>> >>> >>> <https://www.astronomer.io/> >>> >> >> >> -- >> >> Constance Martineau >> Product Manager >> >> Email: consta...@astronomer.io >> Time zone: US Eastern (EST UTC-5 / EDT UTC-4) >> >> >> <https://www.astronomer.io/> >> >>
