We already have `data_interval_start` and `data_interval_end' as fields, and we need something else that can have more "abstract" meaning to apply to the whole run as "single thing". Using interval_date would be a bit ambiguous.
"Did you mean start or end actually when you mentioned interval date?" - is the question that I anticipate happening a lot if we mix those. J. On Sun, Feb 6, 2022 at 6:04 PM Howard Yoo <howard...@gmail.com> wrote: > Now I can understand why the data_date may not be a perfect fit to > describe the term. > > This is not to be against the logical_date, but what about > ‘interval_date?’ We have the schedule interval, which defines the duration > of the interval (e.g. 1day), so wouldn’t interval start and end date be a > better representation of it rather than the logical date? > > Just want to hear whether that has been brought up already or not. > > Howard > > Sent from my iPhone > > On Feb 6, 2022, at 10:25 AM, Jarek Potiuk <ja...@potiuk.com> wrote: > > > I wholeheartedly agree with TP on that one. I think while some time > ago "data date" could make sense, Airflow's future is much more than just > processing data intervals. > This is the primary use case and this is where Airflow shines od course, > but one of the good examples of how Airflow is used out there, and while we > are not really encouraging it, there are not only legitimate, but also > something that I hope Airflow will treat as first-time citizens soon (and > it kind of already is with custom timetables). > > Just an example here - for me one of the most eye-opening talks in last > year's Airflow Summit > https://airflowsummit.org/sessions/2021/provision-as-a-service/ > In this talk Cloudflare engineers explain how they manage the CloudFlare > infrastructure using Airflow. > > The "Data date" has no meaning in this case. But the "logical Date" (which > is the vaguest-possible one as TP explained) continues to have one. This is > the "logical date of the infrastructure provisioning". Thanks to Airflow > (as I understand it) Cloudflare is able to re-provision their services > to "yesterday's logical date infrastructure" today - for example. > > That would not fly with "data date". > > J, > >
