To me I think if we have a trigger rule of run always, then a rule of run never isn't so unreasonable.
Also regarding just adding the code snippet inline in the docs, rather than importing from the DAG: The nice thing about having it come from the system test DAG is that the code snippet stays up to date. Even if the task is being skipped, at least linters, static analysis, or failures due to API changes during parsing will get caught if it is "live code". ________________________________ From: Amogh Desai <amoghdesai....@gmail.com> Sent: Tuesday, August 20, 2024 8:50:33 PM To: dev@airflow.apache.org Subject: RE: [EXT] [Discussion] Add a new TriggerRule: Never CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. I like the idea of having a trigger rule of `never`. I have also come across scenarios while migrating some example_dags to system tests that don't work right out of docs (intentional too, it should be good to run by setting up external systems) I guess if we can have the new trigger rule and by default we modify them to run with `never` and they can be configured (using some env or so) to run with actual trigger rules. Somewhat like this: *DEFAULT_TRIGGER_RULE = TriggerRule.NEVER (override this one based on maybe an env)* *with DAG(dag_id='example_dag') as dag:* * task_1 = DummyOperator(task_id='task_1')* * task_2 = DummyOperator(task_id='task_2')* * for task in [task_1, task_2]:* * task.trigger_rule = DEFAULT_TRIGGER_RULE* Thanks & Regards, Amogh Desai On Mon, Aug 19, 2024 at 11:09 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > Example dags are one place we have historically put dags / tasks that go > in > docs but are not run as system tests. Is that not still the case? > > I think from the very beginning when I joined airflow - example dags were > meant to be "runnable" - so when you include core example dags in airflow - > you were supposed to run them in the UI when you enable them. This is for > "core" examples. > > Extending that concept, when the system tests were introduced ,"provider" > examples had the same assumption - that they are "runnable" examples (this > is basically what "system test" is - it's a runnable DAG that requires an > external system to be configured, but should "work" when used as a DAG in > general). > This has been improved via AIP-47 > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests > - where system tests were simplified and you could really have "runnable > examples" to become "system tests" - AIP-47 defined the way how you can get > the example dag and run it as "system test" and what conditions need to be > fulfilled. > Due to that providers do not have separe "examples" folder - they all place > examples in "tests/system". > > In case we have dashboards (google, amazon, LLM, Teradata) - they are > actually run and tested regularly (including for example today's voting for > providers - where Vincent confirmed they are passing with the new > providers. > In case we do not have dashboards, then they are "possibly runnable > manually" (and people use them when they are developing providers as a way > to test e2e if they work in a number of cases). > > So - similarly as core "example_dags" the function of provider > "example_dags" in providers is three-fold: > > * documentation snippets > * complete examples for people to look at > * system tests if you want to test the provider with it (either run > manually or via automated tests) > > The only difference is that the "provider" example dags require the system > tests and some configuration to connect them - but both "core" examples and > "provider" examples have those three functions: docs, complete examples, > and being runnable. > > Possibly the "manually run" example_dags are always runnable as system > tests and have some errors - but usually they are "close to be" - which > should decrease the burden of having to create them when you add or modify > a provider. And those that are regularly executed, they are quite likely > "working" - by just the fact of running them > > Here is a nice readme describing it in contributing guide: > > https://github.com/apache/airflow/blob/main/contributing-docs/testing/system_tests.rst > And more specific documentation on how to run them: > https://github.com/apache/airflow/tree/main/tests/system > > J. > > > > On Mon, Aug 19, 2024 at 6:17 PM Daniel Standish > <daniel.stand...@astronomer.io.invalid> wrote: > > > Example dags are one place we have historically put dags / tasks that go > in > > docs but are not run as system tests. Is that not still the case? > > > > On Mon, Aug 19, 2024 at 9:01 AM Ferruzzi, Dennis > > <ferru...@amazon.com.invalid> wrote: > > > > > Another part I had not considered is that since it's part of the > > hot-loop, > > > then a user-defined callable could very possibly bring the whole thing > > to a > > > grinding stop, either intentionally or not. So yeah, I guess without > > > massive work to somehow decouple that so the whole scheduler loop > didn't > > > freeze waiting for a callable trigger rule, that could be a very bad > > idea. > > > > > > > > > - ferruzzi > > > > > > > > > ________________________________ > > > From: Jarek Potiuk <ja...@potiuk.com> > > > Sent: Sunday, August 18, 2024 2:46 PM > > > To: dev@airflow.apache.org > > > Subject: RE: [EXT] [Discussion] Add a new TriggerRule: Never > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > While I think "system" tests use case is indeed - as Daniel wrote - > not a > > > good reason, I find it intriguing as a "commenting out" things from > DAG. > > > Sounds like a useful authoring feature to have a "never" trigger rule. > > > > > > The "callable" use case is likely a bit too much - we've been > discussing > > > "flexible" triggering rule, but since they are part of the scheduler > > > hot-loop, it's likely not a good idea to make them "editable" by the > > users, > > > and certainly not a good idea to have the code on trigger rules > > controlled > > > by DAG author (it would mean that code submitted by DAG author would be > > > executable in scheduler context) - we could make it possible via > plugins, > > > but IMHO having freely configurable triggering rules is quite a bit too > > > much of a freedom.. > > > > > > J. > > > > > > > > > On Fri, Aug 16, 2024 at 11:44 PM Daniel Standish > > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > > > It doesn't really sound right to include the task in the system test > > dag, > > > > purely for the purpose of getting it into the docs. Why just put it > in > > > the > > > > docs as an inline example? > > > > > > > > If you want to conditionally run certain tasks, one option is to > raise > > > > AirflowSkipException > > > > > > > > > >
