I like the idea of having a trigger rule of `never`. I have also come across scenarios while migrating some example_dags to system tests that don't work right out of docs (intentional too, it should be good to run by setting up external systems)
I guess if we can have the new trigger rule and by default we modify them to run with `never` and they can be configured (using some env or so) to run with actual trigger rules. Somewhat like this: *DEFAULT_TRIGGER_RULE = TriggerRule.NEVER (override this one based on maybe an env)* *with DAG(dag_id='example_dag') as dag:* * task_1 = DummyOperator(task_id='task_1')* * task_2 = DummyOperator(task_id='task_2')* * for task in [task_1, task_2]:* * task.trigger_rule = DEFAULT_TRIGGER_RULE* Thanks & Regards, Amogh Desai On Mon, Aug 19, 2024 at 11:09 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > Example dags are one place we have historically put dags / tasks that go > in > docs but are not run as system tests. Is that not still the case? > > I think from the very beginning when I joined airflow - example dags were > meant to be "runnable" - so when you include core example dags in airflow - > you were supposed to run them in the UI when you enable them. This is for > "core" examples. > > Extending that concept, when the system tests were introduced ,"provider" > examples had the same assumption - that they are "runnable" examples (this > is basically what "system test" is - it's a runnable DAG that requires an > external system to be configured, but should "work" when used as a DAG in > general). > This has been improved via AIP-47 > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests > - where system tests were simplified and you could really have "runnable > examples" to become "system tests" - AIP-47 defined the way how you can get > the example dag and run it as "system test" and what conditions need to be > fulfilled. > Due to that providers do not have separe "examples" folder - they all place > examples in "tests/system". > > In case we have dashboards (google, amazon, LLM, Teradata) - they are > actually run and tested regularly (including for example today's voting for > providers - where Vincent confirmed they are passing with the new > providers. > In case we do not have dashboards, then they are "possibly runnable > manually" (and people use them when they are developing providers as a way > to test e2e if they work in a number of cases). > > So - similarly as core "example_dags" the function of provider > "example_dags" in providers is three-fold: > > * documentation snippets > * complete examples for people to look at > * system tests if you want to test the provider with it (either run > manually or via automated tests) > > The only difference is that the "provider" example dags require the system > tests and some configuration to connect them - but both "core" examples and > "provider" examples have those three functions: docs, complete examples, > and being runnable. > > Possibly the "manually run" example_dags are always runnable as system > tests and have some errors - but usually they are "close to be" - which > should decrease the burden of having to create them when you add or modify > a provider. And those that are regularly executed, they are quite likely > "working" - by just the fact of running them > > Here is a nice readme describing it in contributing guide: > > https://github.com/apache/airflow/blob/main/contributing-docs/testing/system_tests.rst > And more specific documentation on how to run them: > https://github.com/apache/airflow/tree/main/tests/system > > J. > > > > On Mon, Aug 19, 2024 at 6:17 PM Daniel Standish > <daniel.stand...@astronomer.io.invalid> wrote: > > > Example dags are one place we have historically put dags / tasks that go > in > > docs but are not run as system tests. Is that not still the case? > > > > On Mon, Aug 19, 2024 at 9:01 AM Ferruzzi, Dennis > > <ferru...@amazon.com.invalid> wrote: > > > > > Another part I had not considered is that since it's part of the > > hot-loop, > > > then a user-defined callable could very possibly bring the whole thing > > to a > > > grinding stop, either intentionally or not. So yeah, I guess without > > > massive work to somehow decouple that so the whole scheduler loop > didn't > > > freeze waiting for a callable trigger rule, that could be a very bad > > idea. > > > > > > > > > - ferruzzi > > > > > > > > > ________________________________ > > > From: Jarek Potiuk <ja...@potiuk.com> > > > Sent: Sunday, August 18, 2024 2:46 PM > > > To: dev@airflow.apache.org > > > Subject: RE: [EXT] [Discussion] Add a new TriggerRule: Never > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > While I think "system" tests use case is indeed - as Daniel wrote - > not a > > > good reason, I find it intriguing as a "commenting out" things from > DAG. > > > Sounds like a useful authoring feature to have a "never" trigger rule. > > > > > > The "callable" use case is likely a bit too much - we've been > discussing > > > "flexible" triggering rule, but since they are part of the scheduler > > > hot-loop, it's likely not a good idea to make them "editable" by the > > users, > > > and certainly not a good idea to have the code on trigger rules > > controlled > > > by DAG author (it would mean that code submitted by DAG author would be > > > executable in scheduler context) - we could make it possible via > plugins, > > > but IMHO having freely configurable triggering rules is quite a bit too > > > much of a freedom.. > > > > > > J. > > > > > > > > > On Fri, Aug 16, 2024 at 11:44 PM Daniel Standish > > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > > > It doesn't really sound right to include the task in the system test > > dag, > > > > purely for the purpose of getting it into the docs. Why just put it > in > > > the > > > > docs as an inline example? > > > > > > > > If you want to conditionally run certain tasks, one option is to > raise > > > > AirflowSkipException > > > > > > > > > >
