Another part I had not considered is that since it's part of the hot-loop, then a user-defined callable could very possibly bring the whole thing to a grinding stop, either intentionally or not. So yeah, I guess without massive work to somehow decouple that so the whole scheduler loop didn't freeze waiting for a callable trigger rule, that could be a very bad idea.
- ferruzzi ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Sunday, August 18, 2024 2:46 PM To: dev@airflow.apache.org Subject: RE: [EXT] [Discussion] Add a new TriggerRule: Never CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. While I think "system" tests use case is indeed - as Daniel wrote - not a good reason, I find it intriguing as a "commenting out" things from DAG. Sounds like a useful authoring feature to have a "never" trigger rule. The "callable" use case is likely a bit too much - we've been discussing "flexible" triggering rule, but since they are part of the scheduler hot-loop, it's likely not a good idea to make them "editable" by the users, and certainly not a good idea to have the code on trigger rules controlled by DAG author (it would mean that code submitted by DAG author would be executable in scheduler context) - we could make it possible via plugins, but IMHO having freely configurable triggering rules is quite a bit too much of a freedom.. J. On Fri, Aug 16, 2024 at 11:44 PM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > It doesn't really sound right to include the task in the system test dag, > purely for the purpose of getting it into the docs. Why just put it in the > docs as an inline example? > > If you want to conditionally run certain tasks, one option is to raise > AirflowSkipException >
