Re: [Discussion] Add a new TriggerRule: Never

2024-08-21 Thread Jarek Potiuk
> > > Also regarding just adding the code snippet inline in the docs, rather > than importing from the DAG: The nice thing about having it come from the > system test DAG is that the code snippet stays up to date. Even if the task > is being skipped, at least linters, static analysis, or failures d

Re: [Discussion] Add a new TriggerRule: Never

2024-08-21 Thread Oliveira, Niko
che.org Subject: RE: [EXT] [Discussion] Add a new TriggerRule: Never CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un exp

Re: [Discussion] Add a new TriggerRule: Never

2024-08-20 Thread Amogh Desai
art of the > > hot-loop, > > > then a user-defined callable could very possibly bring the whole thing > > to a > > > grinding stop, either intentionally or not. So yeah, I guess without > > > massive work to somehow decouple that so the whole scheduler loo

Re: [Discussion] Add a new TriggerRule: Never

2024-08-19 Thread Jarek Potiuk
, > > then a user-defined callable could very possibly bring the whole thing > to a > > grinding stop, either intentionally or not. So yeah, I guess without > > massive work to somehow decouple that so the whole scheduler loop didn't > > freeze waiting for a callable

Re: [Discussion] Add a new TriggerRule: Never

2024-08-19 Thread Daniel Standish
d idea. > > > - ferruzzi > > > > From: Jarek Potiuk > Sent: Sunday, August 18, 2024 2:46 PM > To: dev@airflow.apache.org > Subject: RE: [EXT] [Discussion] Add a new TriggerRule: Never > > CAUTION: This email originated f

Re: [Discussion] Add a new TriggerRule: Never

2024-08-19 Thread Ferruzzi, Dennis
idn't freeze waiting for a callable trigger rule, that could be a very bad idea. - ferruzzi From: Jarek Potiuk Sent: Sunday, August 18, 2024 2:46 PM To: dev@airflow.apache.org Subject: RE: [EXT] [Discussion] Add a new TriggerRule: Never CAUTION:

Re: [Discussion] Add a new TriggerRule: Never

2024-08-18 Thread Jarek Potiuk
While I think "system" tests use case is indeed - as Daniel wrote - not a good reason, I find it intriguing as a "commenting out" things from DAG. Sounds like a useful authoring feature to have a "never" trigger rule. The "callable" use case is likely a bit too much - we've been discussing "flexib

Re: [Discussion] Add a new TriggerRule: Never

2024-08-16 Thread Daniel Standish
It doesn't really sound right to include the task in the system test dag, purely for the purpose of getting it into the docs. Why just put it in the docs as an inline example? If you want to conditionally run certain tasks, one option is to raise AirflowSkipException

[Discussion] Add a new TriggerRule: Never

2024-08-16 Thread Ferruzzi, Dennis
Looking for thoughts on the idea of adding a new TriggerRule which will never fire. The use case I have in mind is system tests. Airflow docs pages harvest code snippets from the system tests and on occasion I have run into a situation where the snippet can't run in the context of the test (ru