My preference would be Option 2 as that is closest to prod. Our approach is to spin up Airflow , install the providers that currently have deferrable functionality, and run system tests on those example DAGs on a daily basis . We are also running them on multiple executors (Celery, Kubernetes deployments)
On Sat, Jun 3, 2023 at 4:30 AM Vandon, Raphael <vand...@amazon.com.invalid> wrote: > Hello, > As we implement deferrable capabilities for operators, we're going to want > to update the sample dags/system tests. > The problem we encounter at the moment is that system tests are run with > pytest using the DebugExecutor, and it only runs the "worker part" of the > operators, without handling the "deferred part" that is taking place in the > Triggers. In system tests, a task ran in deferrable mode just hangs forever > in the “deferred” state, never completing. > > I see 3 approaches to that problem: > 1. Patch the DebugExecutor (or an other executor that'd become the > "official" executor for system tests) to handle deferred tasks. This bears > some complexity, but is probably doable with some ad-hoc code that won't be > the same as the "production" code but could execute the right things in > order (Trigger.run after the TaskDeferred exception, and then the > "execute_complete" method with the event returned) > 2. Change the way we run system tests, stop using pytest, and start > airflow in breeze, scheduling the example dags as actual dags. This has the > upside of being the closest to prod, but it's also a more > resource-intensive solution, and gathering the success/failure status and > the logs will require some piping. > 3. Do nothing, and forbid the use of deferrable in system tests. This is > not great for obvious reasons. > > It's a bit hard to say which option is "the best" before digging into it, > but my preference would go to 1. from what I tell so far. > > What do you community think about those options ? Did I forget a potential > solution ? > Other providers who also have system tests and deferrable operators, what > is your approach ? >
