yep - there are a few providers that require more "thorough" changes - ssh for example :) ... We noticed when doing cleanup. But we already can clean-up and remove a lot of the pytest.mark.db_tests that were only there due to connections :)
I am thrilled at the prospect of having all our "providers" tests eventually DB-less. This has been long in the making and is only possible now for Airflow 3 :). While it will not be possible for **ALL** of them - for example "Executors" - like edge3 have to use the DB, the regular providers that only provide hooks/operators/triggers should all be db-less for tests - that will also help us when finally task.sdk will be decoupled from "airflow-core" as we will be able to remove "airflow-core" from being a test dependency in those providers (and we will truly be able to see that indeed there are no left-overs in providers to follow our "airflow 3" architecture. J. On Mon, Jun 23, 2025 at 7:57 AM Amogh Desai <amoghdesai....@gmail.com> wrote: > Thanks for the email, Jarek. > > A quick summary of the change: while working on moving BaseHook to the task > SDK in #51873, > I noticed that many providers rely on `db.merge_conn()` to set up test > connections & there are thousands > of occurrences across the codebase. Doing this comes with few drawbacks: > > - Slows down test execution due to database transactions. > - It introduces complexity by requiring DB setup/teardown. > - It occasionally results in flaky tests due to DB access issues. > > So I replaced those with a fixture that creates these test connections in > the ENV backend, as it is a pre-configured backend > for Airflow. > > There are still a few TODOs I plan to handle in follow ups (if someone > wants to contribute, I will be more than happy to > review too): > > - Replace remaining occurrences in unit tests. > - Clean up some in-code TODO comments I’ve left as placeholders. > - Address similar usage in system tests. > - Possibly improve the fixture to support adding multiple connections at > once. > - Update the Telegram provider tests ( > *providers/telegram/tests/unit/telegram/hooks/test_telegram.py*). > > > Thanks & Regards, > Amogh Desai > > > On Sun, Jun 22, 2025 at 6:37 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Hello here, > > > > After #51930 where Amogh introduced a way how tests can define > connections > > without DB and follow up in #52017 where I made an attempt to remove some > > of the *pytest.mark.db_test,* we should now remove those marks from > > providers, where it is easy. > > > > Some providers still use DB for other things, but likely there are many > > providers that only used db to create a Connection and we can turn those > > tests into non-db tests. > > > > I created an issue - where I have checkboxes for providers that need > review > > - and I have a kind request to the contributors and committers to - as > > usual - help. > > > > The issue is here: > > > > https://github.com/apache/airflow/issues/52020 > > > > And I provided detailed instructions (they are rather easy) on how to do > > it. > > > > Looking forward to your help ! > > > > J. > > >
