Hello, Yeah I realised that there could probably be some places that require more attention and the pattern that I see is that those tests are mostly playing around or accessing the *default* connections.
I am working on a task that can separate out the DB dependency while creating *default* connections using the ENV backend too: Separate out creation of default Connections for tests and non-tests <https://github.com/apache/airflow/pull/52129>. This should help us in the long run and will also make it easier to migrate provider code away from direct DB access! Thanks & Regards, Amogh Desai On Mon, Jun 23, 2025 at 11:57 AM Jarek Potiuk <ja...@potiuk.com> wrote: > yep - there are a few providers that require more "thorough" changes - ssh > for example :) ... We noticed when doing cleanup. But we already can > clean-up and remove a lot of the pytest.mark.db_tests that were only there > due to connections :) > > I am thrilled at the prospect of having all our "providers" tests > eventually DB-less. This has been long in the making and is only possible > now for Airflow 3 :). > > While it will not be possible for **ALL** of them - for example "Executors" > - like edge3 have to use the DB, the regular providers that only provide > hooks/operators/triggers should all be db-less for tests - that will also > help us when finally task.sdk will be decoupled from "airflow-core" as we > will be able to remove "airflow-core" from being a test dependency in those > providers (and we will truly be able to see that indeed there are no > left-overs in providers to follow our "airflow 3" architecture. > > J. > > > On Mon, Jun 23, 2025 at 7:57 AM Amogh Desai <amoghdesai....@gmail.com> > wrote: > > > Thanks for the email, Jarek. > > > > A quick summary of the change: while working on moving BaseHook to the > task > > SDK in #51873, > > I noticed that many providers rely on `db.merge_conn()` to set up test > > connections & there are thousands > > of occurrences across the codebase. Doing this comes with few drawbacks: > > > > - Slows down test execution due to database transactions. > > - It introduces complexity by requiring DB setup/teardown. > > - It occasionally results in flaky tests due to DB access issues. > > > > So I replaced those with a fixture that creates these test connections in > > the ENV backend, as it is a pre-configured backend > > for Airflow. > > > > There are still a few TODOs I plan to handle in follow ups (if someone > > wants to contribute, I will be more than happy to > > review too): > > > > - Replace remaining occurrences in unit tests. > > - Clean up some in-code TODO comments I’ve left as placeholders. > > - Address similar usage in system tests. > > - Possibly improve the fixture to support adding multiple connections > at > > once. > > - Update the Telegram provider tests ( > > *providers/telegram/tests/unit/telegram/hooks/test_telegram.py*). > > > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Sun, Jun 22, 2025 at 6:37 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Hello here, > > > > > > After #51930 where Amogh introduced a way how tests can define > > connections > > > without DB and follow up in #52017 where I made an attempt to remove > some > > > of the *pytest.mark.db_test,* we should now remove those marks from > > > providers, where it is easy. > > > > > > Some providers still use DB for other things, but likely there are many > > > providers that only used db to create a Connection and we can turn > those > > > tests into non-db tests. > > > > > > I created an issue - where I have checkboxes for providers that need > > review > > > - and I have a kind request to the contributors and committers to - as > > > usual - help. > > > > > > The issue is here: > > > > > > https://github.com/apache/airflow/issues/52020 > > > > > > And I provided detailed instructions (they are rather easy) on how to > do > > > it. > > > > > > Looking forward to your help ! > > > > > > J. > > > > > >
