Cool :) On Tue, Jun 24, 2025 at 11:07 AM Amogh Desai <amoghdesai....@gmail.com> wrote:
> Hello, > > Yeah I realised that there could probably be some places that require more > attention and the pattern that I see > is that those tests are mostly playing around or accessing the *default* > connections. > > I am working on a task that can separate out the DB dependency while > creating *default* connections > using the ENV backend too: Separate out creation of default Connections for > tests and non-tests <https://github.com/apache/airflow/pull/52129>. > > This should help us in the long run and will also make it easier to migrate > provider code away from direct DB access! > > Thanks & Regards, > Amogh Desai > > > On Mon, Jun 23, 2025 at 11:57 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > yep - there are a few providers that require more "thorough" changes - > ssh > > for example :) ... We noticed when doing cleanup. But we already can > > clean-up and remove a lot of the pytest.mark.db_tests that were only > there > > due to connections :) > > > > I am thrilled at the prospect of having all our "providers" tests > > eventually DB-less. This has been long in the making and is only possible > > now for Airflow 3 :). > > > > While it will not be possible for **ALL** of them - for example > "Executors" > > - like edge3 have to use the DB, the regular providers that only provide > > hooks/operators/triggers should all be db-less for tests - that will also > > help us when finally task.sdk will be decoupled from "airflow-core" as we > > will be able to remove "airflow-core" from being a test dependency in > those > > providers (and we will truly be able to see that indeed there are no > > left-overs in providers to follow our "airflow 3" architecture. > > > > J. > > > > > > On Mon, Jun 23, 2025 at 7:57 AM Amogh Desai <amoghdesai....@gmail.com> > > wrote: > > > > > Thanks for the email, Jarek. > > > > > > A quick summary of the change: while working on moving BaseHook to the > > task > > > SDK in #51873, > > > I noticed that many providers rely on `db.merge_conn()` to set up test > > > connections & there are thousands > > > of occurrences across the codebase. Doing this comes with few > drawbacks: > > > > > > - Slows down test execution due to database transactions. > > > - It introduces complexity by requiring DB setup/teardown. > > > - It occasionally results in flaky tests due to DB access issues. > > > > > > So I replaced those with a fixture that creates these test connections > in > > > the ENV backend, as it is a pre-configured backend > > > for Airflow. > > > > > > There are still a few TODOs I plan to handle in follow ups (if someone > > > wants to contribute, I will be more than happy to > > > review too): > > > > > > - Replace remaining occurrences in unit tests. > > > - Clean up some in-code TODO comments I’ve left as placeholders. > > > - Address similar usage in system tests. > > > - Possibly improve the fixture to support adding multiple > connections > > at > > > once. > > > - Update the Telegram provider tests ( > > > *providers/telegram/tests/unit/telegram/hooks/test_telegram.py*). > > > > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Sun, Jun 22, 2025 at 6:37 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > Hello here, > > > > > > > > After #51930 where Amogh introduced a way how tests can define > > > connections > > > > without DB and follow up in #52017 where I made an attempt to remove > > some > > > > of the *pytest.mark.db_test,* we should now remove those marks from > > > > providers, where it is easy. > > > > > > > > Some providers still use DB for other things, but likely there are > many > > > > providers that only used db to create a Connection and we can turn > > those > > > > tests into non-db tests. > > > > > > > > I created an issue - where I have checkboxes for providers that need > > > review > > > > - and I have a kind request to the contributors and committers to - > as > > > > usual - help. > > > > > > > > The issue is here: > > > > > > > > https://github.com/apache/airflow/issues/52020 > > > > > > > > And I provided detailed instructions (they are rather easy) on how to > > do > > > > it. > > > > > > > > Looking forward to your help ! > > > > > > > > J. > > > > > > > > > >