Cool :)

On Tue, Jun 24, 2025 at 11:07 AM Amogh Desai <amoghdesai....@gmail.com>
wrote:

> Hello,
>
> Yeah I realised that there could probably be some places that require more
> attention and the pattern that I see
> is that those tests are mostly playing around or accessing the *default*
>  connections.
>
> I am working on a task that can separate out the DB dependency while
> creating *default* connections
> using the ENV backend too: Separate out creation of default Connections for
> tests and non-tests <https://github.com/apache/airflow/pull/52129>.
>
> This should help us in the long run and will also make it easier to migrate
> provider code away from direct DB access!
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Mon, Jun 23, 2025 at 11:57 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > yep - there are a few providers that require more "thorough" changes -
> ssh
> > for example :) ... We noticed when doing cleanup. But we already can
> > clean-up and remove a  lot of the pytest.mark.db_tests that were only
> there
> > due to connections :)
> >
> > I am thrilled at the prospect of having all our "providers" tests
> > eventually DB-less. This has been long in the making and is only possible
> > now for Airflow 3 :).
> >
> > While it will not be possible for **ALL** of them - for example
> "Executors"
> > - like edge3 have to use the DB, the regular providers that only provide
> > hooks/operators/triggers should all be db-less for tests - that will also
> > help us when finally task.sdk will be decoupled from "airflow-core" as we
> > will be able to remove "airflow-core" from being a test dependency in
> those
> > providers (and we will truly be able to see that indeed there are no
> > left-overs in providers to follow our "airflow 3" architecture.
> >
> > J.
> >
> >
> > On Mon, Jun 23, 2025 at 7:57 AM Amogh Desai <amoghdesai....@gmail.com>
> > wrote:
> >
> > > Thanks for the email, Jarek.
> > >
> > > A quick summary of the change: while working on moving BaseHook to the
> > task
> > > SDK in #51873,
> > > I noticed that many providers rely on `db.merge_conn()` to set up test
> > > connections & there are thousands
> > > of occurrences across the codebase. Doing this comes with few
> drawbacks:
> > >
> > >    - Slows down test execution due to database transactions.
> > >    - It introduces complexity by requiring DB setup/teardown.
> > >    - It occasionally results in flaky tests due to DB access issues.
> > >
> > > So I replaced those with a fixture that creates these test connections
> in
> > > the ENV backend, as it is a pre-configured backend
> > > for Airflow.
> > >
> > > There are still a few TODOs I plan to handle in follow ups (if someone
> > > wants to contribute, I will be more than happy to
> > > review too):
> > >
> > >    - Replace remaining occurrences in unit tests.
> > >    - Clean up some in-code TODO comments I’ve left as placeholders.
> > >    - Address similar usage in system tests.
> > >    - Possibly improve the fixture to support adding multiple
> connections
> > at
> > >    once.
> > >    - Update the Telegram provider tests (
> > >    *providers/telegram/tests/unit/telegram/hooks/test_telegram.py*).
> > >
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Sun, Jun 22, 2025 at 6:37 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Hello here,
> > > >
> > > > After #51930 where Amogh  introduced a way how tests can define
> > > connections
> > > > without DB and follow up in #52017 where I made an attempt to remove
> > some
> > > > of the *pytest.mark.db_test,* we should now remove those marks from
> > > > providers, where it is easy.
> > > >
> > > > Some providers still use DB for other things, but likely there are
> many
> > > > providers that only used db to create a Connection and we can turn
> > those
> > > > tests into non-db tests.
> > > >
> > > > I created an issue - where I have checkboxes for providers that need
> > > review
> > > > - and I have a kind request to the contributors and committers to -
> as
> > > > usual - help.
> > > >
> > > > The issue is here:
> > > >
> > > > https://github.com/apache/airflow/issues/52020
> > > >
> > > > And I provided detailed instructions (they are rather easy) on how to
> > do
> > > > it.
> > > >
> > > > Looking forward to your help !
> > > >
> > > > J.
> > > >
> > >
> >
>

Reply via email to