devel-common makes sense to me On 2025/03/04 21:13:47 "Oliveira, Niko" wrote: > +1 to devel-common from me > > ________________________________ > From: Ferruzzi, Dennis <ferru...@amazon.com.INVALID> > Sent: Tuesday, March 4, 2025 11:21:20 AM > To: dev@airflow.apache.org > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution > for development > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne > cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas > confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le > contenu ne présente aucun risque. > > > > devel-common sounds reasonable > > > - ferruzzi > > > ________________________________ > From: Jarek Potiuk <ja...@potiuk.com> > Sent: Tuesday, March 4, 2025 10:53 AM > To: dev@airflow.apache.org > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution > for development > > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne > cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas > confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le > contenu ne présente aucun risque. > > > I am doing a bit more cleanup, and I have found that the easier way to fix > some of the remaining issues will be to clean-up (and remove) the remaining > editable devel dependencies and incorporate them all in the "tests-common" > package. > > You can take a look at the PR: https://github.com/apache/airflow/pull/47281 > - but basically what it means is: > > * all "devel" dependencies are added as required dependencies of > "tests-common" (except "doc" - I will treat doc separately). > * I removed all "legacy" extras from the "airflow" package including "bundle" > extras: "devel-ci", "devel-db" and a few others - except installing "all" > dependencies as they were pretty useless. Except "editable" all only - see > below - we will have no more devel and bundle extras (Ash - I guess this is > what you were looking forward to :) ) > * Instead we have one "all" extra that is available only in editable mode - > it's not documented in user documentation and it is really only useful to > install everything with `pip` (with uv you get the same with `uv sync > --all-extras`) - this is still used internally in the CI image to run `uv > pip install .[all] --constraints` until we switch to use "uv.lock" in the > future > * hatch_build.py is significantly simpler and easier to understand now - with > all the bundle removal and moving all dependencies to ./tests-common > * we still need dynamic dependencies and ./hatch_build.py - but less and > less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in > April we will likely be able to turn our optional dependencies into static > pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs > approval and implementation) we will likely be able to have static > pyproject.toml required dependencies as well. > * I updated install and contributing docs to be "uv first" - presenting as > recommended and the first option to go with `uv` - as it is becoming > deceptively simple now to work with both - airflow and providers (and it will > be even simpler after few next PRs) > > Now. THE BIG QUESTION - naming again. With all those changes..... > `tests-common` is becoming more of a `devel-common` package - because what it > will do - it will contribute to all other sub-projects all the development > tooling that is needed for those other projects to be developed. > > Shall we name it "devel-common" instead of `tests-common`? > > Part of why I think it makes sense is this specification in > `tests-common/pyproject.toml` (see attachment) - also https://ibb.co/1cwZQRm > if you do not see attachment. Note that these are generally "common" devel > dependencies - and each of the packages can contribute their own (including > task-sdk, airflow-core (future), every provider etc.) > > [Screenshot from 2025-03-04 19-50-32.png] > > > J. > > > > On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk > <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: > Hello everyone. > > I created the PR for that https://github.com/apache/airflow/pull/47281. > > It's even nicer than I anticipated. I love the new super-simple workflows > this restructuring finally enabled. > > With `uv` and workspace, and the new structure of tests, developing and > running tests for providers, task-sdk or any of the other future > sub-projects becomes very, very straightforward, we avoid duplication of > pytest options and switching between airflow, task-sdk, providers tests will > be simple and straightforward. > > 1) First of all with this change, we remove `devel-tests` extra. It was > always needed in local venv to install all test dependencies, But this is > completely gone right now. Test dependencies are automatically installed now > when you run `uv sync` - and you do not need to specify `--extra > devel-tests`. If you are still using `pip` (I strongly recommend switching to > uv) - you just install `pip install -e ./tests-common`. > > 2) the previous way of syncing and running tests in "everything installed" > mode works as it worked before: > > uv sync --all-extras > > This installs all possible extras of airflow, allows you to run tests for all > providers, activate the venv in `.venv` and run `pytests tests/always` or > `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should all > work fine as it did before > > 3) you can also install dependencies of a selected provider (or a few of > those) in your .venv > > uv sync --extra mongo > > 3) All the `breeze testing provider-tests --test-type "Providers[mongo]"" and > generally all the ways to replicate what we run in CI inside breeze > containers with selective checks will also continue to work - no changes here. > > But then, a little bit of standardization, a bit of pre-commits that > automatically update `pyproject.toml` files of ours with cross-provider > dependencies and a little of `uv` workspace magic sprinkled over the repo of > ours and now we can do something much more straightforward: > > You can change directory to task_sdk, or provider folder of your choice and > simply run: > > cd providers/mongo > uv run pytest > > or > > cd task_sdk > uv run pytest > > And it will do what you would expect to do - even if this is the first thing > you do after checking out Airflow's repo without any earlier syncing or > preparation. Yes. It's that simple. > > The `uv run pytest` command will now automatically: install python, create a > venv, sync your venv to ONLY have dependencies needed for your provider or > any of the providers imported by your provider, and all test dependencies > needed, it will find only non-system, non-integration tests and run them. > That's it. If you want to only develop one provider - you will not have to > worry any more about breeze image or setting up all other dependencies. > > Go ahead and try it. It's unbelievably simple. > > J. > > > > On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor > <a...@apache.org<mailto:a...@apache.org>> wrote: > I like this approach, lets do it! > > > On 25 Feb 2025, at 16:02, Jarek Potiuk > > <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: > > > > airflow-core > > task-sdk > > tests-common > >
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
