devel-common sounds reasonable
- ferruzzi ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Tuesday, March 4, 2025 10:53 AM To: dev@airflow.apache.org Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution for development CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. I am doing a bit more cleanup, and I have found that the easier way to fix some of the remaining issues will be to clean-up (and remove) the remaining editable devel dependencies and incorporate them all in the "tests-common" package. You can take a look at the PR: https://github.com/apache/airflow/pull/47281 - but basically what it means is: * all "devel" dependencies are added as required dependencies of "tests-common" (except "doc" - I will treat doc separately). * I removed all "legacy" extras from the "airflow" package including "bundle" extras: "devel-ci", "devel-db" and a few others - except installing "all" dependencies as they were pretty useless. Except "editable" all only - see below - we will have no more devel and bundle extras (Ash - I guess this is what you were looking forward to :) ) * Instead we have one "all" extra that is available only in editable mode - it's not documented in user documentation and it is really only useful to install everything with `pip` (with uv you get the same with `uv sync --all-extras`) - this is still used internally in the CI image to run `uv pip install .[all] --constraints` until we switch to use "uv.lock" in the future * hatch_build.py is significantly simpler and easier to understand now - with all the bundle removal and moving all dependencies to ./tests-common * we still need dynamic dependencies and ./hatch_build.py - but less and less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in April we will likely be able to turn our optional dependencies into static pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs approval and implementation) we will likely be able to have static pyproject.toml required dependencies as well. * I updated install and contributing docs to be "uv first" - presenting as recommended and the first option to go with `uv` - as it is becoming deceptively simple now to work with both - airflow and providers (and it will be even simpler after few next PRs) Now. THE BIG QUESTION - naming again. With all those changes..... `tests-common` is becoming more of a `devel-common` package - because what it will do - it will contribute to all other sub-projects all the development tooling that is needed for those other projects to be developed. Shall we name it "devel-common" instead of `tests-common`? Part of why I think it makes sense is this specification in `tests-common/pyproject.toml` (see attachment) - also https://ibb.co/1cwZQRm if you do not see attachment. Note that these are generally "common" devel dependencies - and each of the packages can contribute their own (including task-sdk, airflow-core (future), every provider etc.) [Screenshot from 2025-03-04 19-50-32.png] J. On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: Hello everyone. I created the PR for that https://github.com/apache/airflow/pull/47281. It's even nicer than I anticipated. I love the new super-simple workflows this restructuring finally enabled. With `uv` and workspace, and the new structure of tests, developing and running tests for providers, task-sdk or any of the other future sub-projects becomes very, very straightforward, we avoid duplication of pytest options and switching between airflow, task-sdk, providers tests will be simple and straightforward. 1) First of all with this change, we remove `devel-tests` extra. It was always needed in local venv to install all test dependencies, But this is completely gone right now. Test dependencies are automatically installed now when you run `uv sync` - and you do not need to specify `--extra devel-tests`. If you are still using `pip` (I strongly recommend switching to uv) - you just install `pip install -e ./tests-common`. 2) the previous way of syncing and running tests in "everything installed" mode works as it worked before: uv sync --all-extras This installs all possible extras of airflow, allows you to run tests for all providers, activate the venv in `.venv` and run `pytests tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should all work fine as it did before 3) you can also install dependencies of a selected provider (or a few of those) in your .venv uv sync --extra mongo 3) All the `breeze testing provider-tests --test-type "Providers[mongo]"" and generally all the ways to replicate what we run in CI inside breeze containers with selective checks will also continue to work - no changes here. But then, a little bit of standardization, a bit of pre-commits that automatically update `pyproject.toml` files of ours with cross-provider dependencies and a little of `uv` workspace magic sprinkled over the repo of ours and now we can do something much more straightforward: You can change directory to task_sdk, or provider folder of your choice and simply run: cd providers/mongo uv run pytest or cd task_sdk uv run pytest And it will do what you would expect to do - even if this is the first thing you do after checking out Airflow's repo without any earlier syncing or preparation. Yes. It's that simple. The `uv run pytest` command will now automatically: install python, create a venv, sync your venv to ONLY have dependencies needed for your provider or any of the providers imported by your provider, and all test dependencies needed, it will find only non-system, non-integration tests and run them. That's it. If you want to only develop one provider - you will not have to worry any more about breeze image or setting up all other dependencies. Go ahead and try it. It's unbelievably simple. J. On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <a...@apache.org<mailto:a...@apache.org>> wrote: I like this approach, lets do it! > On 25 Feb 2025, at 16:02, Jarek Potiuk > <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: > > airflow-core > task-sdk > tests-common
