Also see discussion https://github.com/apache/airflow/pull/42252#discussion_r1785191841 for more context why we need it.
On Wed, Oct 2, 2024 at 1:13 PM Jarek Potiuk <ja...@potiuk.com> wrote: > I would like to propose the way we are handling "not-ready" providers for > Airflow 3 development - basically by manually releasing the "dev" version > of the providers to PyPI. > > We are not yet releasing the standard provider as it is marked as > "not-ready" (and that's good because we do not want regular users to > install it). But we need to make it "pre-installed" (i.e. installed when > the "apache-airflow" package is installed from main. I created PR for that > https://github.com/apache/airflow/pull/42676 > > We already have the "standard" provider in our code and we have some > references to it (there were a few CI failures caused by the provider > missing for PROD builds (solved already). But while our CI handles it > (hopefully) nicely, when you install the in-development version of Airflow > in "standard" (non-editable) mode - it will attempt to install the > "apache-airflow-providers-standard" - after > https://github.com/apache/airflow/pull/42676 is merged. > > As I understand it - the prerelease provider will be installed with > "development" version of airflow 3 because when `pip` can only find > pre-release version of the package (and no "released" version) - it will > install the pre-release one, but it's not going to be seen as "official" > release version. > > I already reserved the "apache-airflow-providers-standard" project for the > Apache Airflow organisation so this is the matter of running a single > breeze command to build the package with ".dev0" classifier and uploading > it to PyPI. > > Similar situation will be with "edge" executor - which I think we should > release manually in a similar manner until we mark it as "ready". > > We could probably add pre-release providers to our regular release process > if need be, but I think manually releasing the dev* versions of those > providers as needed is far less hassle (we just need a few PMC members who > have access to PyPI to do it when needed). The benefit of it will be that > it can be done outside of the regular provider release cadence - for > example when we move more Operators to the standard provider, we might need > to push a new .dev version to PyPI so that our PROD image checks will not > fail. > > WDYT? > > > J. >
