100% agree with Hussein. Given our monorepo and existing tools, it's more
important than ever that we split Airflow into as many small, independent
pieces as possible (still within the single monorepo). The fact that these
pieces are super-standard and we have great tooling makes managing them
nearly effortless.
Also, from an installation point of view, you can still put things together
in meta-packages - or install such packages as extra dependencies (`pip
install task-sdk[mypy]` installing `apache-airflow-mypy` as an extra
dependency sounds like a very good idea).
This has traditionally been a trade-off: you either had more standard, more
monolithic pieces, or you needed to invest in tooling, which required
building an additional, specific mental model for building, packaging and
releasing. However, I think this has changed dramatically over the last few
years, and the balance has shifted towards the tuple ("more split," "less
monolith," "keep monorepo").
This offers numerous advantages, ranging from easy code sharing and
dependency separation to stronger security (fewer dependencies to worry
about per component), and greater modularity. This modularity, for example
allows people to focus only on the module they work on, which usually
results in a better architecture.
And to be perfectly honest - if you have 120 distribution in repo, and plan
adding 100 more in the next year, you already have **everything** to manage
pretty much any number of separate distributions (and we do). And even with
Astral being acquired by OpenAI, it does not change much. After several
discussions, I think the ecosystem is prepared. There are already plans for
what to do if things go sideways (from forking to strengthening Hatch or
switching to Pixi—https://github.com/prefix-dev/pixi, for example) or even
Claude-coding your own tooling in some cases.
BTW, And for those who have not yet listened to ->
https://talkpython.fm/episodes/show/540/modern-python-monorepo-with-uv-and-prek
is where we discussed this journey with Amogh. And I think the nice term I
somehow accidentally used there "mini-services" instead of "micro-services"
very nicely describes the boundaries of such "slice and dice" (not too
much, but deep).
J.
On Tue, Mar 24, 2026 at 9:19 PM Hussein Awala <[email protected]> wrote:
> Thanks TP, I considered this as well.
>
> The main concern is dependency weight. The mypy plugins only need mypy
> itself to run — they don't import anything from airflow-core or task-sdk at
> runtime. They just reference Airflow's fully qualified names as strings to
> hook into mypy's type resolution.
>
> If we bundle the plugins into the task SDK, then running mypy checks would
> pull in the full task-sdk (and transitively airflow-core) as dependencies.
> For CI environments that only need type checking — like our own CI in the
> Airflow repo, or users' lightweight linting pipelines — that's a lot of
> unnecessary overhead.
>
> Keeping it as a separate lightweight package means users can just `pip
> install apache-airflow-mypy` in their CI without installing Airflow itself,
> which is exactly how most teams use mypy plugins for framework-specific
> type support (e.g. django-stubs, pydantic mypy plugin, etc.).
>
> Hussein
>
> On Mon, Mar 23, 2026 at 11:52 PM Tzu-ping Chung via dev <
> [email protected]> wrote:
>
> > Would it be a good idea to release the plugins in the sdk instead, and/or
> > under the airflow.sdk namespace? I feel that may be a more consistent
> > interface to users.
> >
> > TP
> >
> > --
> > Sent from my iPhone
> >
> > > On 24 Mar 2026, at 05:10, Jens Scheffler <[email protected]> wrote:
> > >
> > > Like it!
> > >
> > > If somebody has time I also think that the standard "**context"
> > parameters injected by the Task runner in Tasks would benefit from
> special
> > handling such that we can prevent to say:
> > >
> > > @task
> > > def my_task(params: ParamsDict | None = None):
> > > ...
> > >
> > > and mypy accepts
> > >
> > > @task
> > > def my_task(params: ParamsDict):
> > > ...
> > >
> > > (or similar)
> > >
> > > Jens
> > >
> > >> On 23.03.26 21:57, Hussein Awala wrote:
> > >> Hi everyone,
> > >>
> > >> I’d like to propose releasing the existing MyPy plugins as a
> standalone
> > >> package: *apache-airflow-mypy*.
> > >>
> > >>
> > >> *Background*
> > >>
> > >> Apache Airflow currently ships two MyPy plugins (decorators and
> outputs)
> > >> that improve type checking for DAG authors.
> > >>
> > >> At the moment, these plugins live inside the repository but are not
> > >> published to PyPI. As a result, users who want to benefit from type
> > >> checking need to manually copy these plugin files into their own
> > projects.
> > >> This is what several teams are doing today.
> > >>
> > >>
> > >> *Proposal*
> > >>
> > >> Publish *apache-airflow-mypy* as an independently versioned package on
> > >> PyPI, following SemVer.
> > >>
> > >> The package is small and self-contained (two plugins, ~150 lines of
> > code),
> > >> which should make it straightforward to maintain.
> > >>
> > >> PR: https://github.com/apache/airflow/pull/61422
> > >>
> > >> It includes:
> > >>
> > >> - A dedicated package under dev/mypy/ with its own pyproject.toml
> > >> - A Breeze release command: prepare-mypy-distributions
> > >> - Release documentation: dev/README_RELEASE_MYPY.md
> > >> - Tests and Towncrier-based release notes
> > >>
> > >> The name *apache-airflow-mypy* has already been reserved on PyPI
> (thanks
> > >> Jens!).
> > >>
> > >>
> > >> *What the plugins do*
> > >>
> > >> - *outputs plugin*
> > >> Makes operator.output and @task calls return Any instead of
> XComArg,
> > >> allowing DAGs that pass task outputs between tasks to type-check
> > without
> > >> errors.
> > >> - *decorators plugin*
> > >> Adjusts type signatures for certain Google Cloud decorators (e.g.
> > >> fallback_to_default_project_id, etc.).
> > >>
> > >>
> > >> *Usage*
> > >>
> > >> pip install apache-airflow-mypy
> > >>
> > >> Add to mypy.ini:
> > >>
> > >> [mypy]
> > >> plugins =
> > >> airflow_mypy.plugins.decorators
> > >> airflow_mypy.plugins.outputs
> > >>
> > >>
> > >> I’m happy to take ownership of the release process for this package.
> > >>
> > >> Any thoughts or feedback would be very welcome.
> > >>
> > >> Thanks,
> > >> Hussein
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [email protected]
> > > For additional commands, e-mail: [email protected]
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >
>
