+1 in general - but also +1 on what Kaxil wrote. We should hold our horses a bit (though they are definitely galloping in my head as well).
We can almost infinitely continue refactoring - I have quite a few things on my plate on how we should split it even further: * separating dag-processor, api-server, scheduler, trigger - possibly each with different distribution * separating executors, providers, api-server plugins, scheduler plugins into different kind of airflow "extensions" - separate distributions with separate dependencies * improving import airflow.* behaviour and rather than implicit initialization of internal components, do explicit initialization in well-thought and defined sequence (avoiding circular deps and removing lazy initialization hacks we have scattered across all of our code) But I think we should absolutely (for 3.2) focus on two goals: 1) task-sdk is not needed for airflow-core 2) airflow-core is not needed by providers and task-sdk Once we get there, we can focus on the next steps. We should definitely **keep** all the long terms goals in our heads when we implement 1) and 2) - and when we make decisions necessary for 1) and 2) we should implement them in the right way so that it's easier to implement the long term goals (for example explicit initialization of shared components we discussed with Amogh is already needed, solves some of the 1) and 2) problems nicely - even if we do not do explicit initialization of everything and our imports are still doing s*tload of things in somewhat uncontrolled sequence. And that's ok for now. A good example is the initialisation sequence of core/configuration/providers and basically injection of provider configuration instead of configuration "knowing" about providers https://github.com/apache/airflow/pull/60074#issuecomment-3710123173 -> all that is needed to complete the isolation work - and is a step in the right direction IMHO, but we are not yet implement all of it, just what we find is necessary why we complete 1) and 2). J. On Mon, Jan 5, 2026 at 11:46 AM Kaxil Naik <[email protected]> wrote: > Well — this year 😂 its 2026 already haha. > > Happy New Year > > On Mon, 5 Jan 2026 at 10:36, Kaxil Naik <[email protected]> wrote: > > > We should wait to complete the entire decoupling story imo including > > AIP-92. Until then there won’t be a any net benefit. So 3.3 would be the > > earliest for this — mid next year. > > > > On Mon, 5 Jan 2026 at 10:01, Pierre Jeambrun <[email protected]> > > wrote: > > > >> Sounds good to me. > >> > >> Just a technicality, in the api_fastapi folder we already have > `core_api` > >> (public + UI) and `execution_api`. This sub structure wouldn't make > sense > >> anymore, `airflow.core.api_fastapi.core_api`. (all 3 are core server > >> component I guess) > >> > >> On Mon, Jan 5, 2026 at 10:49 AM Aritra Basu <[email protected]> > >> wrote: > >> > >> > +1 from me for the thought, Agree with Jens on this as well, if we're > >> > cleaning up let's also take the opportunity to move them under > >> components > >> > too! > >> > > >> > -- > >> > Regards, > >> > Aritra Basu > >> > > >> > On Mon, 5 Jan 2026, 3:04 pm Rahul Vats, <[email protected]> > wrote: > >> > > >> > > +1 to this. Clear namespace boundaries will definitely help folks > >> > > understand what depends on what. Agree with Jens on organizing by > >> server > >> > > component too. > >> > > > >> > > Regards, > >> > > Rahul Vats > >> > > > >> > > On Mon, 5 Jan 2026 at 14:02, Jens Scheffler <[email protected]> > >> wrote: > >> > > > >> > > > Hallo Amogh, > >> > > > > >> > > > that totally makes sense to get the "house clean" and separate. > >> > > > > >> > > > If we are moving all server components anyway, would it make sense > >> to > >> > > > then define namespaces per server component below airflow.core. > >> > directly > >> > > > such that a future separation into API server/scheduler/dag parser > >> can > >> > > > keep a moved structure? > >> > > > > >> > > > e.g. moving all API server specific stuff to airflow.core.api. and > >> all > >> > > > scheduler specific to airflow.core.scheduler. > >> > > > > >> > > > Jens > >> > > > > >> > > > On 1/5/26 09:17, Amogh Desai wrote: > >> > > > > Hello All, > >> > > > > > >> > > > > Wishing you all a happy new year and hope you spent good time > with > >> > your > >> > > > > loved ones. > >> > > > > > >> > > > > With the ongoing efforts on Airflow Client Server separation, I > >> > figured > >> > > > > that this would be a > >> > > > > good time to discuss a proposal in a similar vein to those > >> efforts. > >> > > > > > >> > > > > I'd like to propose establishing an `airflow.core.*` module > >> namespace > >> > > for > >> > > > > server-only components > >> > > > > as part of our AIP-72 client-server separation work. > >> > > > > > >> > > > > *What:* > >> > > > > Move server-specific modules(API, scheduler, migrations, ORMs, > and > >> > > more) > >> > > > > that are specifically > >> > > > > used by the server components, to a new `airflow.core.*` > >> namespace. > >> > > > > > >> > > > > For example: > >> > > > > - airflow.models.* → airflow.core.models.* > >> > > > > - airflow.jobs.* → airflow.core.jobs.* > >> > > > > - airflow.api_fastapi.* → airflow.core.api.* (or retain > >> > `api_fastapi`) > >> > > > > > >> > > > > Backward compatibility would be maintained for the older paths > >> with > >> > > some > >> > > > > tooling that we already > >> > > > > have > >> > > > > < > >> > > > > >> > > > >> > > >> > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/deprecation_tools.py > >> > > > > > >> > > > > by > >> > > > > issuing deprecation warnings and providing users with a clear > >> > > > > migration path. > >> > > > > > >> > > > > *Why:* > >> > > > > AIP-72 separates airflow into server and client components, but > >> today > >> > > > there > >> > > > > is no way to tell from > >> > > > > an import path whether the code is intended to be server only or > >> > client > >> > > > > only, or shared(shared is > >> > > > > still better). This makes it easy to accidentally couple > >> components > >> > > that > >> > > > > should be independent. > >> > > > > We have some prek hooks in place to avoid this as much as > >> possible, > >> > but > >> > > > > there is a limit to how much > >> > > > > we can restrict. > >> > > > > > >> > > > > Moving the server code to `airflow.core.*` would make the > boundary > >> > much > >> > > > > clearer. The end goal would > >> > > > > look like: > >> > > > > - airflow.core.* = server-only > >> > > > > - airflow.sdk.* = client-side > >> > > > > - airflow._shared.* = shared between both > >> > > > > > >> > > > > It would also bring in some added benefits: > >> > > > > 1. Self documenting architecture: the import paths would now > >> reveal > >> > > > > dependencies between > >> > > > > components > >> > > > > 2. Reduced accidental coupling > >> > > > > > >> > > > > We already have two issues in place to track this: > >> > > > > 1. https://github.com/apache/airflow/issues/51554 > >> > > > > 2. https://github.com/apache/airflow/issues/51555 > >> > > > > > >> > > > > I have deliberately not listed the estimate of efforts here to > >> agree > >> > on > >> > > > the > >> > > > > approach first. > >> > > > > I would love to hear what folks think about this approach until > >> > > Saturday: > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.timeanddate.com/worldclock/fixedtime.html?msg=Voting+ends&iso=20260110T0830&p1=1440 > >> > > > > and will start a lazy consensus after that. > >> > > > > > >> > > > > Thanks & Regards, > >> > > > > Amogh Desai > >> > > > > > >> > > > > >> > > > > >> --------------------------------------------------------------------- > >> > > > To unsubscribe, e-mail: [email protected] > >> > > > For additional commands, e-mail: [email protected] > >> > > > > >> > > > > >> > > > >> > > >> > > >
