Thanks for your inputs. > If we are moving all server components anyway, would it make sense to then define namespaces per server component below airflow.core. directly such that a future separation into API server/scheduler/dag parser can keep a moved structure?
Certainly. We should be aiming to do that with some points Jarek mentioned as well. > Just a technicality, in the api_fastapi folder we already have `core_api` (public + UI) and `execution_api`. This sub structure wouldn't make sense anymore, `airflow.core.api_fastapi.core_api`. (all 3 are core server component I guess) Good call Pierre. Yes, all of them are server components, we could have a structure like this if it makes sense: airflow.core.api -- ui -- public -- execution Or something similar. > * separating dag-processor, api-server, scheduler, trigger - possibly each with different distribution Agreed. > * separating executors, providers, api-server plugins, scheduler plugins into different kind of airflow "extensions" - separate distributions with separate dependencies We should also probably consider separate provider distribution for provider code vs executors in providers. Provider code can solely depend on task sdk but executors will need airflow core(mostly). Did you mean api-server plugins vs worker/task-sdk plugins? > * improving import airflow.* behaviour and rather than implicit initialization of internal components, do explicit initialization in well-thought and defined sequence (avoiding circular deps and removing lazy initialization hacks we have scattered across all of our code) Most certainly. We do some things very eagerly which can be improved with the injection + explicit init pattern we have been following with shared libraries lately. Thanks & Regards, Amogh Desai On Mon, Jan 5, 2026 at 5:27 PM Jarek Potiuk <[email protected]> wrote: > +1 in general - but also +1 on what Kaxil wrote. We should hold our horses > a bit (though they are definitely galloping in my head as well). > > We can almost infinitely continue refactoring - I have quite a few things > on my plate on how we should split it even further: > > * separating dag-processor, api-server, scheduler, trigger - possibly each > with different distribution > * separating executors, providers, api-server plugins, scheduler plugins > into different kind of airflow "extensions" - separate distributions with > separate dependencies > * improving import airflow.* behaviour and rather than implicit > initialization of internal components, do explicit initialization in > well-thought and defined sequence (avoiding circular deps and removing lazy > initialization hacks we have scattered across all of our code) > > But I think we should absolutely (for 3.2) focus on two goals: > > 1) task-sdk is not needed for airflow-core > 2) airflow-core is not needed by providers and task-sdk > > Once we get there, we can focus on the next steps. > > We should definitely **keep** all the long terms goals in our heads when we > implement 1) and 2) - and when we make decisions necessary for 1) and 2) we > should implement them in the right way so that it's easier to implement the > long term goals (for example explicit initialization of shared components > we discussed with Amogh is already needed, solves some of the 1) and 2) > problems nicely - even if we do not do explicit initialization of > everything and our imports are still doing s*tload of things in somewhat > uncontrolled sequence. And that's ok for now. A good example is the > initialisation sequence of core/configuration/providers and basically > injection of provider configuration instead of configuration "knowing" > about providers > https://github.com/apache/airflow/pull/60074#issuecomment-3710123173 -> > all > that is needed to complete the isolation work - and is a step in the right > direction IMHO, but we are not yet implement all of it, just what we find > is necessary why we complete 1) and 2). > > J. > > > > On Mon, Jan 5, 2026 at 11:46 AM Kaxil Naik <[email protected]> wrote: > > > Well — this year 😂 its 2026 already haha. > > > > Happy New Year > > > > On Mon, 5 Jan 2026 at 10:36, Kaxil Naik <[email protected]> wrote: > > > > > We should wait to complete the entire decoupling story imo including > > > AIP-92. Until then there won’t be a any net benefit. So 3.3 would be > the > > > earliest for this — mid next year. > > > > > > On Mon, 5 Jan 2026 at 10:01, Pierre Jeambrun <[email protected]> > > > wrote: > > > > > >> Sounds good to me. > > >> > > >> Just a technicality, in the api_fastapi folder we already have > > `core_api` > > >> (public + UI) and `execution_api`. This sub structure wouldn't make > > sense > > >> anymore, `airflow.core.api_fastapi.core_api`. (all 3 are core server > > >> component I guess) > > >> > > >> On Mon, Jan 5, 2026 at 10:49 AM Aritra Basu <[email protected] > > > > >> wrote: > > >> > > >> > +1 from me for the thought, Agree with Jens on this as well, if > we're > > >> > cleaning up let's also take the opportunity to move them under > > >> components > > >> > too! > > >> > > > >> > -- > > >> > Regards, > > >> > Aritra Basu > > >> > > > >> > On Mon, 5 Jan 2026, 3:04 pm Rahul Vats, <[email protected]> > > wrote: > > >> > > > >> > > +1 to this. Clear namespace boundaries will definitely help folks > > >> > > understand what depends on what. Agree with Jens on organizing by > > >> server > > >> > > component too. > > >> > > > > >> > > Regards, > > >> > > Rahul Vats > > >> > > > > >> > > On Mon, 5 Jan 2026 at 14:02, Jens Scheffler <[email protected]> > > >> wrote: > > >> > > > > >> > > > Hallo Amogh, > > >> > > > > > >> > > > that totally makes sense to get the "house clean" and separate. > > >> > > > > > >> > > > If we are moving all server components anyway, would it make > sense > > >> to > > >> > > > then define namespaces per server component below airflow.core. > > >> > directly > > >> > > > such that a future separation into API server/scheduler/dag > parser > > >> can > > >> > > > keep a moved structure? > > >> > > > > > >> > > > e.g. moving all API server specific stuff to airflow.core.api. > and > > >> all > > >> > > > scheduler specific to airflow.core.scheduler. > > >> > > > > > >> > > > Jens > > >> > > > > > >> > > > On 1/5/26 09:17, Amogh Desai wrote: > > >> > > > > Hello All, > > >> > > > > > > >> > > > > Wishing you all a happy new year and hope you spent good time > > with > > >> > your > > >> > > > > loved ones. > > >> > > > > > > >> > > > > With the ongoing efforts on Airflow Client Server separation, > I > > >> > figured > > >> > > > > that this would be a > > >> > > > > good time to discuss a proposal in a similar vein to those > > >> efforts. > > >> > > > > > > >> > > > > I'd like to propose establishing an `airflow.core.*` module > > >> namespace > > >> > > for > > >> > > > > server-only components > > >> > > > > as part of our AIP-72 client-server separation work. > > >> > > > > > > >> > > > > *What:* > > >> > > > > Move server-specific modules(API, scheduler, migrations, ORMs, > > and > > >> > > more) > > >> > > > > that are specifically > > >> > > > > used by the server components, to a new `airflow.core.*` > > >> namespace. > > >> > > > > > > >> > > > > For example: > > >> > > > > - airflow.models.* → airflow.core.models.* > > >> > > > > - airflow.jobs.* → airflow.core.jobs.* > > >> > > > > - airflow.api_fastapi.* → airflow.core.api.* (or retain > > >> > `api_fastapi`) > > >> > > > > > > >> > > > > Backward compatibility would be maintained for the older paths > > >> with > > >> > > some > > >> > > > > tooling that we already > > >> > > > > have > > >> > > > > < > > >> > > > > > >> > > > > >> > > > >> > > > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/deprecation_tools.py > > >> > > > > > > >> > > > > by > > >> > > > > issuing deprecation warnings and providing users with a clear > > >> > > > > migration path. > > >> > > > > > > >> > > > > *Why:* > > >> > > > > AIP-72 separates airflow into server and client components, > but > > >> today > > >> > > > there > > >> > > > > is no way to tell from > > >> > > > > an import path whether the code is intended to be server only > or > > >> > client > > >> > > > > only, or shared(shared is > > >> > > > > still better). This makes it easy to accidentally couple > > >> components > > >> > > that > > >> > > > > should be independent. > > >> > > > > We have some prek hooks in place to avoid this as much as > > >> possible, > > >> > but > > >> > > > > there is a limit to how much > > >> > > > > we can restrict. > > >> > > > > > > >> > > > > Moving the server code to `airflow.core.*` would make the > > boundary > > >> > much > > >> > > > > clearer. The end goal would > > >> > > > > look like: > > >> > > > > - airflow.core.* = server-only > > >> > > > > - airflow.sdk.* = client-side > > >> > > > > - airflow._shared.* = shared between both > > >> > > > > > > >> > > > > It would also bring in some added benefits: > > >> > > > > 1. Self documenting architecture: the import paths would now > > >> reveal > > >> > > > > dependencies between > > >> > > > > components > > >> > > > > 2. Reduced accidental coupling > > >> > > > > > > >> > > > > We already have two issues in place to track this: > > >> > > > > 1. https://github.com/apache/airflow/issues/51554 > > >> > > > > 2. https://github.com/apache/airflow/issues/51555 > > >> > > > > > > >> > > > > I have deliberately not listed the estimate of efforts here to > > >> agree > > >> > on > > >> > > > the > > >> > > > > approach first. > > >> > > > > I would love to hear what folks think about this approach > until > > >> > > Saturday: > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://www.timeanddate.com/worldclock/fixedtime.html?msg=Voting+ends&iso=20260110T0830&p1=1440 > > >> > > > > and will start a lazy consensus after that. > > >> > > > > > > >> > > > > Thanks & Regards, > > >> > > > > Amogh Desai > > >> > > > > > > >> > > > > > >> > > > > > >> --------------------------------------------------------------------- > > >> > > > To unsubscribe, e-mail: [email protected] > > >> > > > For additional commands, e-mail: [email protected] > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
