To be honest ugliness of it and the messiness of supporting many teams is
deliberate choice. This is also to add friction in supporting 'many' teams.
Supporting 'many' teams had never been a goal for multi-team even in the
last incarnation. Due to scaling of scheduler especially - we should not
give the impression that this solution can scale to 'many' teams.
So that choice is pretty deliberate.
pon., 7 lip 2025, 21:27 użytkownik Oliveira, Niko
<oniko...@amazon.com.invalid> napisał:
> Thanks for getting back to me and sharing a concrete example Jarek!
>
> I think overloading the executor field, yet again, with another dimension
> is going to get quite messy. If you have many teams and many executors per
> team you now have colons (serving two purposes), semi-colons and commas in
> one config that will be tremendously long. I think this is a poor user
> experience. Same goes for the executor configurations themselves.
> And for someone who's looked through the code, I honestly and
> respectfully, disagree that this is going to be simpler code to write and
> maintain. I think it's actually cleaner and easier for both the users and
> our code base to implement this with first class support, rather than
> embedding team-ness into configs one-by-one and hard coding that
> parsing/support across the code base in specific locations where those
> particular configs are used. It's messy and does not scale well (it smells
> of executor coupling problem we had years ago, this is going to be
> team-coupling where random bits of our code base are going to have to have
> these hacks instead of a first class solution flowing through more
> transparently).
>
> Cheers,
> Niko
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Monday, July 7, 2025 11:15:16 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
> event driven dags) exists
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> I realized that I owe Niko an explanation of configuration changes. Again -
> following the philosophy above - minimal set of changes to "airflow
> internals". the "minimum" set of changes that will work. I propose the
> change below that has **no** changes to the way how the
> current configuration "shared" feature works - it will change the way
> executors will retrieve their configuration if they are configured
> "per-team" - and we can 100% bank on existing multi-executors.
> I believe that will absolutely minimise the set of changes needed to
> implement multi-team and we will be able to get it "faster" and with "far
> lower risk" of impacting airflow code and say - 3.1 or 3.2 delivery.
>
> Existing multi-executor configuration will be extended to include team
> prefix. The prefix will be separated with ":", entries for different teams
> will be separated with ";"
>
> [core]
>
> executor = team1:KubernetesExecutor,my.custom.module.Executor
> Class;team2:CeleryExecutor
>
> The configuration of executors will also be prefixed with the same team:
>
> [team1:kubernetes_executor]
>
> api_client_retry_configuration = { "total": 3, "backoff_factor": 0.5 }
>
> The environment variables keeping configuration will use ___ (three
> underscores) to replace ":". For example:
>
> AIRFLOW__TEAM1___KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION`
>
>
> J.
>
> On Thu, Jul 3, 2025 at 8:47 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > The direction this one is taking is interesting. If you're really just
> > trying to make the feature barely possible and mostly targeted towards
> > managed providers to implement the rest, then I suppose this hits the
> mark.
> >
> > Well actually by taking the direction I took, it's not "mostly for
> managed
> > providers" - i see it as it is equally, for managed providers and on-prem
> > users, but also, following the open-source spirit, philosophically, I
> think
> > in Airflow, any such change should be done with those things in mind,
> > because we are at the stage where we are already "established' and by
> > innovating on top what we have we have sometimes more to lose than to
> gain
> > - so I feel with "deployment' features we should be very careful to
> > distinguish 'enabling things" vs. 'doing things". My focus with this
> > iteration was to remove all the roadblocks that make it impossible (or
> > extremely difficult) to implement "real" multi-team and separation
> without
> > modifying airflow core. I though "what is the minimal set of features
> that
> > will make it "possible" for someone motivated to deploy a single airflow
> > for multiple teams.
> >
> > * minimise maintenance effort increase
> > * do not "spoil" the "simple case" - we do not want to add features that
> > make "simple" implementation more complex the current `docker run -it
> > apache/airflow standalone` - should be simple and straightforward to run
> > * if there is anything that involves complex deployment, we should not
> aim
> > to make a "turn-key" solution that we will have to support - similarly
> like
> > we do with our configuration parameters, we have 100s knobs to turn, and
> as
> > long as default settings are reasonable and someone "motivated" can
> > configure and fine-tune - this configuration and fine-tuning should be
> left
> > to them - regardless if they are on-prem or managed. And both should be
> > able to do it.
> >
> > I think it's not only smart technically (we support the low-level basic
> > features and when someone puts them together and makes it more of a
> > turn-key solution they are responsible for designing and implementing it
> -
> > so we have less maintenance effort. But also it's good from a simple
> > "open-source business model point of view" - i.e. it's a smart product
> > decision we should make.
> >
> > Why airflow is #18 in OSS rank - of course we have a huge community and
> > people contributing in their free time, completely voluntarily. And we
> > cherish, support and encourage it. But let's be honest - if not all those
> > that make business on top of airflow did not invest literally millions of
> > dollars (in terms of engineering salary, sponsoring Airflow Summit,
> > supporting people like me (some smart stakeholders at least who
> understand
> > the value of it) who can be good "community spirit" - Airflow would have
> > order of magnitude less activity, reach, Airflow 3 would not be simply
> > possible. And this is a good thing that we have those stakeholders that
> are
> > interested and make money by turning Airflow into a "turn-key" solution.
> > This is a fantastic, symbiotic relationship.
> >
> > So - what my thinking is - we should NOT make things that make airflow
> > more turn-key for those complex cases. We should leave it up to those who
> > want to make it and want to charge money for it. This is cool and great
> > that they can do - and we should not do it "for them" - but on the other
> > hand - we should make it possible that those who want to turn airflow
> into
> > more complex (say multi-team solution) to make it happen - by providing
> > them with minimal set of features that make it possible.
> >
> > And that also - in a way - keeps the balance between on-prem and managed
> > implementation.
> >
> > Something that I've learned as a rule of thumb is that making a feature
> > "generic" compared to custom implementation is 3x-10x more expensive
> (both
> > in implementation and maintenance). And it means that if an on-prem user
> > wants to implement something for them (say turn-key multi-team solution
> for
> > their case) it will cost `x` , but when a managed provider wants to
> > implement a generic multi-team it will cost `10x`. But also managed
> > providers can spread the cost over the premium they will charge to their
> > users so that they don't have to manage Airflow on their own and pay `x`
> > for this mult-team feature to develop on their own. And this is a "fair"
> > choice to make by on-prem users. They might choose what they want to do
> > then. Also it's fair for managed provider - yes they need to invest more,
> > but also they have a chance to shine on promoting it and making it more
> > optimised at scale etc. etc.
> >
> > That is my line of thinking.
> >
> >
> > J.
> >
> >
> > On Thu, Jul 3, 2025 at 1:41 AM Oliveira, Niko
> <oniko...@amazon.com.invalid>
> > wrote:
> >
> >> Hey Jarek,
> >>
> >>
> >> The direction this one is taking is interesting. If you're really just
> >> trying to make the feature barely possible and mostly targeted towards
> >> managed providers to implement the rest, then I suppose this hits the
> mark.
> >>
> >> But this is not something we're asking for at Amazon and personally I
> >> think we should make the feature reasonably usable for those running
> >> self-managed OSS Airflow as well. There are many users running an
> on-prem
> >> Airflow. Getting too hyper-fixated on an implementation that's so
> >> simplified that it's obtuse and difficult to use by most users seems
> like
> >> the wrong approach to me. But you and I have already discussed this at
> >> length and I haven't convinced you so far, so if I'm the only one with
> this
> >> thinking then I'm happy to disagree and commit as we say at Amazon :)
> >>
> >>
> >> > So I would be rather strong on **not** touching the current
> >> configuration and
> >>
> >> simply adding configuration for per-team executors in executor config -
> >> even if it is uglier and more "low-level".
> >>
> >> Can you explain what "adding configuration for per-team executors in
> >> executor config" would look like? I don't have a concrete sense of what
> you
> >> mean by this.
> >>
> >> Thanks for your efforts on trying to get this feature agreed to and
> voted
> >> on. Looking forward to working on the project in the coming weeks!
> >>
> >> Cheers,
> >> Niko
> >>
> >> ________________________________
> >> From: Jarek Potiuk <ja...@potiuk.com>
> >> Sent: Tuesday, July 1, 2025 10:26:55 PM
> >> To: dev@airflow.apache.org
> >> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82
> (External
> >> event driven dags) exists
> >>
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and
> know
> >> the content is safe.
> >>
> >>
> >>
> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe.
> >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> >> le contenu ne présente aucun risque.
> >>
> >>
> >>
> >> Any last comments ? There is a long weekend coming up in the US, so I
> will
> >> likely start voting on the updated AIP on Monday 7th.
> >>
> >> On Fri, Jun 27, 2025 at 12:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> > I'd really love to finalise discussion and put it up to a vote some
> time
> >> > after the recording from the last dev call is posted - so that more
> >> > context, details and the LONG discussion we had on it. There is no
> >> *huge*
> >> > hurry - we have strong dependency on Task Isolation and it seems that
> >> it
> >> > will still take a bit of time to complete, so I'd say I would love to
> >> start
> >> > voting in about a week time - so that maybe at the next dev call we
> can
> >> > "seal" the subject. Happy to see any more comments - especially from
> >> those
> >> > who have opinions but they had no opportunity to express them.
> >> >
> >> > I am personally very happy with the direction it took - simplification
> >> and
> >> > "MVP" kind of approach - also I invite the stakeholders of ours to
> take
> >> a
> >> > close look at the scope and what we really propose - I have a feeling
> >> that
> >> > we can balance it out - there is something we can make to make it not
> >> > "worse" for the offerings they have. I think we have a really good
> >> > symbiotic relationship here, and I would love to leverage that. For
> one
> >> -
> >> > my goal here is to have a minimum number of changes that are impacting
> >> > maintainability of the open-source airflow - but mostly "opening up
> some
> >> > possibilities" - rather than provide turn-key solutions. And mostly
> >> because
> >> > this is good for all sides - less maintenance and complexity for OSS
> >> > maintainers, but more opportunities to make it into "turn-key"
> >> solutions by
> >> > the stakeholders, while also allowing the "on-prem" users - if they
> are
> >> > highly motivated - to use those features by adding the "turn-key"
> layer
> >> on
> >> > their own. Also adding multi-team should not be at the expense of
> >> "simple"
> >> > installations - they should be virtually unaffected.
> >> >
> >> > One example of applying this is cutting on "separate config files". I
> >> > think it moves us closer to a "turn-key" solution but it is not really
> >> > necessary to achieve the three goals above - that's why in the current
> >> > proposal this part is completely removed - Sorry Niko, but I still
> think
> >> > it's one of the things that falls into this bucket. We can easily
> remove
> >> > it, they complicate code, documentation and options the users have,
> and
> >> > even if it is a "little" more complex to manage configuration by
> >> motivated
> >> > users, it's also an opportunity for "turn-key" option that
> stakeholders
> >> can
> >> > build in their products - and we do not have to maintain it in the
> >> > open-source. So I would be rather strong on **not** touching the
> current
> >> > configuration and simply adding configuration for per-team executors
> in
> >> > executor config - even if it is uglier and more "low-level".
> >> >
> >> > So if there are some constructive ideas on what can be done to make it
> >> > "simpler" and less "turn-key" in that respect - I would highly value
> >> such
> >> > ideas and comments. If we can cut down something more that is not
> >> > "necessary" for the three primary goals I came up with - I am more
> than
> >> > happy to do it.
> >> >
> >> > Just to remind - those are the "extracted" goals. I slightly updated
> >> them
> >> > and added to the preamble of the AIP:
> >> >
> >> > * less operational overhead for managing multi-team (once AIP-72 is
> >> > complete) where separate execution environments are important
> >> > * virtual assets sharing between teams
> >> > * ability of having "admin" and "team sharing" capability where dags
> >> from
> >> > multiple teams can be seen in a single Airflow UI (requires custom
> >> RBAC an
> >> > AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager
> >> being a
> >> > reference implementation)
> >> >
> >> > J.
> >> >
> >> >
> >> > On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >> >
> >> >>
> >> >>> One technical observation: Now that the dag table no longer has a
> >> >>> team_id in it, what would the behaviour be when a DAG is attempted
> to
> >> move
> >> >>> between bundles? How do we detect this? (I’m not all convinced that
> we
> >> >>> correctly detect duplicate dag ids across bundles today, so I
> wouldn’t
> >> >>> assume or rely on the current behaviour.)
> >> >>>
> >> >>
> >> >> Of course - yes, I realise that - that problem was also not handled
> in
> >> >> the previous iteration to be honest. That is something that dag
> bundle
> >> >> solution allows to solve eventually - but I do not think it's a
> >> blocker for
> >> >> the proposed implementation. We will have to eventually add some way
> of
> >> >> blocking dags to jump between bundles, we might tackle this
> >> separately. I
> >> >> already wanted to propose a separate update to that - but I did not
> >> want to
> >> >> complicate the current proposal. One thing at a time. I can, however
> -
> >> if
> >> >> you consider that as a blocker, extend the current AIP with it. Not a
> >> big
> >> >> problem. This is however a bit independent from the team_id
> >> introduction.
> >> >>
> >> >> Overall, I am still unconvinced this proposal has enough real user
> >> >>> benefit over actually separate deployments, and on balance of the
> >> added
> >> >>> complexity and maintenance burden I do not think it is worth it.
> >> >>>
> >> >>
> >> >> That makes me sad, I thought that over the course of the discussion I
> >> >> addressed all the concerns (in this case the concern was "is it worth
> >> with
> >> >> the cost and little benefit", but when I did it and heavily limited
> the
> >> >> impact, now the concern is "is it worth at all as changes are really
> >> >> minimal" - and surely, anyone can change and adapt their concerns,
> over
> >> >> time, but that one seems like ever-moving target. I hoped at least
> for
> >> >> some acknowledgment of some concerns (complexity in this case) is
> >> >> addressed, but it seems that you are deeply convinced that we do not
> >> need
> >> >> multi-team at all (which is in stark contrast with at least a dozen
> of
> >> >> bigger and smaller users of Airflow who submitted talks to Airflow
> >> summit
> >> >> (including about 5 or 6 submissions for Airflow 2025) on how they
> spent
> >> >> their engineering effort, time and money on trying to achieve
> something
> >> >> similar - they assessed that it's worth, you assess that it's not
> >> worth.
> >> >> Somehow I trust our users that they were not spending the money, time
> >> and
> >> >> engineering effort to achieve this because they wanted to spend more
> >> money.
> >> >> I think they assessed it's worth it. So I want to make it a bit
> easier
> >> and
> >> >> more "proper" way for them to do that.
> >> >>
> >> >>>
> >> >>> Upgrades: it is not easier to upgrade under this multi team
> proposal,
> >> >>> but much much harder. This is based on hard earned experience from
> >> helping
> >> >>> Astronomer users — having to coordinate upgrades between multiple
> >> teams
> >> >>> turns in to a months long slog of the hardest kind of work — people
> >> work:
> >> >>> getting other teams to agree to do things that they don’t directly
> >> care
> >> >>> about — “It’s working for me, I don’t care about upgrading, we’ll
> get
> >> to it
> >> >>> next quarter” is a refrain I’ve heard many times.
> >> >>>
> >> >>
> >> >> Yes. absolutely - this is why we deferred it until we knew what shape
> >> >> task isolation and other AIPs we depend on take on. Because it is
> clear
> >> >> that pretty much all the problem you explain above are going to be
> >> solved
> >> >> with task isolation. And it's not just my opinion. If you want to
> argue
> >> >> with it, you likely need to argue with yourself:
> >> >>
> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478
> >> .
> >> >> Let me quote what you wrote there last week:
> >> >>
> >> >> Ash Berlin Taylor wrote:
> >> >>
> >> >> > A tight coupling between task-sdk and any "server side" component
> is
> >> >> the opposite to one of the goals of AIP-72 (I'm not sure we ever
> >> explicitly
> >> >> said this, but the first point of motivation for the AIP says
> >> "Dependency
> >> >> conflicts for administrators supporting data teams using different
> >> versions
> >> >> of providers, libraries, or python packages")
> >> >> > In short, my goal with TaskSDK, and the reason for introducing
> CalVer
> >> >> and Cadwyn with the execution API is to end up in a world where you
> can
> >> >> upgrade the Airflow Scheduler/API server interdependently of any
> worker
> >> >> nodes (with the exception that the server must be at least as new as
> >> the
> >> >> clients)
> >> >> > This ability to have version-skew is pretty much non-negotiable to
> me
> >> >> and is (other than other languages) one of primary benefits of AIP-72
> >> >>
> >> >> If you read yourself from that quote it basically means "it will be
> >> easy
> >> >> to upgrade airflow independently of workers". So I am a bit confused
> >> here.
> >> >> Yes, I agree it was difficult, but you yourself explain that when
> >> AIP-72
> >> >> (which since API-67 has been accepted has always beem prerequisite of
> >> it)
> >> >> wrote it will be "easy". So I am not sure why you are bringing it
> now.
> >> We
> >> >> assume AIP-72 will be completed and this problem will be gone. Let's
> >> not
> >> >> mention it any more please.
> >> >>
> >> >> The true separation from TaskSDK will likely only land in about 3.2
> >> time
> >> >>> frame. We are actively working on it, but it’s a slow process of
> >> untangling
> >> >>> lots of assumptions made in the code base over the years. Maybe once
> >> we
> >> >>> have that my view would be different, but right now I think this
> >> makes the
> >> >>> proposal a non-starter. Especially as you are saying that most teams
> >> will
> >> >>> have unique connections. If they’ve got those already, then having
> an
> >> asset
> >> >>> trigger use those conns to watch/poll for activity is a much easier
> >> >>> solution to operate and crucially, to scale and upgrade.
> >> >>>
> >> >>
> >> >> Yes. I perfectly understand that and I am fully aware of potentially
> >> 3.2
> >> >> time-frame. And that's fine. Actually I heartily invite you to listen
> >> to
> >> >> the part of my talk from Berlin Buzzwords when I was asked for the
> >> timeline
> >> >> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
> >> >> timeline in my talk . My answer was basically - "3.1" or "3.2", and I
> >> >> sincerely hope "3.1" but we might not be able to complete it because
> we
> >> >> have other things to do (other - is indeed the Task Isolation work
> >> that you
> >> >> are leading). And that's perfectly fine. And it absolutely does not
> >> prevent
> >> >> us from voting on the AIP now - similarly as we voted on the previous
> >> >> version of the AIP - knowing that it has some prerequisites a few
> >> months
> >> >> ago. Especially that we know that the feature we need from task
> >> isolation
> >> >> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we
> >> know it
> >> >> will be there. Those are your own words.
> >> >>
> >> >>
> >> >>> > I think we can’t compare AIP-82 to sharing virtual assets due to
> >> >>> complexity of it.
> >> >>>
> >> >>> Virtual Assets was a mistake, and not how users actually want to use
> >> >>> them. Mea culpa
> >> >>>
> >> >>
> >> >> This is the first time I hear this - certainly you never raised this
> >> >> concern on the devlist. So if you have some concerns about virtual
> >> assets I
> >> >> think you should raise it on the devlist, because I think everyone
> >> here is
> >> >> missing some conversation (or maybe it's just your private opinion
> that
> >> >> you never shared with anyone, but maybe it's worth). I would be
> >> >> interested to hear how the feature that was absolutely most
> successful
> >> >> feature of airflow 2 was a mistake. According to the 2024 survey
> >> >> https://airflow.apache.org/blog/airflow-survey-2024/ - 48% of
> Airflow
> >> >> users have been using it, even if it was added as one of the last
> >> >> big features of Airflow 2. It's the MOST used feature out of all the
> >> >> features out there. I would be really curious to see how it was a
> >> mistake
> >> >> (but please start a separate thread explaining why you think it was
> a
> >> >> mistake, what are your data points and what do you think should be
> >> fixed.
> >> >> Just dropping "virtual assets were a mistake" in the middle of
> >> multi-team
> >> >> conversation seems completely unjustified without knowing what you
> are
> >> >> talking about. So I think, until we know more, this argument has no
> >> base.
> >> >>
> >> >>
> >> >>>
> >> >>> S
> >> >>> To restate my points:
> >> >>>
> >> >>> - Sharing a deployment between teams today/in 3.1 is operationally
> >> more
> >> >>> complex (both scaling, and upgrades) — this is a con, not a plus.
> >> >>>
> >> >>
> >> >> Surely. But it will be easier when AIP-72 is complete (which I am
> >> >> definitely looking forward to and as clearly explained in AIP-82, is
> a
> >> >> prerequisite of it). Nothing changed here.
> >> >>
> >> >>
> >> >>> - The main user benefit appears to be “allow teams’ DAGs to
> >> communicate
> >> >>> via Assets”, in which case we can do that today by putting more work
> >> in to
> >> >>> AIP-82’s Asset triggers
> >> >>>
> >> >>
> >> >> No. Lower operational complexity for multi-teams (providing that we
> >> >> deliver AIP-72) is another benefit. Virtual assets is another, and
> >> since
> >> >> there is no ground in "virtual assets is a mistake" statement (not
> >> until
> >> >> you explain what you mean by that in a separate discussion) - this is
> >> also
> >> >> still a very valid point.
> >> >>
> >> >>
> >> >>> Soon, we will have then be asked about cross-team governance, policy
> >> >>> enforcement, and potentially unbounded edge cases (e.g.,
> team-specific
> >> >>> secrets, roles, quotas). ain, you get this for free with truely
> >> separate
> >> >>> deployments already
> >> >>> allow different teams to use different executors (including multiple
> >> >>> executors per-team following AIP-61)
> >> >>>
> >> >>
> >> >> Not really. We very explicitly say in the AIP that his is not a goal
> >> and
> >> >> that we have no plans for. And yes, using separate executors per team
> >> is
> >> >> actually back in the AIP-82 in case you did not notice (and the code
> >> needed
> >> >> for it's even implemented and merged already in main by Vincent).
> >> >>
> >> >>
> >> >>> Provably not true right now, and until ~3.2 delivers the full Task
> >> >>> SDK/Core dependency separation this would be _more_ work to upgrade,
> >> not
> >> >>> less, and that work is not shared but still on a central team.
> >> >>>
> >> >>
> >> >> Absolutely - we will wait for AIP-72 completion. I do not want to say
> >> 3.1
> >> >> or 3.2 directly - because there are - as you said - a lot of moving
> >> pieces.
> >> >> So my target for multi-team is "After AIP-72 is completed". Full
> stop.
> >> But
> >> >> there is nothing wrong with accepting the AIP now and doing
> preparatory
> >> >> work in parallel. Similarly as there is no way to have a baby in 1
> >> month by
> >> >> 9 women, there is no way adding more effort to task-sdk isolation
> will
> >> >> speed it up - we alredy have not only 3 people (you leading it, Kaxil
> >> and
> >> >> Amog) but also all the help from me and even 10s of different
> >> contributors
> >> >> (for example with the recent db_test cleanup that I took leadership
> >> on) -
> >> >> and there are people who wish to work on adding multi-team features.
> >> Since
> >> >> the design heavily limits impact on airflow codebase and interactions
> >> with
> >> >> task-sdk implementation, there is nothing wrong with starting
> >> >> implementation in parallel either- amazon team is keen to move it
> >> forward -
> >> >> they even already implemented SQS trigger for assets, and we are
> >> working
> >> >> together on FAB removal, Keycloak authentication manager - and they
> >> seem to
> >> >> still have capacity and drive to progress multi-team. So I am not
> sure
> >> if
> >> >> we are trading off something. There is no "if we work on more on task
> >> sdk
> >> >> and drop multi-team things will be faster". Generally in open source
> >> people
> >> >> work in the area where they feel they can provide best value - such
> as
> >> you
> >> >> working on task-sdk, me on CI,dev env, they will deliver more value
> on
> >> >> multi-team
> >> >>
> >> >>
> >> >>>
> >> >>> So please, as succinctly as possible, please tell me what the direct
> >> >>> benefit to users this proposal is over us putting this effort in to
> >> writing
> >> >>> better Asset triggers instead?
> >> >>>
> >> >>
> >> >>
> >> >> * less operational overhead for managing multi-team (once AIP-72 is
> >> >> complete) where separate execution environments are important
> >> >> * virtual assets sharing
> >> >> * ability of having "admin" and "team sharing" capability where dags
> >> from
> >> >> multiple teams can be seen in a single Airflow UI (requires custom
> >> RBAC)
> >> >>
> >> >> None of this can be done via beter asset triggers
> >> >>
> >> >>
> >> >>>
> >> >>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
> >> >>> >
> >> >>> > My counter-points:
> >> >>> >
> >> >>> >
> >> >>> >> 1. Managing a multi team deployment is not materially different
> >> from
> >> >>> >> managing a deployment per team
> >> >>> >>
> >> >>> >
> >> >>> > It's a bit easier - especially when it comes to upgrades
> (especially
> >> >>> in the
> >> >>> > case we are targetting when we are not targetting multi-tenant,
> but
> >> >>> several
> >> >>> > relatively closely cooperating teams with different dependncy
> >> >>> requiremens
> >> >>> > and isolation need.
> >> >>> >
> >> >>> > 2. The database changes were quite wide-reaching
> >> >>> >>
> >> >>> >
> >> >>> > Yes. that is addressed.
> >> >>> >
> >> >>> >
> >> >>> >> 3. I don’t believe the original AIP (again, I haven’t read the
> >> updated
> >> >>> >> proposal or recent messages on the thread. yet) will meet what
> many
> >> >>> users
> >> >>> >> want out of a multiteam solution
> >> >>> >>
> >> >>> >
> >> >>> > I think we will only see when we try. A lot of people thing they
> >> would,
> >> >>> > even if they are warned. I know at least one user (Wealthsimple)
> who
> >> >>> > definitely want to use it and they got a very detailed explanation
> >> of
> >> >>> the
> >> >>> > idea and understand it well. So I am sure that **some** users
> would.
> >> >>> But we
> >> >>> > do not know how many.
> >> >>> >
> >> >>> >
> >> >>> >> To expand on those points a bit more
> >> >>> >>
> >> >>> >> On 1. The only components that are shared are, I think, the
> >> scheduler
> >> >>> and
> >> >>> >> the API server, and it’s arguable if that is actually a good idea
> >> >>> given
> >> >>> >> those are likely to be the most performance sensitive components
> >> >>> anyway.
> >> >>> >>
> >> >>> >> Additionally the fact that the scheduler is a shared component
> >> makes
> >> >>> >> upgrading it almost a non starter as you would likely need
> buy-in,
> >> >>> changes,
> >> >>> >> and testing form ALL teams using it. I’d argue that this is a
> huge
> >> >>> negative
> >> >>> >> until we finish off the version indepence work of AIP-72.
> >> >>> >>
> >> >>> >
> >> >>> > Quite disagree here - especially that our target is that task-sdk
> is
> >> >>> > supposed to provide all isolation that is needed. There should be
> 0
> >> >>> changes
> >> >>> > in the dags needed to upgrade scheduler, api_server, triggerer -
> >> >>> precisely
> >> >>> > because we introduced backwards-compatible task-sdk.
> >> >>> >
> >> >>> > On 3 my complaint is essentially that this doesn’t go nearly far
> >> >>> enough. It
> >> >>> >> doesn’t allow read only views to other teams dags. I don’t think
> it
> >> >>> allows
> >> >>> >> you to be in multiple teams at once. You can’t share a connection
> >> >>> between
> >> >>> >> teams but only allow certain specified dags to access it, but
> would
> >> >>> have to
> >> >>> >> either be globally usable, or duplicated-and-kept-in-sync between
> >> >>> teams. In
> >> >>> >> short I think it fall short of being useful..
> >> >>> >>
> >> >>> >
> >> >>> > Oh absolutely all that is possible (except sharing single
> >> connections
> >> >>> > between multiple teams - which is a very niche use cases and
> >> >>> duplication
> >> >>> > here is perfectly ok as first approximation - and if we need more
> we
> >> >>> can
> >> >>> > add it later).
> >> >>> >
> >> >>> > Auth manager RBAC and access is abstracted away, and the Keyclock
> >> >>> Manager
> >> >>> > implemented by Vincent allows to manage completely independent and
> >> >>> separate
> >> >>> > RBAC based on arguments and resources provided by Airflow. There
> is
> >> >>> nothing
> >> >>> > to prevent the user who configures KeyCloak RBAC to define it in
> the
> >> >>> way:
> >> >>> >
> >> >>> > if group a > allow to read a and write b
> >> >>> > if group b > alllow to write b but not a
> >> >>> >
> >> >>> > and any other combinations. KeyCloak implementation - pretty
> >> advanced
> >> >>> > already - (and design of auth manager) completely abstracts away
> >> both
> >> >>> > authentication and authorization to KeyCloak and KeyCloak has RBAC
> >> >>> > management built in. Also any of the users can write their own -
> >> even
> >> >>> > hard-coded authentication manager to do the same if they do not
> >> want to
> >> >>> > have configurable KeyCloak. Even SimpleAuthManager could be
> >> hard-coded
> >> >>> to
> >> >>> > provide thiose features.
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >> So on the surface, I’m no more in favour of using dag bundle as a
> >> >>> >> replacement for team id as I think most of the above points still
> >> >>> stand.
> >> >>> >>
> >> >>> >
> >> >>> > We disagree here.
> >> >>> >
> >> >>> >>
> >> >>> >> My counter proposal: We do _nothing_ to core airflow. We work on
> >> >>> improving
> >> >>> >> the event-based trigger o fdags (write more triggers for
> read/check
> >> >>> remote
> >> >>> >> Assets etc) so that teams can have 100% isolated deployments but
> >> still
> >> >>> >> trigger dags based on asset events from other teams.
> >> >>> >>
> >> >>> >
> >> >>> > That does not solve any of the other design goals - only allows to
> >> >>> trigger
> >> >>> > assets a bit more easily (but also it's not entirely solved by
> >> AIP-82
> >> >>> > because it does not solve virtual assets - only ones that have
> >> defined
> >> >>> > triggerer and "something" to listen on - which is way more complex
> >> than
> >> >>> > just defining asset in a Dag and using it in another). I think we
> >> can't
> >> >>> > compare AIP-82 to sharing virtual assets due to complexity of it.
> I
> >> >>> > explained it in the doc.
> >> >>> >
> >> >>> >
> >> >>> > I will now go and catch up with the long thread and updated
> proposal
> >> >>> and
> >> >>> >> come back.
> >> >>> >>
> >> >>> >
> >> >>> > Please. I hope the above explaination will help in better
> >> >>> understanding of
> >> >>> > the proposal, because I think you had some assumptions that do not
> >> >>> hold any
> >> >>> > more with the new proposal.
> >> >>> >
> >> >>> > J.
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >> >>> >>>
> >> >>> >>> Just to clarify the relation - I updated the AIP now to refer to
> >> >>> AIP-82
> >> >>> >> and
> >> >>> >>> to explain relation between the "cross-team" and "cross-airflow"
> >> >>> asset
> >> >>> >>> triggering - this is what I added:
> >> >>> >>>
> >> >>> >>> Note that there is a relation between AIP-82 ("External Driven
> >> >>> >> Scheduling")
> >> >>> >>> and this part of the functionality. When you have multiple
> >> instances
> >> >>> of
> >> >>> >>> Airflow, you can use shared datasets - "Physical datasets" -
> that
> >> >>> several
> >> >>> >>> Airflow Instances can use - for example there could be an S3
> >> object
> >> >>> that
> >> >>> >> is
> >> >>> >>> produced by one airflow instance, and consumed by another. That
> >> >>> requires
> >> >>> >>> deferred trigger to monitor for such datasets, and appropriate
> >> >>> >> permissions
> >> >>> >>> to the external dataset, and you could achive similar result to
> >> >>> >> cross-team
> >> >>> >>> dataset triggering (but cross airflow). However the feature of
> >> >>> sharing
> >> >>> >>> datasets between the teams also works for virtual assets, that
> do
> >> not
> >> >>> >> have
> >> >>> >>> physically shared "objects" and trigger that is monitoring for
> >> >>> changes in
> >> >>> >>> such asset.
> >> >>> >>>
> >> >>> >>> J.
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com>
> >> >>> wrote:
> >> >>> >>>
> >> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
> >> >>> reference
> >> >>> >> to
> >> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
> >> >>> through.
> >> >>> >>>>
> >> >>> >>>> Yep. It did not - because I did not think it was needed or even
> >> very
> >> >>> >>>> important after the simplifications. AIP-82 has a different
> >> scope,
> >> >>> >> really.
> >> >>> >>>> It only helps when the Assets are "real" data files which we
> have
> >> >>> >> physical
> >> >>> >>>> triggers for, it's slightly related - sharing datasets between
> >> teams
> >> >>> >>>> (including those that do not require physical files and
> >> triggers) is
> >> >>> >> still
> >> >>> >>>> possible in the design we have now, but it's not (and never
> was)
> >> the
> >> >>> >>>> **only** reason for having multi-team. There always was (and
> >> still
> >> >>> is)
> >> >>> >> the
> >> >>> >>>> possibility of having a common, distinct environments (i.e.
> >> >>> dependencies
> >> >>> >>>> and providers) per team, the possibility of having connections
> >> and
> >> >>> >>>> variables that are only accessible to one team and not the
> other,
> >> >>> and
> >> >>> >>>> isolating workload execution (all that while allowing to manage
> >> >>> multiple
> >> >>> >>>> team and schedule things with single deployment). That did not
> >> >>> change.
> >> >>> >> What
> >> >>> >>>> changed a lot is that it is now way simpler, something that we
> >> can
> >> >>> >>>> implement without heavy changes to the codebase - and give it
> to
> >> our
> >> >>> >> users,
> >> >>> >>>> so that they can assess if this is something they need without
> >> too
> >> >>> much
> >> >>> >>>> risk and effort.
> >> >>> >>>>
> >> >>> >>>> This was - I believe the main concern, that the value we get
> from
> >> >>> it is
> >> >>> >>>> not dramatic, but the required changes are huge. This
> "redesign"
> >> >>> changes
> >> >>> >>>> the equation - the value is still unchanged, but the cost of
> >> >>> >> implementing
> >> >>> >>>> it and impact on the Airflow codebase is much smaller. I still
> >> have
> >> >>> not
> >> >>> >>>> heard back from Ash if my proposal responds to his original
> >> concern
> >> >>> >> though,
> >> >>> >>>> so I am mostly guessing (also based on the positive impact of
> >> >>> others)
> >> >>> >> that
> >> >>> >>>> yes it does. But to be honest I am not sure and I would love to
> >> hear
> >> >>> >> back,
> >> >>> >>>> I decided to update the AIP to reflect it - regardless,
> because I
> >> >>> think
> >> >>> >> the
> >> >>> >>>> simplification I proposed keeps the original goals, but is
> indeed
> >> >>> way
> >> >>> >>>> simpler.
> >> >>> >>>>
> >> >>> >>>>> This is a very difficult thread to catch up on.
> >> >>> >>>>
> >> >>> >>>> Valid point. Let me summarize what is the result:
> >> >>> >>>>
> >> >>> >>>> * I significantly simplified the implementation proposal
> >> comparing
> >> >>> to
> >> >>> >> the
> >> >>> >>>> original version
> >> >>> >>>> * main simplification is very limited impact on existing
> >> database -
> >> >>> >>>> without "ripple effect" that would require us to change a lot
> of
> >> >>> tables,
> >> >>> >>>> including their primary keys, and heavily impact the UI
> >> >>> >>>> * this is now more of an incremental change that can be
> >> implemented
> >> >>> way
> >> >>> >>>> faster and with far less risk
> >> >>> >>>> * updated idea is based on leveraging bundles (already part of
> >> our
> >> >>> data
> >> >>> >>>> model) to map them (many-to-one) to a team - which requires to
> >> just
> >> >>> >> extend
> >> >>> >>>> the data model with bundle mapping and add team_id to
> connections
> >> >>> and
> >> >>> >>>> variables. Those are all needed DB changes.
> >> >>> >>>>
> >> >>> >>>> The AIP is updated - in a one single big change so It should be
> >> >>> easy to
> >> >>> >>>> compare the changes:
> >> >>> >>>>
> >> >>> >>
> >> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
> >> >>> >>>> -> I even named the version appropriately "Simplified
> multi-team
> >> >>> AIP" -
> >> >>> >> you
> >> >>> >>>> can select and compare v.65 with v.66 to see the exact
> >> differences I
> >> >>> >>>> proposed.
> >> >>> >>>>
> >> >>> >>>> I hope it will be helpful to catch up and for those who did not
> >> >>> follow,
> >> >>> >> to
> >> >>> >>>> be able to make up their minds about it.
> >> >>> >>>>
> >> >>> >>>> J.
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
> >> >>> >> <vik...@astronomer.io.invalid>
> >> >>> >>>> wrote:
> >> >>> >>>>
> >> >>> >>>>> This is a very difficult thread to catch up on.
> >> >>> >>>>> I will take a detailed look at the AIP update to try to figure
> >> out
> >> >>> the
> >> >>> >>>>> changes in the proposal.
> >> >>> >>>>>
> >> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
> >> >>> reference
> >> >>> >> to
> >> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
> >> >>> through.
> >> >>> >>>>>
> >> >>> >>>>> Vikram
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
> >> >>> >>>>> gopidesupa...@gmail.com>
> >> >>> >>>>> wrote:
> >> >>> >>>>>
> >> >>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's
> much
> >> >>> more
> >> >>> >> slim
> >> >>> >>>>>> down.
> >> >>> >>>>>>
> >> >>> >>>>>> left a minor comment. :) Overall looking great.
> >> >>> >>>>>>
> >> >>> >>>>>> Pavan
> >> >>> >>>>>>
> >> >>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
> >> >>> >>>>> <j_scheff...@gmx.de.invalid
> >> >>> >>>>>>>
> >> >>> >>>>>> wrote:
> >> >>> >>>>>>
> >> >>> >>>>>>> Thanks for the rework/update of the AIP-72!
> >> >>> >>>>>>>
> >> >>> >>>>>>> Just a few small comments but overall I like it as it is
> much
> >> >>> leaner
> >> >>> >>>>>>> than originally planned and is in a level of complexity that
> >> it
> >> >>> >> really
> >> >>> >>>>>>> seems to be a benefit to close the gap as described.
> >> >>> >>>>>>>
> >> >>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
> >> >>> >>>>>>>> I updated the AIP - including architecture images and
> >> reviewed
> >> >>> it
> >> >>> >>>>>> (again)
> >> >>> >>>>>>>> and corrected any ambiguities and places where it needed to
> >> be
> >> >>> >>>>> changed.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> I think the current state
> >> >>> >>>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>
> >> >>> >>
> >> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >> >>> >>>>>>>> - nicely describes the proposal.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Comparing to the previous one:
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect
> >> on
> >> >>> >>>>> Airflow
> >> >>> >>>>>>>> 2. There is no need to merge configurations and provide
> >> >>> different
> >> >>> >>>>> set
> >> >>> >>>>>> of
> >> >>> >>>>>>>> configs per team - we can add it later but I do not see why
> >> we
> >> >>> need
> >> >>> >>>>> it
> >> >>> >>>>>> in
> >> >>> >>>>>>>> this simplified version
> >> >>> >>>>>>>> 3. We can still configure a different set of executors per
> >> team
> >> >>> -
> >> >>> >>>>> that
> >> >>> >>>>>> is
> >> >>> >>>>>>>> already implemented (we just need to wire it to the bundle
> ->
> >> >>> team
> >> >>> >>>>>>> mapping).
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> I think it will be way simpler and faster to implement this
> >> way
> >> >>> and
> >> >>> >>>>> it
> >> >>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we
> can
> >> >>> give
> >> >>> >>>>> our
> >> >>> >>>>>>>> users so that they can provide feedback.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> J.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <
> >> ja...@potiuk.com>
> >> >>> >>>>> wrote:
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for
> >> >>> being
> >> >>> >>>>>>> receptive
> >> >>> >>>>>>>>>> to feedback! :)
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>> This now becomes quite close to what was proposing
> before,
> >> we
> >> >>> now
> >> >>> >>>>>> again
> >> >>> >>>>>>>>>> have a team ID (which I think is really needed here, glad
> >> to
> >> >>> see
> >> >>> >>>>> it
> >> >>> >>>>>>> back)
> >> >>> >>>>>>>>>> and it will be used for auth management, configuration
> >> >>> >>>>> specification,
> >> >>> >>>>>>> etc
> >> >>> >>>>>>>>>> but will be carried by Bundle instead of the dag model.
> >> Which
> >> >>> as
> >> >>> >>>>> you
> >> >>> >>>>>>> say
> >> >>> >>>>>>>>>> “For that we will need to make sure that both api-server,
> >> >>> >>>>> scheduler
> >> >>> >>>>>> and
> >> >>> >>>>>>>>>> triggerer have access to the "bundle definition" (to
> >> perform
> >> >>> the
> >> >>> >>>>>>> mapping)"
> >> >>> >>>>>>>>>> which honestly doesn’t feel too much different from the
> >> >>> original
> >> >>> >>>>>>> proposal
> >> >>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring
> >> it’s
> >> >>> >>>>>> available
> >> >>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the
> middle
> >> and
> >> >>> >>>>> keep
> >> >>> >>>>>> it
> >> >>> >>>>>>> on
> >> >>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable
> >> location.
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>> I think the big difference is the "ripple effect" that was
> >> >>> >>>>> discussed
> >> >>> >>>>>> in
> >> >>> >>>>>>>>>
> >> >>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> >> >>> >>>>>> (and I
> >> >>> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger
> >> for
> >> >>> the
> >> >>> >>>>>>>>> discussion) so far what we wanted is to extend the primary
> >> key
> >> >>> and
> >> >>> >>>>> it
> >> >>> >>>>>>> would
> >> >>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API,
> UI
> >> >>> etc.
> >> >>> >>>>> ...
> >> >>> >>>>>>>>> However - we already have `bundle_name" and
> "bundle_version"
> >> >>> in the
> >> >>> >>>>>> Dag
> >> >>> >>>>>>>>> model. So I think when we add a separate table where we
> map
> >> the
> >> >>> >>>>> bundle
> >> >>> >>>>>>> to
> >> >>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not
> >> want
> >> >>> to
> >> >>> >>>>>> change
> >> >>> >>>>>>>>> primary key, we do not want to change UI in any way
> (except
> >> >>> >>>>> filtering
> >> >>> >>>>>> of
> >> >>> >>>>>>>>> DAGs available based on your team - but that will be
> >> handled in
> >> >>> >>>>> Auth
> >> >>> >>>>>>>>> Manager and will not impact UI in any way, I think that's
> a
> >> >>> huge
> >> >>> >>>>>>>>> simplification of the implementation, and if we agree to
> it
> >> - i
> >> >>> >>>>> think
> >> >>> >>>>>> it
> >> >>> >>>>>>>>> should speed up the implementation significantly. There
> are
> >> >>> only a
> >> >>> >>>>>>> limited
> >> >>> >>>>>>>>> number of times where you need to look up the team_id - so
> >> >>> having
> >> >>> >>>>> the
> >> >>> >>>>>>>>> bundle -> team mapping in a separate table and having to
> >> look
> >> >>> them
> >> >>> >>>>> up
> >> >>> >>>>>>>>> should not be a problem. And it has much less complexity
> and
> >> >>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
> >> >>> imagine
> >> >>> >>>>> 100s
> >> >>> >>>>>>> or
> >> >>> >>>>>>>>> thousands already written tests that would have to be
> >> adapted
> >> >>> if we
> >> >>> >>>>>>> changed
> >> >>> >>>>>>>>> the primary key - where there will be pretty much zero
> >> impact
> >> >>> on
> >> >>> >>>>>>> existing
> >> >>> >>>>>>>>> tests if we just add bundle -> team lookup table.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>> One other thing I’d point out is that I think including
> >> >>> executors
> >> >>> >>>>> per
> >> >>> >>>>>>>>>> team is a very easy win and quite possible without much
> >> work.
> >> >>> I
> >> >>> >>>>>> already
> >> >>> >>>>>>>>>> have much of the code written. Executors are already
> aware
> >> of
> >> >>> >>>>> Teams
> >> >>> >>>>>>> that
> >> >>> >>>>>>>>>> own them (merged), I have a PR open to have configuration
> >> per
> >> >>> team
> >> >>> >>>>>>> (with a
> >> >>> >>>>>>>>>> quite simple and isolated approach, I believe you
> approved
> >> >>> Jarek).
> >> >>> >>>>>> The
> >> >>> >>>>>>> last
> >> >>> >>>>>>>>>> piece is updating the scheduling logic to route tasks
> from
> >> a
> >> >>> >>>>>> particular
> >> >>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much
> >> work
> >> >>> >>>>> (though
> >> >>> >>>>>> it
> >> >>> >>>>>>>>>> would be easier if the Task models had a column for the
> >> team
> >> >>> they
> >> >>> >>>>>>> belong
> >> >>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to
> get
> >> >>> the
> >> >>> >>>>>> team) I
> >> >>> >>>>>>>>>> have a branch where I was experimenting with this logic
> >> >>> already.
> >> >>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily
> >> need
> >> >>> to
> >> >>> >>>>>> remove
> >> >>> >>>>>>>>>> this piece from the project's scope if it is already
> partly
> >> >>> done
> >> >>> >>>>> and
> >> >>> >>>>>>> not
> >> >>> >>>>>>>>>> too difficult.
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to
> >> just
> >> >>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we
> >> have
> >> >>> it
> >> >>> >>>>> in.
> >> >>> >>>>>>> And
> >> >>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we
> >> can
> >> >>> also
> >> >>> >>>>>>> include
> >> >>> >>>>>>>>> it in the first iteration. What I wanted to avoid really
> >> (from
> >> >>> the
> >> >>> >>>>>>> original
> >> >>> >>>>>>>>> design) - again trying to simplify it, limit the changes,
> >> and
> >> >>> >>>>> speed up
> >> >>> >>>>>>>>> implementation. And there is one "complexity" that I
> wanted
> >> to
> >> >>> >>>>> avoid
> >> >>> >>>>>>>>> specifically - having to have separate , additional
> >> >>> configuration
> >> >>> >>>>> per
> >> >>> >>>>>>> team.
> >> >>> >>>>>>>>> Not only because it complicates already complex
> >> configuration
> >> >>> >>>>> handling
> >> >>> >>>>>>> (I
> >> >>> >>>>>>>>> know we have PR for that) but mostly because if it is not
> >> >>> needed,
> >> >>> >>>>> we
> >> >>> >>>>>> can
> >> >>> >>>>>>>>> simplify documentation and explain to our users easier
> what
> >> >>> they
> >> >>> >>>>> need
> >> >>> >>>>>>> to do
> >> >>> >>>>>>>>> to have their own multi-team setup. And I am quite open to
> >> >>> keeping
> >> >>> >>>>>>>>> multiple-executors if we can avoid complicating
> >> configuration.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> But I think some details of that and whether we really
> need
> >> >>> >>>>> separate
> >> >>> >>>>>>>>> configuration might also come as a result of updating the
> >> AIP
> >> >>> - I
> >> >>> >>>>> am
> >> >>> >>>>>> not
> >> >>> >>>>>>>>> quite sure now if we need it, but we can discuss it when
> we
> >> >>> >>>>> iterate on
> >> >>> >>>>>>> the
> >> >>> >>>>>>>>> AIP.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> J.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>>
> ---------------------------------------------------------------------
> >> >>> >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> >> >>> >>>>>>> For additional commands, e-mail:
> dev-h...@airflow.apache.org
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>
> >> >>> >>>>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> ---------------------------------------------------------------------
> >> >>> >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> >> >>> >> For additional commands, e-mail: dev-h...@airflow.apache.org
> >> >>>
> >> >>>
> >>
> >
>
