Just to clarify the relation - I updated the AIP now to refer to AIP-82 and
to explain relation between the "cross-team" and "cross-airflow" asset
triggering - this is what I added:
Note that there is a relation between AIP-82 ("External Driven Scheduling")
and this part of the functionality. When you have multiple instances of
Airflow, you can use shared datasets - "Physical datasets" - that several
Airflow Instances can use - for example there could be an S3 object that is
produced by one airflow instance, and consumed by another. That requires
deferred trigger to monitor for such datasets, and appropriate permissions
to the external dataset, and you could achive similar result to cross-team
dataset triggering (but cross airflow). However the feature of sharing
datasets between the teams also works for virtual assets, that do not have
physically shared "objects" and trigger that is monitoring for changes in
such asset.
J.
On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > From a quick glance, the updated AIP didn't seem to have any reference to
> > AIP-82, which surprised me, but will take a more detailed read through.
>
> Yep. It did not - because I did not think it was needed or even very
> important after the simplifications. AIP-82 has a different scope, really.
> It only helps when the Assets are "real" data files which we have physical
> triggers for, it's slightly related - sharing datasets between teams
> (including those that do not require physical files and triggers) is still
> possible in the design we have now, but it's not (and never was) the
> **only** reason for having multi-team. There always was (and still is) the
> possibility of having a common, distinct environments (i.e. dependencies
> and providers) per team, the possibility of having connections and
> variables that are only accessible to one team and not the other, and
> isolating workload execution (all that while allowing to manage multiple
> team and schedule things with single deployment). That did not change. What
> changed a lot is that it is now way simpler, something that we can
> implement without heavy changes to the codebase - and give it to our users,
> so that they can assess if this is something they need without too much
> risk and effort.
>
> This was - I believe the main concern, that the value we get from it is
> not dramatic, but the required changes are huge. This "redesign" changes
> the equation - the value is still unchanged, but the cost of implementing
> it and impact on the Airflow codebase is much smaller. I still have not
> heard back from Ash if my proposal responds to his original concern though,
> so I am mostly guessing (also based on the positive impact of others) that
> yes it does. But to be honest I am not sure and I would love to hear back,
> I decided to update the AIP to reflect it - regardless, because I think the
> simplification I proposed keeps the original goals, but is indeed way
> simpler.
>
> > This is a very difficult thread to catch up on.
>
> Valid point. Let me summarize what is the result:
>
> * I significantly simplified the implementation proposal comparing to the
> original version
> * main simplification is very limited impact on existing database -
> without "ripple effect" that would require us to change a lot of tables,
> including their primary keys, and heavily impact the UI
> * this is now more of an incremental change that can be implemented way
> faster and with far less risk
> * updated idea is based on leveraging bundles (already part of our data
> model) to map them (many-to-one) to a team - which requires to just extend
> the data model with bundle mapping and add team_id to connections and
> variables. Those are all needed DB changes.
>
> The AIP is updated - in a one single big change so It should be easy to
> compare the changes:
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
> -> I even named the version appropriately "Simplified multi-team AIP" - you
> can select and compare v.65 with v.66 to see the exact differences I
> proposed.
>
> I hope it will be helpful to catch up and for those who did not follow, to
> be able to make up their minds about it.
>
> J.
>
>
>
> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka <vik...@astronomer.io.invalid>
> wrote:
>
>> This is a very difficult thread to catch up on.
>> I will take a detailed look at the AIP update to try to figure out the
>> changes in the proposal.
>>
>> From a quick glance, the updated AIP didn't seem to have any reference to
>> AIP-82, which surprised me, but will take a more detailed read through.
>>
>> Vikram
>>
>>
>>
>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>> gopidesupa...@gmail.com>
>> wrote:
>>
>> > Thanks Jarek, that's a great update on this AIP, now it's much more slim
>> > down.
>> >
>> > left a minor comment. :) Overall looking great.
>> >
>> > Pavan
>> >
>> > On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>> <j_scheff...@gmx.de.invalid
>> > >
>> > wrote:
>> >
>> > > Thanks for the rework/update of the AIP-72!
>> > >
>> > > Just a few small comments but overall I like it as it is much leaner
>> > > than originally planned and is in a level of complexity that it really
>> > > seems to be a benefit to close the gap as described.
>> > >
>> > > On 21.06.25 14:52, Jarek Potiuk wrote:
>> > > > I updated the AIP - including architecture images and reviewed it
>> > (again)
>> > > > and corrected any ambiguities and places where it needed to be
>> changed.
>> > > >
>> > > > I think the current state
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>> > > > - nicely describes the proposal.
>> > > >
>> > > > Comparing to the previous one:
>> > > >
>> > > > 1. The DB changes are far less intrusive - no ripple effect on
>> Airflow
>> > > > 2. There is no need to merge configurations and provide different
>> set
>> > of
>> > > > configs per team - we can add it later but I do not see why we need
>> it
>> > in
>> > > > this simplified version
>> > > > 3. We can still configure a different set of executors per team -
>> that
>> > is
>> > > > already implemented (we just need to wire it to the bundle -> team
>> > > mapping).
>> > > >
>> > > > I think it will be way simpler and faster to implement this way and
>> it
>> > > > should serve as MVMT -> Minimum Viable Multi Team that we can give
>> our
>> > > > users so that they can provide feedback.
>> > > >
>> > > > J.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> > > >
>> > > >>
>> > > >>
>> > > >>> I like this iteration a bit more now for sure, thanks for being
>> > > receptive
>> > > >>> to feedback! :)
>> > > >>>
>> > > >>
>> > > >>> This now becomes quite close to what was proposing before, we now
>> > again
>> > > >>> have a team ID (which I think is really needed here, glad to see
>> it
>> > > back)
>> > > >>> and it will be used for auth management, configuration
>> specification,
>> > > etc
>> > > >>> but will be carried by Bundle instead of the dag model. Which as
>> you
>> > > say
>> > > >>> “For that we will need to make sure that both api-server,
>> scheduler
>> > and
>> > > >>> triggerer have access to the "bundle definition" (to perform the
>> > > mapping)"
>> > > >>> which honestly doesn’t feel too much different from the original
>> > > proposal
>> > > >>> we had last week of adding it to Dag table and ensuring it’s
>> > available
>> > > >>> everywhere. but either way I’m happy to meet in the middle and
>> keep
>> > it
>> > > on
>> > > >>> Bundle if everyone else feels that’s a more suitable location.
>> > > >>>
>> > > >> I think the big difference is the "ripple effect" that was
>> discussed
>> > in
>> > > >> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>> > (and I
>> > > >> believe - correct me if I am wrong Ash - important trigger for the
>> > > >> discussion) so far what we wanted is to extend the primary key and
>> it
>> > > would
>> > > >> ripple through all the pieces of Airflow -> models, API, UI etc.
>> ...
>> > > >> However - we already have `bundle_name" and "bundle_version" in the
>> > Dag
>> > > >> model. So I think when we add a separate table where we map the
>> bundle
>> > > to
>> > > >> the team, the "ripple effect" will be almost 0. We do not want to
>> > change
>> > > >> primary key, we do not want to change UI in any way (except
>> filtering
>> > of
>> > > >> DAGs available based on your team - but that will be handled in
>> Auth
>> > > >> Manager and will not impact UI in any way, I think that's a huge
>> > > >> simplification of the implementation, and if we agree to it - i
>> think
>> > it
>> > > >> should speed up the implementation significantly. There are only a
>> > > limited
>> > > >> number of times where you need to look up the team_id - so having
>> the
>> > > >> bundle -> team mapping in a separate table and having to look them
>> up
>> > > >> should not be a problem. And it has much less complexity and
>> > > >> "ripple-effect" through the codebase (for example I could imagine
>> 100s
>> > > or
>> > > >> thousands already written tests that would have to be adapted if we
>> > > changed
>> > > >> the primary key - where there will be pretty much zero impact on
>> > > existing
>> > > >> tests if we just add bundle -> team lookup table.
>> > > >>
>> > > >>
>> > > >>> One other thing I’d point out is that I think including executors
>> per
>> > > >>> team is a very easy win and quite possible without much work. I
>> > already
>> > > >>> have much of the code written. Executors are already aware of
>> Teams
>> > > that
>> > > >>> own them (merged), I have a PR open to have configuration per team
>> > > (with a
>> > > >>> quite simple and isolated approach, I believe you approved Jarek).
>> > The
>> > > last
>> > > >>> piece is updating the scheduling logic to route tasks from a
>> > particular
>> > > >>> Bundle to the correct executor, which shouldn’t be much work
>> (though
>> > it
>> > > >>> would be easier if the Task models had a column for the team they
>> > > belong
>> > > >>> to, rather than having to look up the Dag and Bundle to get the
>> > team) I
>> > > >>> have a branch where I was experimenting with this logic already.
>> > > >>> Any who, long story short, I don’t think we necessarily need to
>> > remove
>> > > >>> this piece from the project's scope if it is already partly done
>> and
>> > > not
>> > > >>> too difficult.
>> > > >>>
>> > > >> Yeah. I hear you here again. Certainly I would not want to just
>> > > >> **remove** it from the code. And, yep I totally forgot we have it
>> in.
>> > > And
>> > > >> if we can make it in, easily (which it seems we can) - we can also
>> > > include
>> > > >> it in the first iteration. What I wanted to avoid really (from the
>> > > original
>> > > >> design) - again trying to simplify it, limit the changes, and
>> speed up
>> > > >> implementation. And there is one "complexity" that I wanted to
>> avoid
>> > > >> specifically - having to have separate , additional configuration
>> per
>> > > team.
>> > > >> Not only because it complicates already complex configuration
>> handling
>> > > (I
>> > > >> know we have PR for that) but mostly because if it is not needed,
>> we
>> > can
>> > > >> simplify documentation and explain to our users easier what they
>> need
>> > > to do
>> > > >> to have their own multi-team setup. And I am quite open to keeping
>> > > >> multiple-executors if we can avoid complicating configuration.
>> > > >>
>> > > >> But I think some details of that and whether we really need
>> separate
>> > > >> configuration might also come as a result of updating the AIP - I
>> am
>> > not
>> > > >> quite sure now if we need it, but we can discuss it when we
>> iterate on
>> > > the
>> > > >> AIP.
>> > > >>
>> > > >> J.
>> > > >>
>> > > >>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> > > For additional commands, e-mail: dev-h...@airflow.apache.org
>> > >
>> > >
>> >
>>
>
