Jarek, Absolutely intend to have you and Elad be co-authors on the proposed solution here.
Jens has done a great job of identifying some of the early parts of the solution as part of the diagrams which we discussed in the last two dev calls. There are a few core elements to this problem and clearly one dimension i.e. Extra Links and Connection Extra Fields were the focus of the discussion earlier this week. I have been thinking about this in the context of making it easier for end users to upgrade Airflow Versions and this is one of the key benefits of AIP-72 and the separation between the Task SDK and Airflow server components. One of the other dimensions which I had been thinking about was the handling of providers for Logging, Secrets, and Notifications. Because of the different dimensions here, it is best to have a write-up. I started a draft proposal which includes the motivation (as I understand it), the different dimensions to be solved, and some early thoughts around the solutions. I will share that with you by early next week. Clearly, both you and Elad are critical in thinking this through and looking forward to working together on this. Best regards, Vikram On Sat, Oct 12, 2024 at 12:26 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Thanks Ash and Jens. I see that we have a doc to write (and Vikram agreed > to do it) - Vikram, Jens - I am happy to join the efforts and have the > first stab and maybe even lead on implementing that part if no-one else > signed up already. I can lead preparation of such a doc or simply be added > as co-author :). > > Just a comment here - I strongly support the split and I think it's very > much worth it to attempt the "extra work" for 3.0 (it does not necessarily > have to be done for 3.0, but if we can bundle it with everything else in > 3.0 it will be easier to introduce as part of the "3.0 breaking changes". > And a bit of consequence of that (which I will add in the doc) - is the > need to publish more variants of reference images for our users to use (but > this is all totally doable and automatable). > > The main reason (and why it's good I might want to take a lead) - it very > much falls into my main focus area. Such change significantly improves the > overall security properties of Airflow installation. This was one of the > learnings so far from the "Airflow Beach Cleaning" exercise - that > generally supply chain is the next "frontier" for Airflow (and generally > any other project) - and if we could decrease the number of dependencies > for some of the sensitive sub-components of Airflow (which > internal-api/webserver certainly is) - the security risks connected with > supply chain problems are generally somewhere between quadratically and > exponentially depend on the number of dependencies of those components. > > J. > > On Sat, Oct 12, 2024 at 7:07 PM Ash Berlin-Taylor <a...@apache.org> wrote: > > > Nothing much more was covered about splitting packages this time, I think > > it happened in the previous meeting, so that might be the best one to > watch > > back. All we covered on Thursday was talking about how not installing > > providers in the web server would need to work (i.e. ExtraLinks and > > connection field strcuture/metadata to display the Add Connection form > > would have to be persisted in the DB most likely) > > > > > > -ash > > > > > > > On 12 Oct 2024, at 07:50, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > Is there a recording available for that dev call? I would very much > like > > to > > > hear especially the discussion about splitting packages as I think this > > is > > > quite an important topic to discuss and I have quite a bit of > experience > > > there that might be useful to comment on :). > > > > > > I could not attend the meeting because I was at Community Over Code > (and > > > actually that was exactly conflicting with "State of The Foundation" > talk > > > that Dave Nalley, the president of the ASF opened the last day of the > > > conference with - so I had to skip it ). > > > > > > BTW. Dave mentioned Airflow, security, and particularly pointed at and > > > mentioned me personally - underlying the efforts and trust we built > > between > > > us personally while working on the security initiatives - during the > > > opening keynote :D, so it's good I was there. > > > > > > J. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
