The recording of this call was added to Airflow's YouTube channel as an unlisted video as there were a lot of discussions & useful presentations on this call: https://youtu.be/7UdvZNmVPHc
On Wed, 18 Sept 2024 at 03:13, Kaxil Naik <kaxiln...@gmail.com> wrote: > Hey all, > > I have updated our meeting notes document to summarize the discussion from our > dev call on 22nd Aug. > > Link: > https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-5September2024 > > To all those who attended, can you please double-check and add if > I have missed anything? > > To all those who didn't join, if you disagree with anything in the > Summary, please voice your opinion. > > I will send a separate email with the agenda for the next meeting on 19th > September, but if someone has an item for the agenda, please reach out to > me or feel free to add it to the doc > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda> > . > > Regards, > Kaxil > > ------ > > Including the Summary here too (might break formatting): > > *Catch-up on action items from last call* > > - Vikram to get back to Vincent or Shubham on why the Astronomer team > didn't use Casbin. They will discuss it during the Airflow Summit. > - Ephraim created a thread here > <https://lists.apache.org/thread/mskrft4j4jdopgw3v7pwdsg39d37r6p1> to > discuss the messaging around DB migrations from FAB > - Brent <https://cwiki.apache.org/confluence/display/~bbovenzi>wrote > up the guidelines in the docs and sent an email here > <https://lists.apache.org/thread/db69p9fyt34d5tgo8pywpsp8ybjvflry> to > the dev mailing list. > - Jens created a GitHub issue > <https://github.com/apache/airflow/issues/42016> for decoupling the > Connection metadata from FAB and potentially from provider dependencies > - OTel & StatsD > - Dennis has started a thread here about StatsD/OTel metric naming > <https://lists.apache.org/thread/hmywc8od2t3tc36yt0hf4mv01788rz2x>. > - After the summit, Kaxil will need to create a thread on the > mailing list about OTel vs StatsD to discuss whether we are ready to > deprecate or remove StatsD support or just make it non-default in > favour of > OTel. > - Kaxil created a VOTE thread here > <https://lists.apache.org/thread/0d3dlly0mbps8n58hlxmmpvdcv9kx68s> about > Airflow 2.11 > <https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj> being > a bridge release and deleting all features from the 2.11 GitHub > milestone <https://github.com/apache/airflow/milestone/102> > > > *Presentations* > > - Jarek gave us a sneak peek of his Airflow Summit keynote talk on > Security: "Airflow Beach Cleaning" project – Security work > - Pierre presented the new UI Rest API to explain how it works, how to > develop it, and how to migrate from Connexion to Fast API > - Kaxil <https://cwiki.apache.org/confluence/display/~kaxilnaik>presented > the Survey around Debugging Improvements > <https://s.apache.org/airflow-debugging-survey2024> created as part of > this GitHub Issue <https://github.com/apache/airflow/issues/40975> by > Iliya Romm, Omkar & Amogh. > > > *Discussion* about deprecation/removal of Core_operator functions if a > provider package should be cut (Elad, Jens, Jarek, Ash) > > - We covered Airflow Standard Provider > <https://github.com/apache/airflow/pull/41564> & deprecations (Rom > Sharon) as part of it. > - The standard provider will contain "directly usable in DAGs" > Operators, Hooks, Sensors that are currently in Airflow "core". Some of the > base classes ( BaseOperator , Base Hooks, SkipMixins will remain in > the airflow.sdk as the "foundation" for all other operators) > - Standard provider 1.0.0 will contain everything we want to extract > there - which means that the provider will be in "not-ready" state until we > migrate all the "standard" components - and we release it then as 1.0.0. > Before the migration we remove all deprecations that were in those > operators - so that the "standard" provider contains "clean Airflow 3 > versions" of those operators. > - We discussed a few paths of upgrade for users: > 1. The user is on Airflow 2; they fixed the deprecation warning and > then installed the new standard provider. Then, switch the DAGs to use > operators from the Standard provider instead of AF 2. They are now > ready to > upgrade to Airflow 3. > 2. The user on Airflow 2 fixes all the deprecations and upgrades to > Airflow 3; the import paths ( airflow.operators.python ) work due > to dynamic imports that Ash suggested on the mailing list. This will now > throw warnings that a user needs to import PythonOperator (as an > example) > from the standard provider ( airflow.providers.standard ) instead > of Airflow, but it will work as the "standard" provider will be > preinstalled with Airflow. A user will now switch the import paths to > using > one from the standard provider. > - The standard provider 1.0.0 will contain changelogs of breaking > changes from the Airflow core. Where applicable, the relevant news fragment > in the current core will be moved to the Airflow standard provider. The > newsfragments (Updating guide) for Airflow 3 docs will say that the core > operators have been moved to the standard provider and it will have a link > to that Standard provider docs for detailed breaking changes. > - The Airflow 2 to 3 migration process will be documented in the > Airflow docs (similar to this > > <https://airflow.apache.org/docs/apache-airflow/stable/howto/upgrading-from-1-10/index.html> > but > better), with a recommended approach from each of the two approaches listed > above. The news fragments are right now just used for tracking breaking > changes, they will be collected & organized to create a detailed upgrade > process and a list of breaking changes. > - Jarek <https://cwiki.apache.org/confluence/display/~potiuk>has also > started a lazy consensus email thread here > <https://lists.apache.org/thread/1ltpczo121tgk2pzx31bgsk0zj4s5o6x> about > the above. > > > >
