Hey all, I have updated our meeting notes document to summarize the discussion from our dev call on 22nd Aug.
Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-5September2024 To all those who attended, can you please double-check and add if I have missed anything? To all those who didn't join, if you disagree with anything in the Summary, please voice your opinion. I will send a separate email with the agenda for the next meeting on 19th September, but if someone has an item for the agenda, please reach out to me or feel free to add it to the doc <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda> . Regards, Kaxil ------ Including the Summary here too (might break formatting): *Catch-up on action items from last call* - Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. They will discuss it during the Airflow Summit. - Ephraim created a thread here <https://lists.apache.org/thread/mskrft4j4jdopgw3v7pwdsg39d37r6p1> to discuss the messaging around DB migrations from FAB - Brent <https://cwiki.apache.org/confluence/display/~bbovenzi>wrote up the guidelines in the docs and sent an email here <https://lists.apache.org/thread/db69p9fyt34d5tgo8pywpsp8ybjvflry> to the dev mailing list. - Jens created a GitHub issue <https://github.com/apache/airflow/issues/42016> for decoupling the Connection metadata from FAB and potentially from provider dependencies - OTel & StatsD - Dennis has started a thread here about StatsD/OTel metric naming <https://lists.apache.org/thread/hmywc8od2t3tc36yt0hf4mv01788rz2x>. - After the summit, Kaxil will need to create a thread on the mailing list about OTel vs StatsD to discuss whether we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. - Kaxil created a VOTE thread here <https://lists.apache.org/thread/0d3dlly0mbps8n58hlxmmpvdcv9kx68s> about Airflow 2.11 <https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj> being a bridge release and deleting all features from the 2.11 GitHub milestone <https://github.com/apache/airflow/milestone/102> *Presentations* - Jarek gave us a sneak peek of his Airflow Summit keynote talk on Security: "Airflow Beach Cleaning" project – Security work - Pierre presented the new UI Rest API to explain how it works, how to develop it, and how to migrate from Connexion to Fast API - Kaxil <https://cwiki.apache.org/confluence/display/~kaxilnaik>presented the Survey around Debugging Improvements <https://s.apache.org/airflow-debugging-survey2024> created as part of this GitHub Issue <https://github.com/apache/airflow/issues/40975> by Iliya Romm, Omkar & Amogh. *Discussion* about deprecation/removal of Core_operator functions if a provider package should be cut (Elad, Jens, Jarek, Ash) - We covered Airflow Standard Provider <https://github.com/apache/airflow/pull/41564> & deprecations (Rom Sharon) as part of it. - The standard provider will contain "directly usable in DAGs" Operators, Hooks, Sensors that are currently in Airflow "core". Some of the base classes ( BaseOperator , Base Hooks, SkipMixins will remain in the airflow.sdk as the "foundation" for all other operators) - Standard provider 1.0.0 will contain everything we want to extract there - which means that the provider will be in "not-ready" state until we migrate all the "standard" components - and we release it then as 1.0.0. Before the migration we remove all deprecations that were in those operators - so that the "standard" provider contains "clean Airflow 3 versions" of those operators. - We discussed a few paths of upgrade for users: 1. The user is on Airflow 2; they fixed the deprecation warning and then installed the new standard provider. Then, switch the DAGs to use operators from the Standard provider instead of AF 2. They are now ready to upgrade to Airflow 3. 2. The user on Airflow 2 fixes all the deprecations and upgrades to Airflow 3; the import paths ( airflow.operators.python ) work due to dynamic imports that Ash suggested on the mailing list. This will now throw warnings that a user needs to import PythonOperator (as an example) from the standard provider ( airflow.providers.standard ) instead of Airflow, but it will work as the "standard" provider will be preinstalled with Airflow. A user will now switch the import paths to using one from the standard provider. - The standard provider 1.0.0 will contain changelogs of breaking changes from the Airflow core. Where applicable, the relevant news fragment in the current core will be moved to the Airflow standard provider. The newsfragments (Updating guide) for Airflow 3 docs will say that the core operators have been moved to the standard provider and it will have a link to that Standard provider docs for detailed breaking changes. - The Airflow 2 to 3 migration process will be documented in the Airflow docs (similar to this <https://airflow.apache.org/docs/apache-airflow/stable/howto/upgrading-from-1-10/index.html> but better), with a recommended approach from each of the two approaches listed above. The news fragments are right now just used for tracking breaking changes, they will be collected & organized to create a detailed upgrade process and a list of breaking changes. - Jarek <https://cwiki.apache.org/confluence/display/~potiuk>has also started a lazy consensus email thread here <https://lists.apache.org/thread/1ltpczo121tgk2pzx31bgsk0zj4s5o6x> about the above.
