Hey all, I have updated our meeting notes document to summarize the discussion from our dev call on 22nd Aug.
Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-22August2024 To all those who attended, can you please double-check and add if I have missed anything? To all those who didn't join, if you disagree with anything in the Summary, please voice your opinion. I will send a separate email with the agenda for the next meeting on 5th September, but if someone has an item for the agenda, please reach out to me or feel free to add it to the doc <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda> . Regards, Kaxil ------ Including the Summary here too (might break formatting): *Catch-up on action items from last call* - AIP-79: Remove Flask AppBuilder as Core dependency <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79%3A+Remove+Flask+AppBuilder+as+Core+dependency> : - Jed made good progress on a working POC – he was able to pull out FAB from the core easily; he is going to verify if he didn't miss anything. He will provide an update on the next call. - Jed , Vincent & Jarek have also been discussing what Auth manager we should have for Core. Vincent's plan is to have a simple auth manager for local installs, and this will be driven via an Airflow config. He will build one for production using Casbin or Keycloak after the Airflow Summit. Currently, his team is doing research on which would be better. - *Action Item*: Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. This can also wait for an in-person chat during the Airflow Summit. - Elad raised a concern about this PR <https://github.com/apache/airflow/pull/41437> that separates FAB migrations from Core. His concern was that this was the first time we would be doing a migration from a provider, so we should think about the messaging to the users via docs (both Airflow & FAB provider) because up until now, we tell users to just upgrade provider dependency without thinking of DB migrations. Ash had an idea to maybe call the FAB thing as a Plugin instead of a provider, we decided to discuss this async. - *Action Item*: Ephraim will send a mailing list thread about how we plan to handle messaging around DB migrations from FAB and if we need to discuss Ash's idea about calling FAB a plugin rather than a provider. - Update on development practices/guide while the new UI is being built (Brent) - Brent proposed 3 guidelines to help committers accept changes to legacy UI until the new UI is ready to be merged. 1. Bugfixes that can be cherry-picked for 2.10.x, 2.11. 1. It is fine to accept these changes 2. Airflow 3.0 features 1. Add minimum changes necessary to unblock dev work without adding a ton of FAB views which will be deleted anyway once new UI is ready 3. Fixes or Updates to existing react views 1. It is fine to accept those changes. The changes will be ported over to the new UI. - Brent proposed that the same rules should apply to non-public API endpoints for UI. - The PR <https://github.com/apache/airflow/pull/41846> to add this new airflow/ui project has been created. - Pierre Jeambrun has joined forces with Brent to work on the UI pieces. - *Action Item*: Brent will write up the guidelines in the docs and send an email with the link to PR on the dev mailing list. - Plan for Decoupling Providers's Connections metadata from FAB (Jens Scheffler) - Jens created this draft PR <https://github.com/apache/airflow/pull/41656> with the POC for it and presented it on the call. - Jarek proposed the idea of dumping the JSON/YAML with connection fields in the Database or loading it via package metadata so we don't load all the dependencies on the webserver. - We will need some plan for external providers on how they can define connections or register them. - The POC successfully proved that we can separate the connection metadata from FAB - *Action Item*: Jens to create a GitHub issue for decoupling the Connection metadata from FAB - Open Telemetry as a StatsD replacement POC Update (Shubham / Dennis) - POC is working fine and the AWS team were able to use Open Telemetry with Cloud Watch. There were some duplicate metrics (based on name, length & limitation) that they will clean up as they decide to move. - Howard Yoo gave a presentation on OTel support in Airflow. Recording details below: - Zoom Recording Link <https://honeycomb.zoom.us/rec/share/73EAoLn9OxnE8loW5phaRjnZxwrvhXTxMXlv-EU9cuapsLLwn-_IY8m1-d8zL-bQ.iQZhZ1QPptQ0zs39> | Passcode: OXzbxW+9 - *Action Item: *This discussion will be continued on the dev mailing list. Kaxil to create a thread on the mailing list about OTel vs StatsD to discuss if we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. *Discussions* - Airflow 3 development guideline <https://github.com/apache/airflow/pull/41457> review / Airflow 2.11: Bridge release (Elad Kalif) - We discussed two open issues: 1. How are we going to manage milestones? There will be two PRs: one against main and one against v2-10-test. In some cases, which milestone should we use on the original PR and the backport PR? 1. We agreed that for now, we will only add milestones to the original PR (targeting the main); for example, if the original PR targets the main but is intended for 2.10.1, it should have a 2.10.1 milestone. 2. How are we backporting/cherry-picking? - We need to remove features from 2.11 GitHub milestone <https://github.com/apache/airflow/milestone/102> - *Action Item*: Kaxil to create a lazy consensus email as a follow-up to this discussion about Airflow 2.11 <https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj> and delete all features from 2.11 GitHub milestone <https://github.com/apache/airflow/milestone/102> - For PRs that just have deprecations changes, we should just merge them to release in the next 2.10 patch release so the users are made aware of it asap so they can start planning. When we have a change that is targeted for the 2.11 bridge release, we will discuss how to handle it since it won't be merged to v2-10-test. Possibly, we would create a v2-11-test branch from v2-10-stable later this year when we are closer to have most of the AIPs on the finishing lines. - More help needed on our CI / infrastructure (Jarek Potiuk ) - Hussein Awala presented the Architecture of the current CI - Shahar Epstein & Buğra Öztürk volunteered to work on CI issues. - New code layout for core/providers/task-SDK (Ash Berlin-Taylor ) - We ran out of time on this one but Ash has started a thread here <https://lists.apache.org/thread/dyv5jhvt65xs6l5o2byc2b67f4wlwf6r> for discussion. *Consolidated Action items list:* - Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. This can also wait for an in-person chat during the Airflow Summit. - Ephraim will send a mailing list thread about how we plan to handle messaging around DB migrations from FAB and if we need to discuss Ash's idea about calling FAB a plugin rather than a provider. - Brent will write up the guidelines in the docs and send an email with the link to PR on the dev mailing list. - Jens to create a GitHub issue for decoupling the Connection metadata from FAB - Kaxil to do some homework on OpenTelemetry support so we can confirm an opinion to keep or drop StatsD support when we make OpenTelemetry the default - Kaxil to create a thread on the mailing list about OTel vs StatsD to discuss if we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. - Kaxil to create a lazy consensus email as a follow-up to this discussion about Airflow 2.11 <https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj> and delete all features from 2.11 GitHub milestone <https://github.com/apache/airflow/milestone/102>
