Here is the recording of @Hussein Awala <[email protected]> 's presentation on Airflow CI infrastructure: https://youtu.be/Y949SQi7axI
On Fri, 30 Aug 2024 at 15:11, Kaxil Naik <[email protected]> wrote: > Hey all, > > I have updated our meeting notes document to summarize the discussion from our > dev call on 22nd Aug. > > Link: > https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-22August2024 > > To all those who attended, can you please double-check and add if > I have missed anything? > > To all those who didn't join, if you disagree with anything in the > Summary, please voice your opinion. > > I will send a separate email with the agenda for the next meeting on 5th > September, but if someone has an item for the agenda, please reach out to > me or feel free to add it to the doc > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda> > . > > Regards, > Kaxil > > ------ > > Including the Summary here too (might break formatting): > > *Catch-up on action items from last call* > > - AIP-79: Remove Flask AppBuilder as Core dependency > > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79%3A+Remove+Flask+AppBuilder+as+Core+dependency> > : > - Jed made good progress on a working POC – he was able to pull out > FAB from the core easily; he is going to verify if he didn't miss > anything. He will provide an update on the next call. > - Jed , Vincent & Jarek have also been discussing what Auth manager > we should have for Core. Vincent's plan is to have a simple auth manager > for local installs, and this will be driven via an Airflow config. He > will > build one for production using Casbin or Keycloak after the Airflow > Summit. > Currently, his team is doing research on which would be better. > - *Action Item*: Vikram to get back to Vincent or Shubham on why > the Astronomer team didn't use Casbin. This can also wait for an > in-person > chat during the Airflow Summit. > - Elad raised a concern about this PR > <https://github.com/apache/airflow/pull/41437> that separates FAB > migrations from Core. His concern was that this was the first time we > would > be doing a migration from a provider, so we should think about the > messaging to the users via docs (both Airflow & FAB provider) because up > until now, we tell users to just upgrade provider dependency without > thinking of DB migrations. Ash had an idea to maybe call the FAB thing > as a > Plugin instead of a provider, we decided to discuss this async. > - *Action Item*: Ephraim will send a mailing list thread about > how we plan to handle messaging around DB migrations from FAB and if > we > need to discuss Ash's idea about calling FAB a plugin rather than a > provider. > - Update on development practices/guide while the new UI is being > built (Brent) > - Brent proposed 3 guidelines to help committers accept changes to > legacy UI until the new UI is ready to be merged. > 1. Bugfixes that can be cherry-picked for 2.10.x, 2.11. > 1. It is fine to accept these changes > 2. Airflow 3.0 features > 1. Add minimum changes necessary to unblock dev work without > adding a ton of FAB views which will be deleted anyway once new > UI is ready > 3. Fixes or Updates to existing react views > 1. It is fine to accept those changes. The changes will be > ported over to the new UI. > - Brent proposed that the same rules should apply to non-public > API endpoints for UI. > - The PR <https://github.com/apache/airflow/pull/41846> to add this > new airflow/ui project has been created. > - Pierre Jeambrun has joined forces with Brent to work on the UI > pieces. > - *Action Item*: Brent will write up the guidelines in the docs and > send an email with the link to PR on the dev mailing list. > - Plan for Decoupling Providers's Connections metadata from FAB (Jens > Scheffler) > - Jens created this draft PR > <https://github.com/apache/airflow/pull/41656> with the POC for it > and presented it on the call. > - Jarek proposed the idea of dumping the JSON/YAML with connection > fields in the Database or loading it via package metadata so we don't > load > all the dependencies on the webserver. > - We will need some plan for external providers on how they can > define connections or register them. > - The POC successfully proved that we can separate the connection > metadata from FAB > - *Action Item*: Jens to create a GitHub issue for decoupling the > Connection metadata from FAB > - Open Telemetry as a StatsD replacement POC Update (Shubham / Dennis) > - POC is working fine and the AWS team were able to use Open > Telemetry with Cloud Watch. There were some duplicate metrics (based on > name, length & limitation) that they will clean up as they decide to > move. > - Howard Yoo gave a presentation on OTel support in Airflow. > Recording details below: > - Zoom Recording Link > > <https://honeycomb.zoom.us/rec/share/73EAoLn9OxnE8loW5phaRjnZxwrvhXTxMXlv-EU9cuapsLLwn-_IY8m1-d8zL-bQ.iQZhZ1QPptQ0zs39> > | > Passcode: OXzbxW+9 > - *Action Item: *This discussion will be continued on the dev > mailing list. Kaxil to create a thread on the mailing list about OTel vs > StatsD to discuss if we are ready to deprecate or remove StatsD support > or > just make it non-default in favour of OTel. > > > *Discussions* > > - Airflow 3 development guideline > <https://github.com/apache/airflow/pull/41457> review / Airflow 2.11: > Bridge release (Elad Kalif) > - We discussed two open issues: > 1. How are we going to manage milestones? There will be two PRs: > one against main and one against v2-10-test. In some cases, which > milestone > should we use on the original PR and the backport PR? > 1. We agreed that for now, we will only add milestones to the > original PR (targeting the main); for example, if the original PR > targets > the main but is intended for 2.10.1, it should have a 2.10.1 > milestone. > 2. How are we backporting/cherry-picking? > - We need to remove features from 2.11 GitHub milestone > <https://github.com/apache/airflow/milestone/102> > - *Action Item*: Kaxil to create a lazy consensus email as a > follow-up to this discussion about Airflow 2.11 > <https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj> and > delete all features from 2.11 GitHub milestone > <https://github.com/apache/airflow/milestone/102> > - For PRs that just have deprecations changes, we should just merge > them to release in the next 2.10 patch release so the users are made > aware > of it asap so they can start planning. When we have a change that is > targeted for the 2.11 bridge release, we will discuss how to handle it > since it won't be merged to v2-10-test. Possibly, we would create a > v2-11-test branch from v2-10-stable later this year when we are > closer to have most of the AIPs on the finishing lines. > - More help needed on our CI / infrastructure (Jarek Potiuk ) > - Hussein Awala presented the Architecture of the current CI > - Shahar Epstein & Buğra Öztürk volunteered to work on CI issues. > - New code layout for core/providers/task-SDK (Ash Berlin-Taylor ) > - We ran out of time on this one but Ash has started a thread here > <https://lists.apache.org/thread/dyv5jhvt65xs6l5o2byc2b67f4wlwf6r> for > discussion. > > > > *Consolidated Action items list:* > > - Vikram to get back to Vincent or Shubham on why the Astronomer team > didn't use Casbin. This can also wait for an in-person chat during the > Airflow Summit. > - Ephraim will send a mailing list thread about how we plan to handle > messaging around DB migrations from FAB and if we need to discuss Ash's > idea about calling FAB a plugin rather than a provider. > - Brent will write up the guidelines in the docs and send an email > with the link to PR on the dev mailing list. > - Jens to create a GitHub issue for decoupling the Connection metadata > from FAB > - Kaxil to do some homework on OpenTelemetry support so we can confirm > an opinion to keep or drop StatsD support when we make OpenTelemetry the > default > - Kaxil to create a thread on the mailing list about OTel vs StatsD to > discuss if we are ready to deprecate or remove StatsD support or just make > it non-default in favour of OTel. > - Kaxil to create a lazy consensus email as a follow-up to this > discussion about Airflow 2.11 > <https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj> and > delete all features from 2.11 GitHub milestone > <https://github.com/apache/airflow/milestone/102> > >
