[PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

Berlin-Taylor and Vikram Koka

Re: [PROPOSAL][AIP-32] Airflow REST API

In response to the point raised by Jarek, i.e. 'API first' or 'Code first', looking at the document, one of the stated goals is: "The API will be intended for use by any third party. It should not be related to a specific application, e.g. a React UI" In my opinion, the 'Code first' approach on

Re: Language matters - don't use the term "whitelist" in Apache Airflow

Totally appreciate you doing this Ash, Love it! On Tue, Jun 9, 2020 at 11:43 PM Tao Feng wrote: > +1 > > On Mon, Jun 8, 2020 at 1:55 AM Soma S Dhavala > wrote: > > > +1 👍 > > > > On Mon, Jun 8, 2020 at 2:05 PM Kaxil Naik wrote: > > > > > 👏 > > > > > > On Mon, Jun 8, 2020, 08:28 Robin Edwards

Re: [VOTE] AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor

+1 (non-binding) for this AIP. I really like the concept and the efficiency improvements. The general SmartSensor concept and the ability to add additional sensor classes is elegant. >From an implementation perspective, my one area of concern is the "sharding" concept and the configuration / mana

[PROPOSAL][AIP-36 DAG Versioning]

y feedback, suggestions, and comments would be greatly appreciated. Best Regards, Kaxil Naik, Ryan Hamilton, Ash Berlin-Taylor, and Vikram Koka

Re: [Meeting Notes] Airflow 2.0 Dev Call #1 - Planning

Thank you Kaxil, this looks great. I just updated the list of attendees to include a couple of people who I had noticed in the meeting. On Tue, Aug 11, 2020 at 5:17 PM Kaxil Naik wrote: > Hi all, > > I have created a document to summarize the discussion from the first Dev > call for Airflow 2.0

Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020

Jarek, Thank you, this is very helpful. I assume that you would like comments in the document itself? Or, would you like them in email? Best regards, Vikram On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk wrote: > As promised during the last call I prepared the proposal on how we can > approach

Re: [Meeting Notes] Airflow 2.0 Dev Call #3 - 7 Sep 2020

Thanks Kaxil, this looks right to me as well. I updated the main Airflow 2.0 planning page as well to reflect the current scope and milestones based on this meeting. *Doc Link*: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning I also wanted to thank Kevin from the AirBnb

Re: Import style in Airflow codebase

+1 on absolute import. Honestly, a huge fan of doing it as an absolute vs. relative. On Wed, Sep 9, 2020 at 4:09 AM Kaxil Naik wrote: > No strong opinion but absolute import seems better from a user's > perspective. > > On Wed, Sep 9, 2020 at 12:07 PM Jarek Potiuk > wrote: > > > +1! > > > > On

Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

Jarek, It has been our intention (as Astronomer) to release the Scheduler HA work directly to open source as part of Airflow 2.0. We realized early on that the Scheduler reliability and performance were highlighted as the key issues from the community as part of the latest survey results from las

Re: Consider using stale bot for issues

I agree. Paola and Elad, I would like to help on this one as well. Let's get together and nail this. I spent a couple of hours looking through the open Github issues earlier today to look for some patterns. Currently, we have 576 open and "not invalid" issues. Out of these, we have around: - 270

Re: Consider using stale bot for issues

> Anyone else willing to join our Triage squad :) ? > > J. > > > On Sun, Sep 13, 2020 at 2:31 AM Vikram Koka wrote: > > > I agree. > > > > Paola and Elad, I would like to help on this one as well. Let's get > > together and nail this. > >

Re: [VOTE] AIP-8 Split Providers into Separate Packages for Airflow 2.0

+1 non-binding On Sun, Sep 13, 2020 at 2:20 PM Kaxil Naik wrote: > +1 binding > > On Sun, Sep 13, 2020 at 10:18 PM Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > +1 (binding). > > > > via Newton Mail > > [ > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&sourc

Re: Consider using stale bot for issues

ogle Container DevEx team's experience > > contributing > > > > to popular open-source projects, such as minikube, Skaffold, and > > Kaniko. Is > > > > this something we should consider? > > > > > > > > How do other projects handl

Re: Airflow Pluggable Scheduler

Hi Ping, Conceptually, I have a similar reaction to Jarek and Tomek above, but I really want to understand the problem you have described with (2) before I comment further. Can you please elaborate on the problems: Airflow 2.0 treats all DagRuns with the same scheduling priority (see code

Re: [VOTE] August 2022 PR of the Month

If I can vote twice it would be for: 25610 and 25888. Hard to choose between them, they are both really good and critical On Wed, Aug 31, 2022 at 10:00 AM Jeambrun Pierre wrote: > Hello, > > I really love the new graph view for the datasets. > > My vote goes to 25707. > > Best, > Pierre > > On

Re: [VOTE] Release Airflow 2.4.0 from 2.4.0rc1

+1 Tested the Data Dependency Management and Data Driven Scheduling (defined in AIP-48), with Datasets created and configured as dependencies between DAGs. Tested the DAG triggering using these. Also tested the new Datasets View and the updated Dag Dependencies View. This looks amazing and so exc

Re: [ANNOUNCE] New PMC member: Tzu-ping Chung

Congratulations TP! Very well deserved! Vikram On Thu, Sep 22, 2022 at 12:55 AM Jarek Potiuk wrote: > Hello Airflow Community, > > I have the pleasure to announce that The Project Management Committee (PMC) > for Apache Airflow has invited Tzu-ping Chung to become Apache Airflow PMC > Member

Re: [ANNOUNCE] New PMC member: Daniel Standish

Congratulations Daniel! Very well deserved! On Thu, Sep 22, 2022 at 12:57 AM Jarek Potiuk wrote: > Hello Airflow Community, > > I have the pleasure to announce that The Project Management Committee (PMC) > for Apache Airflow has invited Daniel Standish to become Apache Airflow > PMC Member and w

Re: [ANNOUNCE] New Committer: Pierre Jeaambrun

Congratulations Pierre! Very well deserved On Thu, Sep 22, 2022 at 8:35 AM Josh Fell wrote: > Here here! Congrats Pierre! > > On Thu, Sep 22, 2022 at 8:10 AM Brent Bovenzi > wrote: > >> Congratulations Pierre! Very much well deserved! >> >> On Thu, Sep 22, 2022 at 3:51 AM Jarek Potiuk wrote:

Re: Apache Airflow Newsletter | September 2022

+1 to the comments. Very useful digest. I really like this edition of the newsletter! Good work! On Wed, Oct 5, 2022 at 6:30 AM Michael Robinson wrote: > Thank you so much and glad you are liking the newsletter! FYI: I’ve been > shipping it lately, but producing it is a team effort with Ross T

Re: 【New provider】Inquire

David, Great to hear this interest regarding integration with Airflow and happy to help guide as well. Similar to XD, I would very much like to understand both the user base sizing and areas of interest from an integration standpoint. >From your perspective, at the risk of repeating both XD and

Re: [ANNOUNCE] New committer Niko Oliveira (o-nikolas)

Congratulations Niko! On Tue, Dec 20, 2022 at 10:31 AM Andrey Anshin wrote: > Congrats Niko! 👏 🔥 > > > Best Wishes > *Andrey Anshin* > > > > On Tue, 20 Dec 2022 at 19:01, Beck, Vincent > wrote: > >> Congrats Niko!! 😊 >> >> >> >> *From: *Pankaj Singh >> *Reply-To: *"dev@airflow.apache.org

Re: [VOTE] AIP-52 Automatic setup and teardown tasks

+1 binding Vikram On Mon, Jan 9, 2023 at 11:23 AM Ping Zhang wrote: > +1 binding > > Thanks, > > Ping > > > On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi < > ephraimanier...@gmail.com> wrote: > >> +1 binding >> >> On Mon, 9 Jan 2023 at 19:55, Frank Cash wrote: >> >>> +1 (non-binding) >>> >

Re: [VOTE] AIP-53 OpenLineage in Airflow

+1 binding. I have been looking at the doc and having lineage integrated with Airflow as a provider makes sense to me. On Mon, Feb 13, 2023 at 2:38 AM Kaxil Naik wrote: > +1 binding , this should make lineage a first-class citizen for Airflow > users. Excited for this one > > On Sun, 12 Feb 202

Re: Seeking Feedback for Airflow Multi-Tenant Model Proposal

Shubham and Vincent, Let me start by saying that I apologize for my delayed response to your original email. I appreciate the detailed write-up and the thought behind it. I completely agree with your use case and understand how this is applicable to enterprises with multiple data teams using Airf

Re: [VOTE] Move K8S / Celery (and related) executors to respective providers

+1 binding On Tue, Feb 21, 2023 at 12:04 PM Mehta, Shubham wrote: > + 1 non-binding > > > > *From: *Elad Kalif > *Reply-To: *"dev@airflow.apache.org" > *Date: *Tuesday, February 21, 2023 at 10:44 AM > *To: *"dev@airflow.apache.org" > *Subject: *RE: [EXTERNAL][VOTE] Move K8S / Celery (and rela

Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds

Thanks Jarek, for resolving this quickly. I agree that it is important to figure out a plan and next steps here before the next release for sure. Will sync and get back. Vikram On Thu, Feb 23, 2023 at 6:28 AM Jarek Potiuk wrote: > The temporary fix worked - for now. The < 1.10.15 docs are no

Re: [VOTE] February PR of the Month

+1 to #27758 as well, people have really wanted that! On Fri, Feb 24, 2023 at 3:20 PM Pierre Jeambrun wrote: > +1 to #27758, this is really cool! > > Le ven. 24 févr. 2023 à 23:55, Jed Cunningham > a écrit : > >> +1 to #27758 >> >

Re: [ANNOUNCE] New PMC member: Brent Bovenzi

Awesome! Congratulations Brent, well deserved! On Wed, Mar 15, 2023 at 12:10 PM Pierre Jeambrun wrote: > Congratulations Brent, well deserved :) > > Le mer. 15 mars 2023 à 18:55, Oliveira, Niko > a écrit : > > > Congrats Brent!! > > > > > > From: Jorrick Sleijs

Re: [ANNOUNCE] New PMC member: Pierre Jeambrun

Excellent! Congratulations Pierre, very well deserved! On Wed, Mar 15, 2023 at 10:55 AM Oliveira, Niko wrote: > Congrats Pierre, well deserved! > > > From: Kaxil Naik > Sent: Wednesday, March 15, 2023 2:47:31 AM > To: dev@airflow.apache.org > Subject: RE: [EXTER

Re: [VOTE] March PR of the Month

What a great set of PRs to choose from. Amazing work everyone! So hard to choose, but my vote goes for 29413 - looks lovely! On Fri, Mar 31, 2023 at 9:52 AM Kaxil Naik wrote: > One more for 29413 > > On Fri, 31 Mar 2023 at 16:15, Jed Cunningham > wrote: > > > Another for 29413. > > >

Re: [VOTE] Release Airflow 2.6.0 from 2.6.0rc4

+1 (non-binding) On Fri, Apr 28, 2023 at 6:37 PM Kaxil Naik wrote: > +1 binding > > On Fri, 28 Apr 2023 at 14:54, Jed Cunningham > wrote: > > > +1 (binding) > > > > Checked signatures, checksums, licences. Used it with the helm chart > with a > > few different configs > > >

Re: [ANNOUNCE] New committer: Pankaj Singh

Awesome! Congratulations Pankaj Singh! On Mon, Jun 12, 2023 at 3:37 PM Oliveira, Niko wrote: > Woo! Congrats Prankaj! > > > From: Ankit Chaurasia > Sent: Monday, June 12, 2023 3:12:42 PM > To: dev@airflow.apache.org > Subject: RE: [EXTERNAL][ANNOUNCE] New commi

Re: [ANNOUNCE] New committers: Vincent Beck, Phani Kumar, Maciej Obuchowski

Congratulations to all of you! Great work and very well deserved! On Wed, Jun 28, 2023 at 1:43 PM Pierre Jeambrun wrote: > Great news!!! Congratulations 🎉 > > On Wed 28 Jun 2023 at 18:40, Ankit Chaurasia wrote: > > > Congrats > > > > > > > > > > > > > > > > On Wed, Jun 28, 2023 at 10:09 PM utka

Re: [VOTE] June 2023 PR of the Month

+1 for #31123. Great work @pierrejeambrun On Thu, Jun 29, 2023 at 8:33 AM Wei Lee wrote: > +1 for #31123 🙂 > > > On Jun 29, 2023, at 11:14 PM, Phani Kumar > > > wrote: > > > > +1 for #31123 > > > > On Thu, 29 Jun 2023, 20:28 Kaxil Naik, wrote: > > > >> +1 for #31123 👏 > >> > >> On Thu, 29 J

Re: Support for Datasets with execution-time values for use cases requiring more fine-grained dataset specs

Hi Jeff, Thank you for bringing this up. This is definitely on my radar and part of a larger AIP which I have been in the process of writing up. We have thought of this use case and deliberately deferred it in the earlier AIP. Doing is definitely quite complex and I think it needs a couple of int

Re: [VOTE] The daskexecutor, cncf.provider, celery providers in 2.7.0. as regular not-preinstalled providers

I am a little torn on this, but with the separation already in place, I would also vote for option (a). On Mon, Jul 31, 2023 at 1:48 PM Ferruzzi, Dennis wrote: > Pretty unanimous so far, but I'll also say [a]. If the intent is to be > executor/service agnostic then this makes the most sense to

Re: [VOTE] Airflow Providers prepared on July 29, 2023

Not trying to hold up the release, but I thought there was a "race condition" bug discovered with the Celery executor as a provider and Airflow main. Is that resolved now? Or, did I mistake the origin of that? On Mon, Jul 31, 2023 at 8:19 AM Beck, Vincent wrote: > +1 (non-binding). I tested the

Re: [VOTE] Airflow Providers prepared on July 29, 2023

7c7f5daeab5f74fb3224a4d5e33cec95d_8.4.0rc1.html > > > > > > > > I would still like to do some more testing around executors. > > > > > > > > P.S. > > > > I think the Celery issue Vikram is mentioning is this one: > > https://github.c

Re: [VOTE] Restore dag_run.conf UI triggering option for 2.7.0

+1 (with emphasis) On Sun, Aug 13, 2023 at 5:23 AM Ephraim Anierobi wrote: > +1 binding > > On Sun, Aug 13, 2023 at 6:57 AM Elad Kalif wrote: > > > +1 binding > > > > On Sat, Aug 12, 2023 at 9:32 PM Jarek Potiuk wrote: > > > > > > The vote doesn't say the release must happen, just that it can

Re: [ANNOUNCE] Apache Airflow 2.7.0 Released

Awesome! Great work team, this is definitely one of the biggest new Airflow releases! On Fri, Aug 18, 2023 at 9:16 AM Jarek Potiuk wrote: > Great! > > On Fri, Aug 18, 2023 at 6:08 PM Ephraim Anierobi < > ephraimanier...@apache.org> > wrote: > > > Dear Airflow community, > > > > I'm happy to ann

Re: [VOTE] Drop MsSQL as supported backend

+1 (binding) On Thu, Aug 31, 2023 at 8:09 AM Josh Fell wrote: > +1 (binding) > > On Thu, Aug 31, 2023 at 8:35 AM Ankit Chaurasia > wrote: > > > +1 (non-binding) > > *Ankit Chaurasia* > > > > > > > > > > > > > > On Thu, Aug 31, 2023 at 5:47 PM Phani Kumar > > wrote: > > > > > +1 (non-binding) >

Re: [ANNOUNCE] Pankaj Koti and Amogh Desai as committers

Congratulations Pankaj Koti and Amogh Desai! Very well deserved! On Tue, Sep 19, 2023 at 10:32 AM Shubham Raj wrote: > Congratulations Amogh and Pankaj. It’s great. > > On Tue, 19 Sep 2023 at 10:27 AM, utkarsh sharma > wrote: > > > Congratulations @Pankaj and @Amogh :) > > > > On Tue, 19 Sep

Re: [VOTE] Add providers for Pinecone, OpenAI & Cohere to enable first-class LLMOps

+1 (Binding) On Wed, Oct 25, 2023 at 7:20 AM Wei Lee wrote: > +1 (non-binding) > > Best, > Wei > > > On Wed, Oct 25, 2023 at 10:44 PM Vincent Beck wrote: > > > +1 binding > > > > On 2023/10/25 13:32:49 Pierre Jeambrun wrote: > > > +1 (binding) > > > > > > Le mer. 25 oct. 2023 à 13:29, Pankaj Si

Re: Initiative on Strengthenig security for Apache Airflow

Thanks for sharing. This is exciting news and I'm happy to see this! On Tue, Nov 7, 2023 at 10:40 AM Jarek Potiuk wrote: > Hello everyone, > > I wanted to share some news (not so much news for us but - it's just now > reached publication stage) that we have nice security / release process > impr

Re: [ANNOUNCE] Apache Airflow 2.8.0 Released

Very cool, Great to see this come together especially after all the challenges last week! Thanks to the team who pulled this together! On Mon, Dec 18, 2023 at 7:27 PM Amogh Desai wrote: > Awesome! 🤩 > > On Tue, 19 Dec 2023 at 1:35 AM, Bolke de Bruin wrote: > > > 🥳 > > > > B. > > > > Sent from

Re: [VOTE] Remove experimental API

-1 As much as I would like to see this removed, I feel the same way as Jed above. In response to the question raised regarding "Experimental features", the reason why this one seems different is because though this was marked as "experimental", it was the only way to interact with Airflow before

Re: [VOTE] AIP-64: Keep TaskInstance try history

+1 binding On Mon, Mar 25, 2024 at 6:19 PM Mehta, Shubham wrote: > +1 (non-binding). > > Shubham > > On 2024-03-25, 5:10 PM, "Wei Lee" weilee...@gmail.com>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can con

Re: [ANNOUNCE] New committer: Wei Lee

Awesome! Congratulations Wei! Very well deserved. A tremendous amount of contributions in a relatively short time! On Mon, Apr 8, 2024 at 2:27 PM Oliveira, Niko wrote: > Congrats Wei! Well deserved :) > > > From: Scheffler Jens (XC-AS/EAE-ADA-T) .INVALID> > Sen

Re: [DISCUSS] Redis licencing changes impact

Agree with your assessment Jarek, and that we don't have to do anything here. On Thu, Apr 11, 2024 at 5:29 AM Jarek Potiuk wrote: > Hello here, > > I've raised the discussion on private@ and it seems that there are no > private/controversies there, so I bring the discussion to devlist where it

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

A wonderful and exciting Saturday morning discussion! Thank you Jarek for bringing the offline conversations into the mailing list. I completely agree on the necessity of Airflow 3. I also agree that Gen AI is the trigger i.e. the answer to "Why now"? Having been thinking about this for a while f

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

Good point Jed. I responded back to your comment in the doc as well and very open to changing the term in the doc. Used the term "interactive DAG run" as the ability to invoke or trigger a DAG run through the API, with the expectation of getting back a result immediately. An alternate term could b

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

even after > > the explanation of Vikram I still don't have a clue what we want to > > accomplish there :-P. > > > > I would like to see a mantra or team for Airflow 3. That helps nudging > > people in the same direction. Suggestions in the comments. > > &g

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

Definitely a fast moving thread on the mailing list. I haven’t been able to respond for a few days and feel very far behind already. A few comments on topics discussed the last few days: - Jarek, in response to your comments around being more aggressive than in Airflow 2 about deprecation and drop

Re: [VOTE] AIP-69 Remote Executor

I agree with Jarek and Ash on this. I believe that the AIP as written documents the “what” and the “why” very well, but is too light on the “how”. I very much would like to see this AIP become reality as well, but I believe that we need some foundational elements such as API-44 and the “task cont

[DISCUSS] Proposal to enhance Backfills

include at least: - Making the Airflow Scheduler responsible for scheduling decisions on all DagRuns (instead of the current where it purposefully ignores backfill runs) - A new API endpoint to submit a "backfill request". -- Best regards, Vikram Koka, Ash Berlin-Taylor, Kaxil Naik, and Constance Martineau

[DISCUSS] Proposal for Synchronous (aka Interactive) DAG Execution

- Return the failure status to the invoking API as a key task in the DAG has failed (inc upstream_failed etc.), rather than waiting for DagRun completion (i.e waiting for teardown task completion). --- Best regards, Vikram Koka, Ash Berlin-Taylor, Kaxil Naik, and Constance Martineau

Re: [VOTE] May 2024 PR of the Month

Agree that there were many excellent nominations, but my vote goes to #39336 Vikram On Tue, May 28, 2024 at 6:11 PM Wei Lee wrote: > Hard to choose only one, but my vote goes to #39336. > > Best, > Wei > > > On May 29, 2024, at 4:45 AM, Scheffler Jens (XC-AS/EAE-ADA-T) < > jens.scheff...@de.bos

[DISCUSS] Proposal around the Injection of Task Execution Secrets

Internal API covers the interaction between the Airflow workers and Airflow metadatabase for heartbeat information, persisting XComs, and so on. -- Best regards, Vikram Koka, Ash Berlin-Taylor, Kaxil Naik, and Constance Martineau

Re: [DISCUSS] Proposal to enhance Backfills

or this to become a reality, Backfills need to be handled > by > > > the > > > Airflow Scheduler as a normal DAG execution > > > > > > I think it's a good idea. > > > It should solve natively problems like > > > https://github.com/apac

Re: [Airflow 3] The Airflow actors

Thanks Elad, this is a very good beginning. I left a bunch of comments on the wiki page in the interest of centralized tracking and review. On Thu, Jun 13, 2024 at 3:26 AM Kaxil Naik wrote: > Awesome, thanks Elad > > On Thu, 13 Jun 2024 at 07:50, Jarek Potiuk wrote: > > > Nice one. > > > > On

Re: Re: [DISCUSS] Implement common.dataframe (or similar) provider based on Ibis

Very interested in this. I am quite positive and supportive of adding support for a generic dataframe abstraction within Airflow. However, I do have a few questions around how and where to include this within Airflow from a dependency perspective. I do wonder if this needs to be in Core Airflow o

Re: [Meeting Notes] Airflow 3.0 Dev call - 20 June 2024

Thanks Kaxil, I really appreciate the diligent follow up here. Both the preparation and follow through is excellent! Vikram On Mon, Jun 24, 2024 at 5:09 PM Kaxil Naik wrote: > Hey all, > > Apologies for the delay! > > I have updated our meeting notes document to summarize the discussion > from

Re: Using AI / Dosu to help us with triaging issues

+1 Love it! On Wed, Jun 26, 2024 at 1:23 PM Vincent Beck wrote: > Fantastic idea! > > On 2024/06/26 20:12:43 Jarek Potiuk wrote: > > Hello everyone, > > > > Together with Elad, Kaxil, and the Dosu team [1], we’ve been looking into > > employing AI / Natural Language processing to help us triage

Re: [VOTE] June 2024 PR of the Month

Voting for #39355. Love the dark mode feature! On Wed, Jun 26, 2024 at 9:45 PM Amogh Desai wrote: > My vote would definitely go to > https://github.com/apache/airflow/pull/39355! > > Developers love dark mode, and I mean it is done fantastically! > > Thanks & Regards, > Amogh Desai > > > On Thu,

Re: [ANNOUNCE] New committer: Ryan Hatter

Awesome! Congratulations Ryan! On Fri, Jun 28, 2024 at 11:07 AM Pierre Jeambrun wrote: > Well done Ryan :) > > Le ven. 28 juin 2024 à 19:42, Ferruzzi, Dennis > > a écrit : > > > Hey, congrats! > > > > > > - ferruzzi > > > > > > > > From: Pankaj Koti > > Sent:

Re: [PROPOSE] Agree and document Ad-hoc release process for providers

+1 on clarifying the policy for sure. However, I have concerns around the rest of this from a reliability standpoint. Unless this is to be used only for critical situations and for selective providers only, this seems like a speed vs. reliability tradeoff. I am uncomfortable with this. On Wed, J

Re: [VOTE] AIP-72: Task Execution Interface aka Task SDK

+1 binding On Thu, Jul 11, 2024 at 6:58 AM Vincent Beck wrote: > +1 binding > > On 2024/07/11 13:32:14 Igor Kholopov wrote: > > +1, non-binding > > > > Some alignment with AIP-66 might be required, but the general vision > > implementation looks clear to me. > > > > Thanks for leading this eff

Re: [DISCUSS] AIP-78 scheduler-managed backfill

In my mind, there are two separate discussions here. 1. Locking and deadlocks This I believe will get better as part of the set of changes in this AIP as well as AIP-72. Of course, pending the "mini-scheduler" discussion. I am personally in two minds about that, having been both a proponent and a

Re: [ANNOUNCE] New committers: Rom Sharon & Shahar Epstein

Congratulations Rom and Shahar! Vikram On Mon, Jul 15, 2024 at 9:26 AM Ferruzzi, Dennis wrote: > Congrats! > > > - ferruzzi > > > > From: Shahar Epstein > Sent: Monday, July 15, 2024 8:30 AM > To: dev@airflow.apache.org > Subject: RE: [EXT] [ANNOUNCE] New com

Re: [RESULT] Accepted - AIP-72: Task Execution Interface aka Task SDK

Thank you all. I believe that this is an important foundational improvement for Airflow and really appreciate your support in moving this forward. Vikram On Tue, Jul 16, 2024 at 7:34 AM Ash Berlin-Taylor wrote: > This vote has passed with 13 binding +1s and five non-binding votes. > > Thanks a

Re: [Meeting Notes] Airflow 3.0 Dev call - 11 July 2024

Thanks for the meeting notes Kaxil, this looks right. One key takeaway from the meeting was the relationship between AIP-38, AIP-68, and now AIP-79. The sooner we can clarify the dependencies here and establish the fallbacks i.e. with respect to FAB auth, the better from a risk management perspect

Re: [DISCUSS] AIP-66: DAG Bundles & Parsing

Agreed. This looks really clean now. On Tue, Jul 16, 2024 at 1:25 PM Jarek Potiuk wrote: > It looks very good now :) ! Yeah. The "execution environment" has many, > many traps and edge cases. > > On Tue, Jul 16, 2024 at 10:19 PM Jed Cunningham > wrote: > > > I've removed my attempt to bring pyt

Re: [DISCUSS] To AIP-44 or not to AIP-44

I am very relieved with the "experimental, don't try this at home" label on this body of work, since this will be replaced shortly. If there is belief that this will help accelerate learning towards AIP-69 (Remote Executor), I understand the rationale for proceeding forward with this given the cav

Re: [VOTE] (v2) AIP-69 Remote Executor

I agree that this AIP is well thought out and needs to be worked on. Just making a note that this is dependent on AIP-44 which was discussed in an earlier thread on the dev list, which is currently proposed to be marked "experimental". I approve this AIP with the explicit caveat that this also be

Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)

Thanks for the updated write up, Jarek. I read the Confluence doc a couple of times and left a couple of small comments there as well. The biggest items I am trying to wrap my head around now are: 1. The roles and responsibilities of what you have called "Organization Deployment Managers" vs. "Tea

Re: [VOTE] AIP-65: Improve DAG history in UI

+1 This is possibly the highest voted feature from the Airflow survey. On Fri, Jul 19, 2024 at 1:50 PM Jed Cunningham wrote: > I’m calling for a vote on AIP 65: > https://cwiki.apache.org/confluence/x/T4qSEQ > > Discussion thread: > https://lists.apache.org/thread/vvm43tfchyo92hmf40fqvmq0f5845

Re: [VOTE] AIP-65: Improve DAG history in UI

Sorry, should have said +1 (binding) On Fri, Jul 19, 2024 at 2:18 PM Vikram Koka wrote: > +1 > > This is possibly the highest voted feature from the Airflow survey. > > > On Fri, Jul 19, 2024 at 1:50 PM Jed Cunningham > wrote: > >> I’m calling for a vote on AIP 6

Re: [VOTE] AIP-66: DAG Bundles and Parsing

+1 binding On Fri, Jul 19, 2024 at 1:43 PM Jed Cunningham wrote: > I’m calling for a vote on this AIP 66: > https://cwiki.apache.org/confluence/x/ZIqSEQ > > Discussion thread: > https://lists.apache.org/thread/l8ksl144xd43jfk1wk3kz77t1xgbbq7z > > The vote will run for 5 days and last till next

Re: [VOTE] AIPs 73 74 75: Data Awareness with Assets

+1 binding On Mon, Jul 22, 2024 at 11:14 PM Utkarsh Sharma wrote: > +1 binding > > Thanks, > Utkarsh Sharma > > On Tue, Jul 23, 2024 at 7:45 AM Wei Lee wrote: > > > +1 binding for all the 3 AIPs > > > > Best, > > Wei > > > > > On Jul 22, 2024, at 10:49 PM, Aritra Basu > > wrote: > > > > > > +1

Re: [VOTE] AIP-78 Scheduler-managed backfills

+1 binding On Thu, Jul 25, 2024 at 7:36 AM Eugen Kosteev wrote: > +1 (non-binding) > > On Thu, Jul 25, 2024 at 4:17 PM Vincent Beck wrote: > > > I am all in for simplification! +1 binding > > > > On 2024/07/25 12:16:19 Igor Kholopov wrote: > > > +1 (non-binding) > > > > > > On Thu, Jul 25, 2024

Re: [DISCUSSION] Proposal to Add OTEL Provider to Apache Airflow

Howard, I am intrigued by this, but unclear on what this would actually look like and what benefits it would add. Specifically, I believe that AIP-49 adds support for OTEL emission of metrics and traces, but NOT task logs from Airflow. I am probably being dense here, but I don't quite understand

Re: [DISCUSS] AIP-81: Enhanced Security in CLI via Integration of API

Thanks for writing this up! I left a quick question as a comment in the proposal. Best regards, Vikram On Wed, Jul 24, 2024 at 2:15 PM Buğra Öztürk wrote: > Hey all, > I have created a proposal for an Airflow 3.0 workstream: to utilize API for > CLI > > Details in https://cwiki.apache.org/con

Re: [DISCUSS] External event driven scheduling in Airflow

Vincent, Thanks for writing this up. The overview looks really good! I will leave my comments in the AIP as well, but at a high level they are both relatively focused on the "how", rather than the "what". With respect to the pull / polling approach, I completely agree that some incarnation of thi

Re: AWS Open Source Credits Program

Awesome! Very cool, thanks for sharing On Fri, Jul 26, 2024 at 12:40 PM Jarek Potiuk wrote: > Whoa. Fantastic! > > This means that we should double our efforts to bring back self-hosted > runners based on the K8S controller. > > The past framework is broken as of a week so we switch to all-publi

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

After I read about the migration issues, I was very concerned about this AIP and was leaning against it. I like where the discussion is heading now and generally feel more positive at this point. I am still struggling however, to understand the timing of what would be delivered when and in which r

Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)

Thanks Jarek, both the comments you added above make sense and help me understand the steps here. I was definitely struggling with the dependencies of this AIP on AIP-72 (task isolation) and AIP-66(DAG Bundles). Specifically, there were a couple of references which were key in my mind. - Workload

Re: [DISCUSSION] AIP-83 Remove Execution Date Unique Constraint from DAG Run

Completely agreed on this. +1 binding On Mon, Jul 29, 2024 at 3:19 PM Tzu-ping Chung wrote: > Hi all, > > I have published a draft AIP to remove the unique constraint on the > execution_date field of table dag_run in the meta database. This is a > long-planned todo item from AIP-39 (where we in

Re: [DISCUSSION] AIP-83 Remove Execution Date Unique Constraint from DAG Run

he existing replies toward real votes when the thread is > posted. Drop a word if I shouldn’t. > > TP > > > > On 30 Jul 2024, at 11:40, Phani Kumar > wrote: > > > > +1 binding > > > > On Tue, Jul 30, 2024 at 8:45 AM Jarek Potiuk wrote: > > > >

Re: [VOTE] AIP-68 Extended Plugin Interface for React Views

+1 binding On Fri, Jul 26, 2024 at 11:44 PM Shahar Epstein wrote: > +1 (binding) > > > On 2024/07/26 16:01:21 "Scheffler Jens (XC-AS/EAE-ADA-T)" wrote: > > Hi, > > > > Brent and me have revised the AIP-68 based on the other Airflow 3.x > discussions we had and as no further discussions are open

Re: [DISCUSS] AIP-76 Asset Partitions

Thanks TP, I left a bunch of comments in the AIP doc. But overall, this looks really good Vikram On Mon, Jul 29, 2024 at 2:23 PM Kaxil Naik wrote: > For example: Partitioning the Data: > > @asset(..., schedule="@daily", partition=PartitionByInternal("@daily")) > def daily_sales_data(): > ..

Re: [DISUSSION] AIP-78 & AIP 84 Remove FAB as core dependency and UI REST API

Thanks Jed and Brent. Just to be precise, I believe you meant AIP-79, not AIP-78 for the "Remove Flask AppBuilder as a Core Dependency". On Wed, Jul 31, 2024 at 2:07 PM Jed Cunningham wrote: > Discuss quickly though, we intend to start the vote tomorrow so we hit the > August 1sth deadline :)

Re: [DISUSSION] AIP-78 & AIP 84 Remove FAB as core dependency and UI REST API

Thanks Jed and Brent, both of these seem quite straightforward. I am really glad to see the AIP-79 Remove Flask AppBuilder approach because of the mitigation of the migration challenges. On Wed, Jul 31, 2024 at 4:40 PM Vikram Koka wrote: > Thanks Jed and Brent. > > Just to be p

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

+1 binding on both AIP-79 and AIP-84. Vikram On Thu, Aug 1, 2024 at 11:35 AM Vincent Beck wrote: > +1 binding for both > > On 2024/08/01 18:33:51 Jed Cunningham wrote: > > +1 binding! > > > > - > To unsubscribe, e-mail: dev-un

Re: [DISCUSS] External event driven scheduling in Airflow

> > > send > > > > >> > > >> notifications from external (e.g. cloud provider) to > Airflow > > > > >> > > environment. > > > > >> > > >> > > > > >> > > >> - I up

Re: [RESULT][VOTE] July 2024 PR of the Month

Very cool! Congratulations Niko! On Thu, Aug 1, 2024 at 5:42 PM Kaxil Naik wrote: > Congrats Niko > > On Fri, 2 Aug 2024 at 00:00, Ferruzzi, Dennis > > wrote: > > > Congrats Niko, and almost Vincent :P > > > > - ferruzzi > > > > > > > > From: Briana Okyere > >

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

Agreed Jarek on the parallel workstream for auth and also that should not be a blocker for 3.0. I don't know if the right answer is actually Keycloak. There was some research done by my colleagues within Astronomer using Casbin for the same, but I don't know the differences between those and other

Re: [VOTE] AIP-76: Asset Partitions

This has been one of the harder AIPs to review and I have asked a lot of questions on this, but I am happy about where it is now. +1 binding On Thu, Aug 1, 2024 at 7:39 PM Wei Lee wrote: > +1 (binding) > > Best, > Wei > > > On Aug 2, 2024, at 9:09 AM, Kaxil Naik wrote: > > > > We should resolv

Re: Airflow 3 AIP Gold Rush! -- Great job everyone

+100 to Kaxil's lovely email summary. I could not be more proud of the Airflow Community and I am honored to be part of it! Jarek sent out the email starting the discussion about Airflow towards the end of April, and in the three months since we have discussed, debated, and finally aligned on a s

  1   2   3   4   >