Re: [DISCUSS] "Use existing venv" support for PythonVirtualenvOperator as counterpart to AIP-46

Would like to vote for ExternalPythonOperator. Cause usually Virtualenv have symbolic links for python binaries untill used —copies to make it fully portable. Additionally there is option to use differently compiled python altogether (For example pypy or jython

Re: [Discussion] Airflow Newsletter name and branding

Hi All, I would like to suggest “Atmosphere”. This came to my mind when I think about Airflow in Clouds. Even different layers of Atmosphere have cool names. Maybe it’s a lame suggestion. Thanks, Abhishek On 15-Nov-2022 at 12:30:00 AM, John Thomas wrote: > Hi All, > > I'm looking to add a bit

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

Hi, I Beg to differ with Alexander and agree with Jarek. There are multiple ways to deploy Airflow. Mostly commonly used is docker images, in that case using one image for all components is standard practice. If using native pip installations, airflow components are launched by a single pip module

Re: [VOTE] April 2023 PR of the Month

+1 for #30375 is must to have. Thanks, Abhishek Bhakat Airflow Engineer On 26-Apr-2023 at 01:31:22, Jarek Potiuk wrote: > Indeed. Extremely tough choice this time, but I'd go for 30175 for me -> > Kafka provider has been long in the making and Dylan had done a lot of > effor

Re: [VOTE] Airflow Providers prepared on September 08, 2023

+1 (non-binding) Tested the OpenLineage providers. Looks good to me. On Fri, Sep 8, 2023 at 6:55 PM Vincent Beck wrote: > +1 (non-binding) > > I tested successfully the Amazon provider package by running some testing > DAGs and it looks good. > > On 2023/09/08 15:55:38 Elad Kalif wrote: > > Hey

Re: [VOTE] New Airflow Community Provider: Teradata

+1 non binding > On 17-Jan-2024, at 03:01, Aritra Basu wrote: > > +1 non binding > > -- > Regards, > Aritra Basu > > On Wed, Jan 17, 2024, 5:35 AM Wei Lee wrote: > >> +1 non binding >> >> Best, >> Wei >> >>> On Jan 17, 2024, at 2:54 AM, Josh Fell >> wrote: >>> >>> +1 (binding) >>> >>> O

Re: [VOTE] AIP-62 Getting Lineage from Hook Instrumentation

+1 (non-binding) Regards, Avi > On Apr 4, 2024, at 10:23, Pankaj Koti > wrote: > > +1 binding. Sounds exciting. > > Sorry I missed the discussion earlier. I have some questions but those are > implementation related > and I think the PRs will help me understand those :) > > > Best regards,

Re: [ANNOUNCE] New committer: Wei Lee

Yay! Congrats Wei! On Mon, Apr 8, 2024 at 11:36 AM Aritra Basu wrote: > Congrats wei! Great job! > > -- > Regards, > Aritra Basu > > On Mon, Apr 8, 2024, 4:17 PM wrote: > > > Congrats Wei! > > > > > On Apr 8, 2024, at 5:31 AM, Pankaj Singh > > wrote: > > > > > > Congrats Wei, very well deserv

Re: Call with Nielsen team demoing their DAG debugging feature

Hi Jarek, I would also like to join as well, please. Thanks, Avi On Sat, Jun 8, 2024 at 3:32 PM Buğra Öztürk wrote: > Hello Jarek, > > Thanks for sharing! It sounds very interesting. I would like to join. Could > you please forward to me as well? > > Thanks! > > On Sat, 8 Jun 2024, 17:28 Jed C

Re: [ANNOUNCE] New committer: Ryan Hatter

Congratulations Ryan 👏 On Sat, Jun 29, 2024 at 7:13 PM Ankit Chaurasia wrote: > Congratulations Ryan! > > *Ankit Chaurasia* > > > On Sat, 29 Jun 2024 at 20:02, Hemkumar Chheda > wrote: > > > Great News! Congratulations Ryan!! > > > > > On 29 Jun 2024, at 1:55 PM, Shahar Epstein wrote: > > > >

Re: [ANNOUNCE] New committers: Rom Sharon & Shahar Epstein

Congrats Rom & Sharan 🥳 On Mon, Jul 15, 2024 at 1:28 PM utkarsh sharma wrote: > Congratulations! 🎉🎉. > > Thanks, > Utkarsh Sharma > > On Mon, Jul 15, 2024 at 6:38 PM Jarek Potiuk wrote: > > > Wooohooo! Congrats! > > > > On Mon, Jul 15, 2024 at 2:29 PM Pavankumar Gopidesu < > > gopidesupa...@gma

Re: Operator Templating in Airflow 3

Seems nice. For the 2nd proposed change, I'm thinking another method for compat may be a dag_policy that drop-in replaces the params of any Operator. Avi

Re: [ANNOUNCE] New committers: Rom Sharon & Shahar Epstein

Congratulations, Rom and Shahar! 🎉 On Tue, Jul 16, 2024 at 4:49 PM Pierre Jeambrun wrote: > Congrats! > > On Tue 16 Jul 2024 at 09:04, Scheffler Jens (XC-AS/EAE-ADA-T) > wrote: > > > Welcome Rom and Shahar! > > > > Mit freundlichen Grüßen / Best regards > > > > Jens Scheffler > > > > Alliance:

Re: [ANNOUNCE] New PMC member: Jens Schaffler

Congratulations Jens 🎉 On Tue, Aug 6, 2024 at 7:52 AM Michał Modras wrote: > Congratulations Jens, well deserved! > > On Tue, Aug 6, 2024 at 9:51 AM Jarek Potiuk wrote: > > > The Project Management Committee (PMC) for Apache [PROJECT] > > has invited Jens to become a PMC member and we are pleas

Re: [VOTE] Release Airflow 2.3.0 from 2.3.0rc2

-1 (non-binding) An issue with parsing dynamic tasks, causes the scheduler to crash. https://github.com/apache/airflow/issues/23361 On Fri, Apr 29, 2022 at 7:58 PM Collin McNulty wrote: > +1 (non-binding) > > Made and ran several test DAGs using mapped tasks. > > On Fri, Apr 29, 2022 at 3:54 AM

Re: [VOTE] Release Airflow 2.3.0 from 2.3.0rc2

this isn't a regression in 2.3 I'd ask if you could remove your -1 > vote? > > On Fri, Apr 29 2022 at 20:13:31 +0530, Abhishek Bhakat > wrote: > > -1 (non-binding) > An issue with parsing dynamic tasks, causes the scheduler to crash. > https://github.com/apache/airf

Re: [VOTE] Release Airflow 2.3.0 from 2.3.0rc2

ince this is no worse in 2.3.0rc2 than it was in >>>> 2.2.5 it shouldn't stop the release of 2.3.0. >>>> >>>> Cheers, >>>> Ash >>>> >>>> On Fri, Apr 29 2022 at 20:28:49 +0530, Abhishek Bhakat >>>>

Re: Please vote on the PR of the Month for this month's Airflow Newsletter!

I also vote in favor of #24249 On 28-Jun-2022 at 7:28:55 PM, Phani Kumar wrote: > I also vote for 24249 :) > > On Tue, Jun 28, 2022 at 7:23 PM Jarek Potiuk wrote: > >> Let the fight begin :) >> >> On Tue, Jun 28, 2022 at 3:43 PM Brent Bovenzi >> wrote: >> > >> > I also vote for 24249 >> > >>

Re: [DISCUSS] "Magic Loop" as fist class citizen in Airflow ?

Can vote for making it as an optional approach for fine-tuning (only for advance users). On 12-Jul-2022 at 7:44:35 AM, Jarek Potiuk wrote: > Not interesting :) ? > > On Thu, Jul 7, 2022 at 10:41 AM Jarek Potiuk wrote: > > > Hello everyone, > > > We have just published a blog on our medium - > h

Re: [VOTE] AIP-44 - Airflow Internal API

+1 (non-binding) for gRPC only. On 11-Aug-2022 at 8:42:52 AM, Ash Berlin-Taylor wrote: > +1 (binding) now with that change. Thank you very much. > > I'm also okay with us to proceed with only gRPC for now -- with this > architectural change it's much easier to replace it in future if we > want/

Re: Airflow should deprecate the term "DAG" for end users

Agreed that the word DAG makes very less sense to someone new to workflow orchestration. But it does also show the nature of being acyclic. Sure, as Bas mentioned, there are ways to workaround it. Still, in my opinion, there is generally no need for cyclic behavior in workflow orchestration. Most (

Re: [DISCUSSION] Replace Poke & Reschedule mode from Sensors for Airflow 3 in favor of Deferrable

+1 to 3. For cases where absolute minimal latency is critical, and worker resources aren't constrained, poke mode could still be the optimal choice. I don't see any value in reschedule mode anymore, deferrable should be the default. On Wed, Nov 13, 2024 at 12:21 PM Kaxil Naik wrote: > There is 4

Re: Updating "zombie task" terminology to "task heartbeat timeout"

+1 to no more "zombie" Avi On Wed, Feb 12, 2025 at 12:38 PM Jarek Potiuk wrote: > +1 on both. Changing and Airflow 3. Apart of some concerns about the name > itself, I never remember what kind of tasks are zombies and what triggers > that. > > I think zombie is a bit overloaded term especially

Re: [DISCUSS] removal of uniqueness in airflow and partition-driven dag style

Can we do a hybrid of Option 2 and 3? class DAG: def __init__( self, dag_id: str, partition: Optional[str] = "auto", # "auto", "strict", "none" logical_dates: bool = True, # For backward compatibility ): 1. partition_mode="auto": Backward compat, logs

Re: Very strange (AI generated) issues

A temporary block should be fine if people can still create discussions. On which committers can help create issues if it's reasonable. Avi On Wed, Jan 22, 2025 at 12:57 PM Jarek Potiuk wrote: > - Iceberg dev to not flood them :) (in bcc:) > > It looks like the flood had been somehow flood-gat

Re: [DISCUSS]: Proposing a "common message queue" abstraction

+1 yay! event driven scheduling soon Regards, Avi On Thu, Feb 13, 2025 at 2:02 AM Wei Lee wrote: > +1 to this idea! > > Best, > Wei > > > On Feb 13, 2025, at 5:29 AM, Vikram Koka > wrote: > > > > Hey everyone, > > > > I created the following (very early) draft PR > >

Re: New committer: Rahul Vats

Congratulations Rahul 🥳 On Wed, Apr 2, 2025 at 8:13 AM Shahar Epstein wrote: > Congratulation Rahul! :) > > On Mon, Mar 31, 2025, 21:20 Jed Cunningham > wrote: > > > The Project Management Committee (PMC) for Apache Airflow > > has invited Rahul Vats to become a committer and we are pleased > >

Re: [DISCUSS] Decisions made on devlist

IMO having the discussion on Github has very helpful for me. That's just the first place we look for. And if been asked on why was "*this particular change*" was made, I can answer it brought up a discussion on the dev list and the community voted on it. In a gist, I don't have a problem with the

Re: Au revoir Flask

Yay! FABulous milestone! On Tue, Apr 8, 2025 at 2:27 AM Amogh Desai wrote: > Awesome news Vincent! > > Thanks & Regards, > Amogh Desai > > > On Tue, Apr 8, 2025 at 6:58 AM Wei Lee wrote: > > > This is nice! > > > > Best, > > Wei > > > > > On Apr 8, 2025, at 3:35 AM, Vikram Koka > > wrote: > >

Re: New committers: Shubham Raj, Buğra Öztürk, LIU ZHE YOU, Kalyan Reddy !

Congratulations All 🎉 Regards, Avi On Sat, Mar 15, 2025 at 3:12 AM Josix Wang wrote: > Congratulations! That's awesome 🎉 > > Best, > Josix > > Oliveira, Niko 於 2025年3月15日 週六 上午3:06寫道： > > > Congrats all! This is awesome to see :) > > > > > > From: Ephraim Anier

Re: [DISCUSS] Time to say goodbye to the old UI?

Where do I find the docs for plugins with the new web UI alternative for < https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-view-plugin.html >? Read the AIP-68 (plugin interface), but is it in a ready state yet. - Avi On Tue, Feb 18, 2025 at 5:26 AM Ankit Chaurasia wrote: > +1

Re: [DISCUSS] Time to say goodbye to the old UI?

18 Feb 2025, at 11:54, Michał Modras .INVALID> > > wrote: > > > > > > Would removing it imply that there's not going to be Airflow 2.12 for > > sure? > > > Do we want to limit ourselves this way? > > > > > > On Tue, Feb 18, 2025

Re: [DISCUSS] Auth backends

+1 to option 1 as well. - Avi On Fri, Feb 28, 2025 at 6:32 PM Jarek Potiuk wrote: > I very much like Option 1 only > > Especially if we can generate a python client that can easily "hide" the > necessary auth workflow and extend it by different mechanisms easily. I > think the fact that the JWT

Re: [DISCUSS] The `uv` as the only supported dev tool

+1 for uv. Quick question: We will still make the branches for constraints as we used to, correct? - Avi On Sat, Mar 22, 2025 at 7:24 AM Jarek Potiuk wrote: > Also one more important point - if we drop `pip` support - we will be able > to rely on `uv.lock` for constraint generation. Currently

Re: [VOTE] March 2025 PR of the Month

+1 for 47320 - Avi On Mon, Mar 24, 2025 at 10:57 PM Briana Okyere wrote: > Hey All, > > It’s once again time to vote for the PR of the Month! > > With the help of the `get_important_pr_candidates` script in dev/stats, > we've identified the following candidates: > > PR #47320: Disable ORM acces

[DISCUSS] Migrating to SQLAlchemy 2

Hi All, I've hit a little deadlock. We want to use SQLAlchemy 2.x ORM typing (Mapped[...] = mapped_column(...)) with PEP 604 union types (int | None) in model annotations and must support Python 3.9 (which does not support int | None at runtime). We do not want to use Optional[...] (ruff UP007 i