Re: [DISCUSS] Consistent test assertion style: pytest-native vs unittest-style

Agreed, I personally also find the current way to be easier to read and in most cases we want to assert if "something" was called, irrespective of the order it was called in. So the dedicated function based way works well for me. If I want to test a order, I'd rather call parts of code that I want

Re: [DISCUSS] - LLM-Powered DAG Generation

Strongly concur with everything Jarek had to tell. I would be in favour of a strong NO for this. The use case of having an embedded UI to type in natural language is as good as opening your IDE locally with integrated LLM models and asking it to generate the dag using natural language and drop it

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

> @Jarek Potiuk a little confused on what you mean there, I am understanding the direction but could you elaborate a bit more please? Let me elaborate: As I understand (maybe I am wrong?), the proposal is that we have a "core-and-task-sdk" folder which is a shared distribution that is vendored-

Re: [Meeting Notes] Airflow 3 dev call - 3 July 2025

Hi Eugene, Good point, I should have put in the link for the scope/timeline. It is here I will expand on this with details / links, etc. and also make it more easily findable on the wiki befor

Re: [HELP NEEDED / Contribution Opportunity] Migrate Provider Packages to use `BaseHook` via version_compat

That's awesome , thank you everyone. Pavan On Mon, Jul 7, 2025 at 10:09 AM Amogh Desai wrote: > Hi All, > > Here are a few stats that highlight the strength and dedication of this > community: > > * ✅ ~62 PRs were merged in under 4 days > * 📆 Half of that time was over a weekend — including a l

[DISCUSS] Use ~= for python requires?

Hello here, If you have not noticed - we have a little bit of drama because in the latest `uv` version, Astral unilaterally decided to make the `~=3.10` de-facto invalid specification. The `~=3.10` is a perfectly valid specification widely recognized as `>=3.10,<4` and specified like that (named

Re: [DISCUSS] Use ~= for python requires?

I prefer the later (>=3.10, <4) somewhat more, but not strongly. It’s easier to understand IMO. Best, Wei > On Jul 7, 2025, at 5:12 PM, Jarek Potiuk wrote: > > Hello here, > > If you have not noticed - we have a little bit of drama because in the > latest `uv` version, Astral unilaterally dec

Re: [Meeting Notes] Airflow 3 dev call - 3 July 2025

Thank you! On Mon, Jul 7, 2025 at 10:12 AM Vikram Koka wrote: > Hi Eugene, > > Good point, I should have put in the link for the scope/timeline. It is > here > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153069#Airflow3.0-Airflow3.1scope > > > > > I will expand on thi

Re: [HELP NEEDED / Contribution Opportunity] Migrate Provider Packages to use `BaseHook` via version_compat

Oh that's not nice :) Sending the graph as an attachment and also as raw text. Author | PRs Merged -| @sunank200 | 30 @amoghrajesh | 13 @bdsoha | 6 @phanikumv | 5 @dominikhei | 2 @fweilun | 2 @mingdaoy| 1

Re: [DISCUSS] Use ~= for python requires?

Thanks Jarek. I do not have a strong objection to either form. Both the ways are functionally the same and valid per PEP 440. If I had to slightly lean one way, I’d prefer ~=3.10 for its brevity and simplicity. That said, I would also understand why some might prefer the more explicit >=3.10,<4,

Re: [HELP NEEDED / Contribution Opportunity] Migrate Provider Packages to use `BaseHook` via version_compat

NICE! On Mon, Jul 7, 2025 at 12:10 PM Amogh Desai wrote: > Oh that's not nice :) > > Sending the graph as an attachment and also as raw text. > > Author | PRs Merged > > -| > > @sunank200 | 30 > > @amoghrajesh | 13 > > @bdsoha | 6 > > @pha

[ACCELERATED VOTE] Release Airflow 3.0.3 from 3.0.3rc4 & Task SDK 1.0.3 from 1.0.3rc4

Hey fellow Airflowers, The release candidates for *Apache Airflow 3.0.3rc4 *and *Task SDK 1.0.3rc4* are now available for testing! This email is calling for an accelerated vote on the release, which will last at least until 8th July 1:30 pm UK time and until 3 binding +1 votes have been received.

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Okay, so it seems we have agreement on the approach here, so I’ll continue with this, and on the dev call it was mentioned that “airflow-common” wasn’t a great name, so here is my proposal for the file structure; ``` / task-sdk/... airflow-core/... shared/ kuberenetes/ pyproject.

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Oh, and all of this will be explain in shared/README.md > On 7 Jul 2025, at 13:41, Ash Berlin-Taylor wrote: > > Okay, so it seems we have agreement on the approach here, so I’ll continue > with this, and on the dev call it was mentioned that “airflow-common” wasn’t > a great name, so here is m

Re: [DISCUSS] Consistent test assertion style: pytest-native vs unittest-style

def test_get_account(self, mock_paginate: Mock, mock_http_run: Mocj, conn_id, account_id): Might fix that? IDEs in general do not cope well with purest tests, and are missing context on what most of the variables are, be it parameterised values or fixture values, so this isn’t a problem that i

Re: [DISCUSS] Fixing (removing) initializations in "airflow.__init__`) in 3.1 or 3.2 ?

Yeah, this has been a long time bugbear of mine and would love to remove the magic and the side-effects of `import airflow`. Do you have any plans or thoughts about how to actually achieve this? This might be related, or it might not be, but I think I would also love it if we moved all of “cor

RE: [DISCUSS] Use ~= for python requires?

Yes, in fact many (all?) universal resolution tools, such as uv, simply ignore the upper bound of requires-python when resolving, so it's likely not doing what you think it should be doing. Damian -Original Message- From: Jarek Potiuk Sent: Monday, July 7, 2025 10:23 AM To: dev@airflow

Re: Discuss: new option [secrets]backend_order

> – add display of this variable in the UI(the main question where it should be)? I think it should be displayed in the "connections" and "variables" page. Currently we have some information - that variables and connections visible here are only coming from the DB and others sources (I remember ev

Re: [DISCUSS] Use ~= for python requires?

You don’t find anyone actively saying it’s bad, but you also would not find anyone recommending it with a good reason. With Python’s versioning scheme, having <4 makes as much sense as (random example) <3.10. I believe the trend was started by Poetry having all version range specifications to u

Re: [DISCUSS] Use ~= for python requires?

Makes sense. Maybe we should then mention that in the https://github.com/astral-sh/uv/issues/14422 ? I do not want to "spoil" relationship with Astral, and while I strongly believe that they should not make such decisions unilaterally and forcing people to change their choices, having a `pip` maint

Re: [DISCUSS] Fixing (removing) initializations in "airflow.__init__`) in 3.1 or 3.2 ?

> This might be related, or it might not be, but I think I would also love it if we moved all of “core” (scheduler, jobs, api server etc) to airflow_core.* python modules , and out of `airflow.*` entirely. (Meaning `airflow` would be left for just `airflow.sdk` and `airflow.providers`, plus some co

Re: [Meeting Notes] Airflow 3 dev call - 3 July 2025

Hi Vikram. Thanks for the summary from the meeting. I missed a call (assuming that it will be in 2 weeks from previous). Can you, please, share what is the timeline for the Airflow 3.1 release. It is not mentioned in the summary, and I can't find it on wiki. Thank you - Eugene On Mon, Jul 7, 20

Re: [DISCUSS] Use ~= for python requires?

It looks like https://discuss.python.org/t/requires-python-upper-limits/12663 - it is indeed a widely adopted practice to not upper-bind python-requires at all. So neither ~=3.10, nor >=3.10,<4 but `>=3.10` seems to be the **right** approach. And I am advocating now for changing the warning message

Re: Discuss: new option [secrets]backend_order

Hi everyone! First of all, I’d like to thank all the participants in this discussion. Based on what I’ve read, am I understanding correctly that in order for this PR to be merged into the main Airflow codebase, I need to: – add usage examples to the Airflow documentation, – add display of this

Re: [DISCUSS] Fixing (removing) initializations in "airflow.__init__`) in 3.1 or 3.2 ?

+1 for doing this. I recently went through some pain along similar veins and came to the conclusion that `import airflow` does a lot! Not a concrete plan, but a starting to investigate the airflow/__init__.py and all other init's to see what is being initialised (config, ORM, logging, etc etc) wo

Re: [DISCUSS] Use ~= for python requires?

Thanks Jarek, I am just catching up with this discussion. I agree that this is unilaterally forcing us to make changes, though the one (~=3.10) is also the standard one we have been using. I am in favour of using our existing convention ~=3.10. Pavan On Mon, Jul 7, 2025 at 10:12 AM Jarek Potiuk

Re: [DISCUSS] - LLM-Powered DAG Generation

Yup agreed with Jarek. A strong no from my side. We don't want to allow authoring DAGs from Airflow UI especially just to provide an LLM interface. On Mon, 7 Jul 2025 at 12:27, Jarek Potiuk wrote: > Also you might take a look at Airflow Summit videos > https://www.youtube.com/playlist?list=PLG

[CANCELLED][VOTE] Release Airflow 3.0.3 from 3.0.3rc3 & Task SDK 1.0.3 from 1.0.3rc3

Unfortunately, https://github.com/apache/airflow/issues/52907 was identified so cancelling this vote. The fix for it has already been merged. Creating 3.0.3rc4 in a bit On Mon, 7 Jul 2025 at 07:04, Wei Lee wrote: > +1 non-binding. > > Tested my migration fixes and ran a few dags. > > Best, > Wei

Re: Discuss: new option [secrets]backend_order

I also think it would be beneficial for users / someone editing / accessing connections or variables from the UI to know "where" they are editing it. Right now it's the metadata DB but with the proposed PR that probably could change (for cases when DB is highest priority / middle priority?) But g

Re: [HELP NEEDED / Contribution Opportunity] Migrate Provider Packages to use `BaseHook` via version_compat

Fantastic -> can you also share the image elsewhere - our devlist is not very good in getting embedded images through -> so all is is broken image :D On Mon, Jul 7, 2025 at 11:12 AM Pavankumar Gopidesu wrote: > That's awesome , thank you everyone. > > Pavan > > On Mon, Jul 7, 2025 at 10:09 AM Am

[DISCUSS] Fixing (removing) initializations in "airflow.__init__`) in 3.1 or 3.2 ?

I would like to raise another discussion here - about fixing `airflow.__init__.py` excessive initialization pattern - potentially. It results from https://github.com/apache/airflow/pull/52952#discussion_r2188492257 discussion. This is something we have been seeing for quite some time in Airflow 1

Re: [DISCUSS] Use ~= for python requires?

> Note that while, as mentioned, ~= is a standard and well-supported operator, it is generally NOT RECOMMENDED to be used for Python version specification. It is also not recommended to use the equivalent expression composed with >= and <. Interesting - could you point where it is specified that i

Re: [DISCUSS] Use ~= for python requires?

And ... interestingly enough currently the `uv` warning actually proposes `>= and <` , so we might want to tell them that they should not recommend it if it is not recommended by PyPA. On Mon, Jul 7, 2025 at 1:39 PM Jarek Potiuk wrote: > > Note that while, as mentioned, ~= is a standard and well

Re: [DISCUSS] Consistent test assertion style: pytest-native vs unittest-style

I'd like to follow up on our previous discussion about pytest-native vs unittest-style assertions. While working on the following test case: ```python @pytest.mark.parametrize( argnames="conn_id, account_id", argvalues=[(ACCOUNT_ID_CONN, None), (NO_ACCOUNT_ID_CONN, ACCOUNT_ID)], ids=

Re: [DISCUSS] Use ~= for python requires?

I feel that we are arriving at a good solution - and with a good reasoning and proper advice :) >From the ambiguity of ~= and advising to use >=,< to avoiding < altogether and doing just `>=` ... Nice! And Concise. On Mon, Jul 7, 2025 at 4:43 PM Damian Shaw wrote: > Yes, in fact many (all?) u

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

Thanks for getting back to me and sharing a concrete example Jarek! I think overloading the executor field, yet again, with another dimension is going to get quite messy. If you have many teams and many executors per team you now have colons (serving two purposes), semi-colons and commas in one

[CANCELLED][ACCELERATED VOTE] Release Airflow 3.0.3 from 3.0.3rc4 & Task SDK 1.0.3 from 1.0.3rc4

Cancelling this vote, more bugs On Mon, 7 Jul 2025 at 17:44, Kaxil Naik wrote: > Hey fellow Airflowers, > > The release candidates for *Apache Airflow 3.0.3rc4 *and *Task SDK > 1.0.3rc4* are now available for testing! > > This email is calling for an accelerated vote on the release, which will >

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

I like "core_and_task_sdk the same like core-and-task-sdk - I have no problem and it is a path only. if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is a bit bulky we then should name it "all-not-api-server" :-D On 07.07.25 22:57, Ash Berlin-Taylor wrote: In case I did a bad

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

How about splitting it even more and having each shared "thing" named? "logging", "config" and sharing them explicitly and separately with the right "user" ? That sounds way more modular and we will be able to choose which of the shared "utils" we use where. J. On Mon, Jul 7, 2025 at 11:13 PM J

Re: [VOTE] Release Apache Airflow Helm Chart 1.18.0 based on 1.18.0rc1

I'm cancelling this vote, a regression was found. I will cut and start a vote on RC2 here shortly.

Re: [DISCUSS] Fixing (removing) initializations in "airflow.__init__`) in 3.1 or 3.2 ?

Thanks Jarek for starting the discussion which I needed to think about (and was busy all day) until reading the thread - is already well evolved with details! I assume also we need to make 1-2 PoCs considering how this is made, more with @decorators (like @requires[ORM] to signal what dependen

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Looks good but I think we should find some better logical name for core_and_sdk :) pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler napisał: > Cool! Especially the "shared" folder with the ability to have > N-combinations w/o exploding project repo root! > > On 07.07.25 14:43, Ash Berlin-Taylor

Re: [DISCUSS] Fixing (removing) initializations in "airflow.__init__`) in 3.1 or 3.2 ?

Nope. No need to defer the fix for edge. I think it's good enough for now as long as we have a clear road ahead :) pon., 7 lip 2025, 21:58 użytkownik Jens Scheffler napisał: > Thanks Jarek for starting the discussion which I needed to think about > (and was busy all day) until reading the thread

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Thanks Ash! Looks cool! I like the structure. This will enable all the combinations and structure looks easy to grasp. No strong stance on the naming other than maybe it is a bit long with `and`, `core_ctl` could be shorter, since no import path is defined like that, we can give any name for sure.

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

I realized that I owe Niko an explanation of configuration changes. Again - following the philosophy above - minimal set of changes to "airflow internals". the "minimum" set of changes that will work. I propose the change below that has **no** changes to the way how the current configuration "share

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

In case I did a bad job explaining it, the “core and task sdk” is not in the module name/import name, just in the file path. Anyone have other ideas? > On 7 Jul 2025, at 21:37, Buğra Öztürk wrote: > > Thanks Ash! Looks cool! I like the structure. This will enable all the > combinations and s

Re: [HELP NEEDED / Contribution Opportunity] Migrate Provider Packages to use `BaseHook` via version_compat

Hi All, Here are a few stats that highlight the strength and dedication of this community: * ✅ ~62 PRs were merged in under 4 days * 📆 Half of that time was over a weekend — including a long weekend in the US * This high volume of contributions helped us uncover and resolve a system test failure

Re: [DISCUSS] Use ~= for python requires?

Note that while, as mentioned, ~= is a standard and well-supported operator, it is generally NOT RECOMMENDED to be used for Python version specification. It is also not recommended to use the equivalent expression composed with >= and <. Python does not use semantic versioning, and backward inco

[VOTE] AIP-67 Mutlti-team deployment of Airflow Components (RELOADED)

Hello Airflow community, I would like to call a vote on "reloaded" version of the AIP https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components The last discussion thread about "reloading" of the AIP is here https://lists.apache.org/thread/oqj804phspmr

Re: [DISCUSS] - LLM-Powered DAG Generation

Hi, also sharing the opinion that no dedicated UI is needed for this - but would be very welcoming to share experience and maybe expereince and starting promt to get going. So if you want to post (somewhere, e.g. medium) an article about how this is possible, that migth be worthwile to share.

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Cool! Especially the "shared" folder with the ability to have N-combinations w/o exploding project repo root! On 07.07.25 14:43, Ash Berlin-Taylor wrote: Oh, and all of this will be explain in shared/README.md On 7 Jul 2025, at 13:41, Ash Berlin-Taylor wrote: Okay, so it seems we have agree

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

To be honest ugliness of it and the messiness of supporting many teams is deliberate choice. This is also to add friction in supporting 'many' teams. Supporting 'many' teams had never been a goal for multi-team even in the last incarnation. Due to scaling of scheduler especially - we should not giv

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

To be honest I am glad you raised this argument - because it means that the goal I had in mind has been achieved. People will not think that it is easy to add more and more teams to the picture - the more they add, the more messy it will get. That's quite cool if you take into account what AIP-67

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

I like the folder structure proposed by Ash and have no objections with it. "core_and_task_sdk" sounds good to me and justifies what it should do pretty well. @Jarek Potiuk a little confused on what you mean there, I am understanding the direction but could you elaborate a bit more please? Nami