User Kinceo1 posting spam review approvals

2022-08-16 Thread Tzu-ping Chung
Hi all, Recently the user Kinceo1 on GitHub has been posting review approvals in a ton of PRs. Those reviews all contain one single emoji, and are done arbitrarily since multiple of those PRs are very obviously not in reviewable shape with significant errors. These spams are quite annoying and

Re: [DISCUSS] "Use existing venv" support for PythonVirtualenvOperator as counterpart to AIP-46

2022-08-16 Thread Tzu-ping Chung
One thing I thought of (but never bothered to write about) is to introduce a separate operator instead, say ExternalPythonOperator (bike shedding on name is welcomed), that explicitly takes a path to the interpreter (say in a virtual environment) and just use that to run the code. This also enab

Re: User Kinceo1 posting spam review approvals

2022-08-19 Thread Tzu-ping Chung
commenter%3AKinceo1 > <https://github.com/apache/airflow/issues?q=commenter%3AKinceo1> > > -ash > > On Wed, Aug 17 2022 at 07:32:40 +08:00:00, Tzu-ping Chung > wrote: >> Hi all, >> >> Recently the user Kinceo1 on GitHub has been posting review approval

Should value passed via XComArg be templated?

2022-08-28 Thread Tzu-ping Chung
Context: https://github.com/apache/airflow/issues/26016 Currently if a value is passed to a downstream via XComArg, e.g. @task def get_templated_echo(): return 'echo "The date is {{ ds }}"' BashOperator(task_id="echo_task", bash_command=get_t

Re: Should value passed via XComArg be templated?

2022-08-29 Thread Tzu-ping Chung
is nothing particular to XcomArg either, just the way Xcom had always > worked.) > > I don't think we can change the current/default behaviour without it likely > being a breaking change to someone's workflow. > > If there's enough desire and a good enough use case we

Re: [DISCUSS] "Use existing venv" support for PythonVirtualenvOperator as counterpart to AIP-46

2022-09-01 Thread Tzu-ping Chung
use very few > >> people realise that running Python from a virtualenv actually > >> implicitly "activates" the venv. > >> I think maybe deprecating the old PythonVirtualenvOperator and > >> introducing two new operators: PythonInCreatedVirtual

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

2023-11-26 Thread Tzu-ping Chung
Some context: I’ve been looking into putting the Connection URI to more use, most importantly to consolidate it with OpenLineage’s URI format. Skipping the details, one of the conclusions I had is get_uri on Connection is not actually that useful, due to how different libraries (SQLAlchemy, Airf

[Discussion] AIP-60 Standard URI representation for Airflow Datasets

2024-01-02 Thread Tzu-ping Chung
Happy 2024 everyone! I’m going to kick off the new year by formally proposing a new AIP. This attempts to standardise the URI format used by Dataset events. This is driven a lot by the lack of adoption of Datasets. It turns out (maybe not surprisingly when I think about it) simply triggering ev

[VOTE] Accept AIP-60 (Standard URI representation for Airflow Datasets)

2024-01-18 Thread Tzu-ping Chung
AIP page: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-60+Standard+URI+representation+for+Airflow+Datasets Discussion thread: https://lists.apache.org/thread/rf6c80ljjkml0l15h2jys7k713q3os1d Reaction on the proposal seems to mostly positive, with most comments around what documentati

Re: [DISCUSS] Common.util provider?

2024-02-21 Thread Tzu-ping Chung
It would help in the sense mentioned in previous posts, yes. But one thing I want to point out is, for the provider to actually be helpful, it must be treated a bit differently from normal providers, but more like a separate third-party dependency. Specifically, the provider should not have a de

Re: [DISCUSS] Common.util provider?

2024-02-22 Thread Tzu-ping Chung
hat might lead to a very lightweight approach where we introduce >> new common functionality which is immediately reusable in providers without >> the hassle of taking care about backwards compatibility, and managing the >> "common.util" provider. At the expense of a bit co

Re: [VOTE] January 2024 PR of the Month

2024-02-26 Thread Tzu-ping Chung
I want to nominate #36797.[1] This one has been a pretty popular request as well.[2] arguably dating even further back since dynamic task mapping was first launched. Nobody bothered to put in the effort until Ryan stepped up. January was a month with multiple such stories. Two are already nomin

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

2024-02-26 Thread Tzu-ping Chung
This kind of ties back to Bolke’s Operator Must Die manifesto—you shouldn’t use the operator class here at all in this situation, but the corresponding hook instead. Regarding preventing this anti-pattern, I can think of two ways: Add a check in BaseOperator to detect whether __init__ is called

Re: [VOTE] AIP-64: Keep TaskInstance try history

2024-03-25 Thread Tzu-ping Chung
+1 binding. This is something we should just do even outside of the DAG versioning context (which we also should do, to be clear). TP > On Mar 26, 2024, at 02:12, Aritra Basu wrote: > > +1 (non-binding) > This is quite good, I'd thought in passing that it'd be useful to have. > > -- > Regar

Re: [DISCUSS] AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage

2024-06-08 Thread Tzu-ping Chung
On Git (and other VCS for the matter) specifically, I believe respec only supports GitHub because it uses the GitHub API instead of Git. I’m only guessing, but using Git for random access would have terrible performance and is likely not an option for them. Airflow does not have the same proble

Re: [DISCUSS] AIP-78 scheduler-managed backfill

2024-07-09 Thread Tzu-ping Chung
How does the user cancel or pause the entire backfill process? The proposal only says this should be possible, but does not touch on how exactly. My intuition while reading the document was to have a flag on BackfillRun, but that does not seem to be the case in your illustrative code. TP > On

Re: More ruff style rules

2024-07-15 Thread Tzu-ping Chung
+1 to enforcing E731, using lambdas is objectively worse in every way except you save two lines and a few characters at most. The lambda is only “needed” in the sense that you can’t do (say) functool.partial due to variable scoping. A nested function does exactly the same thing. -- Sent from my

Re: Operator Templating in Airflow 3

2024-07-16 Thread Tzu-ping Chung
obj for that? Oh, yes, I thought of something a bit different but forgot to put it in, like StringTemplate("...", converter=bool). It is probably better this way since we can render either a bool or a int from either a string or file. I’ll add it to the document. > > Bas &

Re: Operator Templating in Airflow 3

2024-07-16 Thread Tzu-ping Chung
Good idea. Personally I think changing the import might be easier to read, but having a DAG policy is better if the goal is to rewrite DAG files as little as possible. Not sure if that’s a goal I’d agree with, but that’s not my decision to make 😛 I’ll find a way to add it to the document. >

[DISCUSS] AIPs 73 74 & 75: Data Awareness with Assets

2024-07-17 Thread Tzu-ping Chung
Hi all, We have (finally) stabilised the proposal around assets, and would like to propose them formally for discussion. Data Awareness is a very broad topic, and we plan to do a lot of things in it. Therefore, AIP-73 is made an umbrella AIP that discusses the topic at a very high level, and o

[DISCUSS] AIP-80: Explicit Template Fields in Operator Arguments

2024-07-19 Thread Tzu-ping Chung
(WAS: Operator Templating in Airflow 3) Hi all, Following comments on the Google Doc, and some experiments on my end, I’ve revised the document a bit, and created an entry in Confluence as AIP-80: Explicit Template Fields in Operator Arguments https://cwiki.apache.org/confluence/display/AIRFLOW

[VOTE] AIPs 73 74 75: Data Awareness with Assets

2024-07-22 Thread Tzu-ping Chung
Hi all, Since feedback has died down on these AIPs, I’m calling for a vote on the first three Data Awareness AIPs listed above. AIP-73 Expanded Data Awareness https://cwiki.apache.org/confluence/x/TAyTEg AIP-74 Introducing Data Assets https://cwiki.apache.org/confluence/x/QQ2TEg AIP-75 New Ass

[VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-07-25 Thread Tzu-ping Chung
Hi all, I’m calling for a vote on AIP-80: Explicit Template Fields in Operator Arguments. https://cwiki.apache.org/confluence/x/2grOEg This proposal aims to improve how Airflow defines template fields, and help users avoid annoying pitfalls currently exist. Discussion thread: https://lists.apa

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-07-25 Thread Tzu-ping Chung
n. If we leave it to the users > and do not provide suitable process/tooling/incremental approach they could > take, this one might be a huge blocker (because of the risk that the users > will have to take during migration). > > J. > > On Thu, Jul 25, 2024 at 10:00 AM Aritra Basu

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-07-25 Thread Tzu-ping Chung
I understand the concern, but would also wonder that a code base, be it user DAGs or third-party operators, lacks support in such a way, they would be correct to not encourage users to upgrade anyway. Airflow 3 is already going to have various breakages, some of them significantly less obvious t

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-07-26 Thread Tzu-ping Chung
I considered breakages that are likely to happen for a third-party operator. Without actually looking into them, from experience, they generally need to do one of the following two things (or both) in the task: 1. Access the templated values in execute() 2. Access the arguments containing a temp

Re: [DISCUSS] Approaches to bugfixes for 2.10 after main becomes 3.0

2024-07-26 Thread Tzu-ping Chung
P mentioned that he knew of workflows used by some existing projects that we > can compare and consider as an option. Could you expand on that, @Tzu-ping > Chung <mailto:t...@astronomer.io> ? > > If anyone has other ideas or thoughts, let's discuss them here. > > Regards, > Kaxil >

[RESULT][VOTE] AIPs 73 74 75: Data Awareness with Assets

2024-07-28 Thread Tzu-ping Chung
The vote for AIPs 73, 74, and 75 has passed. Binding +1 votes: - TP Chung - Marciej Obuchowski - Phone Kumar - Jarek Potiuk - Wei Lee - Utkarsh Sharma - Vikram Koka - Kaxil Naik - Jens Scheffler - Shahar Epstein - Brent Bovenzi - Ephraim Anierobi - Utkarsh Sharma Non-binding votes: - Aritra Basu

[DISCUSS] AIP-76 Asset Partitions

2024-07-28 Thread Tzu-ping Chung
Hi all, Following the acceptance of previous AIPs, let’s go through the last AIP on assets targeting Airflow 3.0. Note that this AIP is a part of AIP-73, depending on 74 and 75. Discussion of previous AIPs https://lists.apache.org/thread/6rp4jhflwg3czhtvjszoctdry85vfv8r AIP-76 Asset Partitions

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-07-29 Thread Tzu-ping Chung
>> just develop providers in "Airflow 3" mode that will keep automatically >> compatibility for Airlfow 2 (for a long-ish time) - I'd change my vote from >> +0.5 to +1. That would alleviate all my concerns. >> >> On Fri, Jul 26, 2024 at 5:49 PM Sh

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-07-29 Thread Tzu-ping Chung
I have updated the AIP to include the additional compatibility discussions in this thread. Please take a look again. Specifically (although by no means exclusively) it would be awesome if Michał you could have a look and see if it addresses more of the concerns and could be viable for you. Alth

[DISCUSSION] AIP-83 Remove Execution Date Unique Constraint from DAG Run

2024-07-29 Thread Tzu-ping Chung
Hi all, I have published a draft AIP to remove the unique constraint on the execution_date field of table dag_run in the meta database. This is a long-planned todo item from AIP-39 (where we introduced logical date and make DAG runs unique on run_id), and we can finally do this compatibility-br

Re: [DISCUSSION] AIP-83 Remove Execution Date Unique Constraint from DAG Run

2024-07-30 Thread Tzu-ping Chung
arek Potiuk wrote: > >> +1. Yes. That's cool. >> >> On Tue, Jul 30, 2024 at 12:34 AM Vikram Koka >> >> wrote: >> >>> Completely agreed on this. >>> >>> +1 binding >>> >>> On Mon, Jul 29, 2024 at 3:19 PM T

[VOTE] AIP-83 Remove Execution Date Unique Constraint from DAG Run

2024-07-31 Thread Tzu-ping Chung
Hi, This is the actual vote thread for AIP-83 AIP-83 Remove Execution Date Unique Constraint from DAG Run https://cwiki.apache.org/confluence/x/SA3OEg Discussion thread: https://lists.apache.org/thread/8rokf8ogp7g5pf0koqksxhcwqlhr9217 As mentioned before, I’m adding the following +1 votes from

Re: [VOTE] AIP-83 Remove Execution Date Unique Constraint from DAG Run

2024-07-31 Thread Tzu-ping Chung
Oops, forgot to attach an end date /facepalm Let’s do the minimum since there seems to be not much concerns behind this. The vote will run until 4th Aug 2024, 6:09 UTC. TP > On 1 Aug 2024, at 14:08, Tzu-ping Chung wrote: > > Hi, > > This is the actual vote thread for AIP

[VOTE] AIP-76: Asset Partitions

2024-08-01 Thread Tzu-ping Chung
Hi all, Kicking start the vote on the final data awareness AIP targeting 3.0. AIP-76 Asset Partitions https://cwiki.apache.org/confluence/x/2QyTEg Discussion thread https://lists.apache.org/thread/3d80921801j6p23x3xgq6yy09gy8yvm4 The vote will run until Thursday, 2024-08-08, 08:00 UTC. This mak

Re: [DISCUSSION] AIP-83 Remove Execution Date Unique Constraint from DAG Run

2024-08-01 Thread Tzu-ping Chung
Replied inline > On 2 Aug 2024, at 03:52, Daniel Standish > wrote: > > Hi @Tzu-ping Chung <mailto:t...@astronomer.io> > > I just had a chance to look over the AIP after noticing that the vote thread > started. > > I have a few questions / things to note.

Re: [VOTE] AIP-83 Remove Execution Date Unique Constraint from DAG Run

2024-08-04 Thread Tzu-ping Chung
o...@gmail.com> >>> <mailto:vish.chiluko...@gmail.com>> >>>> wrote: >>>>> >>>>> +1 non-binding >>>>> >>>>> On Thu, Aug 1, 2024 at 2:22 PM Ankit Chaurasia >>>> <mailto:sunank...@gmail.com

[RESULT][VOTE] AIP-83 emove Execution Date Unique Constraint from DAG Run

2024-08-04 Thread Tzu-ping Chung
Hi all, The vote for AIP-83 has passed. AIP-83 Remove Execution Date Unique Constraint from DAG Run https://cwiki.apache.org/confluence/x/SA3OEg Discussion thread: https://lists.apache.org/thread/8rokf8ogp7g5pf0koqksxhcwqlhr9217 Vote thread https://lists.apache.org/thread/wkjl9prj274psjqm32pqvg

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-08-04 Thread Tzu-ping Chung
nt from Outlook for iOS<https://aka.ms/o0ukef> > ____ > From: Tzu-ping Chung > Sent: Monday, July 29, 2024 8:54:58 PM > To: dev@airflow.apache.org > Cc: michalmod...@google.com > Subject: Re: [VOTE] AIP-80: Explicit Template Fields in Ope

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-08-07 Thread Tzu-ping Chung
l be able to run part of your DAGS with >> "old ways" and gradually converting your DAGs to "new ways", ability to >> track the progress and "nagging warnings" - and you have a converter that >> works "out-of-the-box" for "almost&qu

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-08-07 Thread Tzu-ping Chung
:)), it actually will discentivise some users from moving to >> Airflow 3 entirely. If we care for Airflow 3 adoption and not putting >> blockers for users for it, we should go with 2). I believe it should be at >> least a provider and maintained. >> >> >> On Wed, Au

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-08-08 Thread Tzu-ping Chung
is not going > to receive new features, and the new syntax should not break compatibility > for until the theoretical 4.0. > > I read the above statement as "yes we are adding the "Airflow 2 operators > and DAGs working with Airflow 3" compatibility layer as part of the AI

[RESULT][VOTE] AIP-76: Asset Partitions

2024-08-08 Thread Tzu-ping Chung
Hi all, The vote for AIP-76 has passed. AIP-76 Asset Partitions https://cwiki.apache.org/confluence/x/2QyTEg Discussion thread https://lists.apache.org/thread/3d80921801j6p23x3xgq6yy09gy8yvm4 Vote thread: https://lists.apache.org/thread/g2c83wsrx1fy04v7kjzh65smykpf6c26 Binding +1: - TP Chung -

Re: [RESULT][VOTE] Release Airflow 2.10.0 from 2.10.0rc1

2024-08-15 Thread Tzu-ping Chung
I think we need to get this backport into 2.10 https://github.com/apache/airflow/pull/41469 > On 16 Aug 2024, at 03:14, Utkarsh Sharma > wrote: > > Hello, > > Apache Airflow 2.10.0 (based on RC1) has been accepted. > > 3 "+1" binding votes received: > - Kaxil Naik > - Jed Cunningham > - Eph

Re: [RESULT][VOTE] Release Airflow 2.10.0 from 2.10.0rc1

2024-08-15 Thread Tzu-ping Chung
obi wrote: > > Looks like the PR is late. Is it ok if we include it in the next Airflow 2 > release? > > On Thu, 15 Aug 2024 at 20:29, Tzu-ping Chung > wrote: > >> I think we need to get this backport into 2.10 >> >> https://github.com/apache/airflow/pull/

Re: [RESULT][VOTE] Release Airflow 2.10.0 from 2.10.0rc1

2024-08-15 Thread Tzu-ping Chung
2.11.0 if it can't be in a patch release. > > On Thu, 15 Aug 2024 at 21:07, Tzu-ping Chung > wrote: > >> I think this needs to be in Airflow 2 before 3.0 is released so people >> actually receive the notification for the breaking change we’re going to >> make

Re: [DISCUSS] New provider Common.time

2024-08-16 Thread Tzu-ping Chung
Random idea, how about standard? Like how we can Python’s stock libraries standard libraries. > On 16 Aug 2024, at 22:19, Elad Kalif wrote: > > What about primary provider? > > בתאריך יום ו׳, 16 באוג׳ 2024, 16:49, מאת Jarek Potiuk ‏: > >> I also think "core" is not the best one as we are usi

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-08-19 Thread Tzu-ping Chung
etc.) - here's where our views > differ. I think the discussion should be continued with more stakeholders > in the Airflow 3 dev calls. > > On Thu, Aug 8, 2024 at 11:12 AM Tzu-ping Chung > wrote: > >> The topic here are TWO compatibility layers in this mess

Re: [VOTE] AIP-76: Asset Partitions

2024-08-26 Thread Tzu-ping Chung
I meant to do this last week but totally forgot. Since no-one really expressed any more concerns, I’ll finish the process and declare the results in a new thread. TP > On 5 Aug 2024, at 14:15, Ephraim Anierobi wrote: > > +1 (binding) > > On Sun, 4 Aug 2024 at 21:14, Jarek Potiuk wrote: >

[RESULT][VOTE] AIP-80: Explicit Template Fields in Operator Arguments

2024-08-26 Thread Tzu-ping Chung
Hi, The vote for AIP-80: Explicit Template Fields in Operator Arguments has concluded successfully. The proposal has been accepted. Since there’s an update midway through the vote, I’ll calculate votes in two parts. If you voted in more than once, only the last one counts. (Sorry I really mean

Re: Naming things: What should the imports in dag files for DAG etc. be?

2024-08-30 Thread Tzu-ping Chung
> On 30 Aug 2024, at 17:48, Ash Berlin-Taylor wrote: > > Where should DAG, TaskGroup, Labels, decorators etc for authoring be imported > from inside the DAG files? Similarly for DagRun, TaskInstance (these two > likely won’t be created directly by users, but just used for reference > docs/t

Re: PIP 21 officially supported for next version of Airflow

2021-04-27 Thread Tzu-ping Chung
> On 27 Apr 2021, at 16:16, Jarek Potiuk wrote: > > Big thanks to Tzu-ping Chung from the PIP team for help and useful > discussions. We are one of the more complex projects with > 400 > dependencies and I hope we will help to drive PIP 21 adoption :). I have some resolver

Re: [DISCUSS] Using semantic/conventional commits in Airflow?

2021-06-13 Thread Tzu-ping Chung
There is also a tool called commitizen that helps teams standardise commit messages. There is a Python implementation: https://commitizen-tools.github.io/commitizen/ I’m not a huge fan of semantic commit messages and never personally used any tooling for it, but one of commitizen’s co-maintain

Re: [DISCUSS] Using semantic/conventional commits in Airflow?

2021-06-14 Thread Tzu-ping Chung
cs.openstack.org/reno/latest/ This also natively handles our cheery pick process, and also gives us the ability to edit changelogs if we notice a typo for instance. -ash On 14 June 2021 03:39:50 BST, Tzu-ping Chung wrote: There is also a tool called commitizen that helps teams standardise

Re: [VOTE] Disable pylint ?

2021-06-24 Thread Tzu-ping Chung
Pylint has always been a mix-bag to me. There are a lot of good checks that help prevent intermediate level errors, but there are also too many opinionated checks enforcing superfluous metrics (some I’d even say wrong, e.g. too-few-public-methods and too-many-return-statements). Airflow’s Pyth

AIP 39: schedule_date renamed to logical_date

2021-08-03 Thread Tzu-ping Chung
Hi all, I want to give a heads-up on a minor modification I just made to AIP-39. AIP-39 originally proposed renaming execution_date to schedule_date since the old name was confusing (it’s not when the DAG run is actually executed). However, while implementing the AIP and drafting documentation t

Re: AIP 39: schedule_date renamed to logical_date

2022-01-31 Thread Tzu-ping Chung
for the first time users, I just wanted to let out my > opinion on that, and propose something different. > Hope this helps, > Howard > > On 2021/08/03 09:35:15 Tzu-ping Chung wrote: > > Hi all, > > > > I want to give a heads-up on a minor modification I just ma

Re: [DISCUSS] deprecate `days_ago` dates utility

2022-01-31 Thread Tzu-ping Chung
I was brought here by https://github.com/apache/airflow/pull/20508#issuecomment-1026414890 Also +1 to deprecation from me. Since the function cannot be safely used in start_date and end_date, the only sensible way for a general, non-advanced user to use the function is in a task Python callable

Re: Dynamic Workflows Queries

2024-10-09 Thread Tzu-ping Chung
I believe both are impossible. Airflow is designed to run statically defined workflows. The triggering of the workflow can be dynamic, and each step of the workflow can be dynamic, but the overall structure of the workflow must be static; you cannot add things or change their ordering on the fly

Re: Airflow should deprecate the term "DAG" for end users

2024-10-21 Thread Tzu-ping Chung
I totally agree with doing away with the term DAG. The only problem (aside from actually telling people—including myself—to stop using the term) is to come up with a reasonable alternative. I can’t recall who, but someone mentioned “workflow” is not very accurate for Airflow. The term “definiti

Re: Naming things: What should the imports in dag files for DAG etc. be?

2024-09-17 Thread Tzu-ping Chung
herwise we will have to have some ugly conditional >>>>> code - >>>>>>> when you have and when you do not have database access. >>>>>>> >>>>>>> As an example - If we go for "airflow.sdk

Re: [LAZY CONSENSUS] Use Ruff as the official linter to detect Airflow 3 incompatibilities in user code

2024-11-24 Thread Tzu-ping Chung
The lazy consensus has passed. I am currently working on a step-by-setp writeup on how to implement a Ruff rule for Airflow migration. Thanks all, TP - To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional com

Re: [ANNOUNCE] Cherry Picker Automation Now in GitHub Actions!

2024-11-18 Thread Tzu-ping Chung
What happens if a back port cannot be created automatically? (Or asked another way, how do I know when I need to manually do the back port PR?) > On 18 Nov 2024, at 20:35, Pavankumar Gopidesu wrote: > > Hello All, > > We have set up Cherry Picker Automation that supports both automated > and

Re: [DISCUSS] Remove Deprecated Context Items in Airflow 3

2024-12-04 Thread Tzu-ping Chung
We already removed those entries in the deprecated section as a part of AIP-83. Those are also planned in the Ruff migration rule so people should be notified if they use them in a PythonOperator (including @task). There are actually some other template variables I feel we should remove (ds and

Re: [DISCUSS] Changing how XCom keys are changed using `.output`

2025-02-06 Thread Tzu-ping Chung
I’ve always hated the magic __getitem__ interface. My previous ideas include: a. Make task.output an accessor (say XComCollection) instead of an XComArg. This accessor type uses __getitem__ to let you specify a key. task.output[None] references the default XCom (what you get right now with task.

[LAZY CONSENSUS] Use Ruff as the official linter to detect Airflow 3 incompatibilities in user code

2024-11-21 Thread Tzu-ping Chung
Hi all, I am calling a LAZY CONSENSUS to officially anoint Ruff as the tool to implement lint rules that detect potential incompatibilities in existing user code (including DAG files and plugins) that may break in Airflow 3. The rationale is * Ruff provides the best set of features for our nee

Re: [REMINDER]: Airflow 3 Dev call on 5th December - agenda

2024-12-03 Thread Tzu-ping Chung
Hi, Since the dev call is quite late in my timezone, I’ll share a recording for the update before the meeting. Same as last time for AIP-74. TP > On 4 Dec 2024, at 08:13, Vikram Koka wrote: > > Hi everyone, > > Here is a quick reminder that we have our Airflow 3 dev call the day after > tomo

Re: [VOTE] Amendment to AIP-83 re uniqueness of logical date

2025-01-21 Thread Tzu-ping Chung
+1 from me > On 22 Jan 2025, at 00:22, Daniel Standish > wrote: > > Ok as described in discussion > thread > , I am > calling for a vote on amendments to AIP-83.

Re: [DISCUSS] confusing alert re SimpleAuthManager

2025-03-27 Thread Tzu-ping Chung
Username and password being always the same is also a problem; username is viewable as plain text in the UI and things like password managers. > On 28 Mar 2025, at 02:56, Vincent Beck wrote: > > Is the security issue only printing out the passwords in stdout? If yes, I > can easily remove tha

Re: Discuss: lets' stop posting on X and muffle our account there.

2025-03-14 Thread Tzu-ping Chung
I disagree with the deleting post part. While it is legitimate to want to distance from X, the history of association exists, and erasing the history is self preservation at the expense of erasing internet history and potential users that rely on a past permalink. It is better to preserve the hi

Re: Two Hard Things: Deadline Alerts Edition

2025-02-24 Thread Tzu-ping Chung
I like StartsFrom, but “DeadlineStartsFrom” sounds like when the deadline is. It should have another word in front. DeadlineReference sounds reasonable to me. Something like DeadlineBase or DeadlineBasis could be possible too, although I actually like reference better. I kind of feel DeadlineMe

Re: [DISCUSS] Python 3.13 support

2025-02-20 Thread Tzu-ping Chung
The timezone conversion in stdlib is an intentional design choice. Not converting date times to matching timezones is a giant footgun that fires when you least expect it to. I did not like the design until I had the chance to listen to Paul Ganssle explain his decisions, and have to admit he mad

Re: Should we drop support for pre_execute & post_execute for AF 3.0

2025-03-28 Thread Tzu-ping Chung
Passing post_execute as an argument is somewhat useful for operators that don’t support assets natively (most of them) but when you want to emit a dynamic path. For example: def _send_asset_event(context, result): # Rendered value! name = context["task"].output # Trigger an event aga

Clean up the task context dict

2025-03-25 Thread Tzu-ping Chung
Hi all, There was an issue a while ago proposing to add more values into the context dict.[1] I responded at the time that the proposed values are already available by accessing 'ti' although it’s unclear whether the internals of the variable (which was a concrete SQLAlchemy model TaskInstance

Re: [DISCUSS] Special Terms in UI Translations?

2025-05-22 Thread Tzu-ping Chung
The problem is, the line whether a term is technical or not is blurry at best. Dag—sure XCom—yeah Asset—probably Variable—hmm Connection—hmm? Running—This has a specific meaning to Airflow tasks, but are you sure you don’t want to translate it? If you keep going down the slope, you end up with a

Re: [VOTE] May 2025 PR of the Month

2025-05-29 Thread Tzu-ping Chung
My vote is also 50626, but special shoutout to 50924. It would have won if opened any other month! TP -- Sent from my iPhone > On 30 May 2025, at 09:46, Ferruzzi, Dennis > wrote: > > +1 for 626, and not just because of the Lilo and Stitch reference 😝 > > > - ferruzzi > > > _

Re: [VOTE] June 2025 PR of the Month

2025-06-27 Thread Tzu-ping Chung
Mine goes to 51153. Overdue doc work and I’m so happy we finally have proper documentation so people don’t need to read the source code (and end up using things they shouldn’t). This has been a problem even before the task sdk exists. TP -- Sent from my iPhone > On 27 Jun 2025, at 18:33, Wei

Re: [DISCUSS] Use ~= for python requires?

2025-07-07 Thread Tzu-ping Chung
ot recommended - that was the first time I hear it, > but If I know that Python Packaging actually made that recommendation, I > would be more than happy to just switch and not make any fuss about it. > >> On Mon, Jul 7, 2025 at 1:12 PM Tzu-ping Chung >> wrote: >> >>

Re: [DISCUSS] Use ~= for python requires?

2025-07-07 Thread Tzu-ping Chung
Note that while, as mentioned, ~= is a standard and well-supported operator, it is generally NOT RECOMMENDED to be used for Python version specification. It is also not recommended to use the equivalent expression composed with >= and <. Python does not use semantic versioning, and backward inco

Re: [DISCUSS] Consistent test assertion style: pytest-native vs unittest-style

2025-07-09 Thread Tzu-ping Chung
Personally I dislike things like assert_called_once_with etc. since they are easy to miss when you scan a test to see what they are trying to check. An 'assert' keyword stands out (it’s always the first word in the line), especially with syntax highlighting. I do agree the proposed Pytest style

Re: [DISCUSS] Consistent test assertion style: pytest-native vs unittest-style

2025-07-10 Thread Tzu-ping Chung
ote: > > One tiny comment regarding TP's suggestion - IMHO it's better to avoid > `unittest.mock` in favor of the equivalent `mocker` fixture provided by > `pytest-mock`. > > On 2025/07/10 06:30:22 Tzu-ping Chung wrote: > > Personally I dislike things like assert_call