Re: Setting to add choice of schedule at end or schedule at start of interval

quot;cron with > a UI", so we should try to be pragmatic and fit in power-user features > where we can do so without compromising the project's overall goals. > > On Wed, Aug 28, 2019 at 2:24 PM Daniel Standish > wrote: > > > I am just thinking there is the pote

Re: Setting to add choice of schedule at end or schedule at start of interval

circuit > operators over and over. It's a hassle. Allowing for scheduling dags > at the left edge of an interval and allowing it to behave more like > cron, where it runs at the time specified, not schedule + interval, > would make things much less complicated for users like myself that

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

I like putting them with source. On Sat, Sep 21, 2019, 12:57 PM Philippe Gagnon wrote: > Another way to resolve this could be to define a convention such that a > "cross transfer" operator should belong to either the source or destination > cloud provider's package. > > Personally though I do no

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

One case popped up for us recently, where it made sense to make a MsSql *From*S3Operator . I think using "source" makes sense in general, but in this case calling this a S3ToMsSqlOperator and putting it under AWS seems silly, even though you could say s3 is "source" here. I think in most of these

Re: Can an xcom push and pull to the same task instance on the same execution date?

In current releases xcom is deleted at task start. Recently a PR was merged to not delete until XCOM is again pushed I am not sure if this has made it into 1.10.6rc After this change, I think what you are trying to do would work. But it's interestin

Re: [VOTE] AIP-21 update for Airflow 1.10.* backportability

Re For example S3 -> GCS should be in "google" provider, but GCS-> S3 should > be in "amazon > So if there were a BigQueryToS3 or SnowflakeToS3 operator, would you put this in AWS? I feel like storage should be a secondary consideration concerning object naming. Using snowflake as an example, w

Re: Airflow DAG explodes with RecursionError when triggered

Did you check the max_active_runs param in your dag? On Mon, Dec 23, 2019, 11:47 PM Shubham Gupta wrote: > Hello maintainers > > A programmatically generated DAG of mine is exploding (the bomb > text-picture) when triggered with following stack-trace > > File > "/home/admin/.pyenv/versions/3.7.3

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

I also am a big fan of adding better support for stateful tasks, though I know this is a thorny subject in airflow community. There are many data warehousing tasks where state makes a lot of sense. While idempotence is a nice design pattern it's not the solution for every problem. XCom may not be

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

jan. 2020 om 04:21 schreef Kaxil Naik : > > > > > Hey all, > > > > > > Really good document Jacob. > > > > > > Below are my thoughts on different topics discussed in the docs and the > > > mailing list: > > > > > > > &g

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

To banish anything stateful seems arbitrary and unnecessary. Airflow is more than its canonical task structure: hook / operator framework and ecosystem, scheduling, retry, alerting, distributed computing, etc etc etc etc. As long as support for the canonical task is preserved, what's the harm in

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

@Jarek & Fokko Firstly, thank you for the thorough and thoughtful replies -- I appreciate it. I recognize that the execution_date-based idempotence pattern is core to airflow. I understand that it motivated a lot of the design decisions, and that it is a pattern that is endorsed and explicitly en

Re: Stateful Tasks (was Poke Reschedule)

Airflow tasks as stateless as possible > but > > > from my perspective real world systems are not stateless (or without > > side > > > effect) and modeling that state is sometimes the only viable solution. > > The > > > motivation of this thre

[DISCUSS] AIP-32 Creds Backend

We can currently retrieve connections from environment variables or the metastore database. AIP-33 provides a way to retrieve them from other sources, for example AWS SSM parameter store. There are many instances in airflo

Re: [DISCUSS] AIP-33 Creds Backend

vailable at different times > than when connection is retrieved. Often secrets are short-living and they > should be retrieved "just" before they are used - limiting it only to when > the Hook is created and only to URI is quite a limitation I think. > > Again - feel free to d

Re: [DISCUSS] AIP-33 Secrets Backend

I think is using Environment Variable. >Of course (2) can solve this, for example, we can pass >*GOOGLE_APPLICATION_CREDENTIAL* JSON file path to > *creds_backend_kwargs *which >would then authenticate with GCP >Or do we want to get that value from* Connection object s

Re: [DISCUSS] AIP-33 Secrets Backend

PR has been updated with simplified config as suggested by Kaxil: [secrets] backend_name = backend_kwargs = Now we can only specify one secrets backend. If an alternative backend is specified, search path is alternative > env vars > metastore. Otherwise it is env vars > metastore. No other co

Re: POC - Kubernetes and Airflow

I think it is good to start simple. I would start out using a single machine, use local executor, running on docker, using docker compose, with the puckel image: https://github.com/puckel/docker-airflow (or your own customization thereof). I would use a cloud database running postgres, e.g. on RD

Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection balancing HA )

+1 (non-binding) :) On Fri, Apr 24, 2020 at 10:00 AM Xinbin Huang wrote: > +1 non-binding > > Best, > Bin > > On Fri, Apr 24, 2020 at 9:20 AM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> > wrote: > > > +1 binding > > > > Tomek > > > > > > On Fri, Apr 24, 2020 at 6:15 PM QP Hou wrote: > > >

Re: Setting to add choice of schedule at end or schedule at start of interval

Inspired by James, I tried this out... For others interested, here is sample dag to test it out: class MyDAG(DAG): def following_schedule(self, dttm): pen_dt = pendulum.instance(dttm).replace(second=0, microsecond=0) minutes = pen_dt.minute minutes_mod = minutes % 10

Re: Airflow Dev on Windows using WSL2

I too was looking forward to trying WSL 2 because we have some windows users. One thing to keep in mind with WSL 2 is that it does not like to share files with windows. Here is a doc on the differences: https://docs.microsoft.com/en-us/windows/wsl/compare-versions And a note specifically on when

Re: Airflow Dev on Windows using WSL2

ies not in Windows directories. Assuming WSL 2 becomes > > popular it seems likely that more dev tools will support this way of > > working. I had a quick test of pycharm by pointing it to the automatic > > WSL2 network share but it seems that it doesn't support all pycharm &

Re: [DISCUSS] Naming of the transfer operators/Hooks

I also vote [1]: S3ToHiveOperator Transfer is redundant. And why not 3 --- as of now there is no such distinction and I am not convinced it is justified. As of now, this is just another operator. Probably most operators do some kind of "transferring" of data and trying to decide what is transfer a

Re: [DISCUSS] Enable 'Black' for Auto Code Formatting

+1 no string normalization On Wed, Jul 1, 2020 at 3:20 PM Kamil Breguła wrote: > Hello, > > I would also prefer to make this change after the release of Airflow 2.0. > > I would like to suggest that we use black without normalizing strings. In > my opinion, using two apostrophes in one file styl

Re: Defining Airflow idempotence

We should be careful not to treat every line in the docs as "constitution" -- i.e. as a commandment. And in the docs, I think we would be better off if we more clearly distinguished (1) the description of what *is* from (2) opinion about what *should be.* *This line should be chopped* Case in po

Re: [VOTE] Release Airflow 2.4.0 from 2.4.0rc1

+1 (non-binding) On Sun, Sep 18, 2022 at 6:57 PM Phani Kumar wrote: > + non binding. Tested the Dataset functionality and working fine > > On Sun, Sep 18, 2022 at 11:09 PM Ephraim Anierobi < > ephraimanier...@gmail.com> wrote: > >> +1 (binding) >> >> On Sun, 18 Sep 2022 at 16:57, Jeambrun Pierre

[DISCUSS] string normalization in black

Black, our python formatter, can "normalize" strings to prefer double quotes, and we disable this feature. I have always been a single quotes person unless using f-string and supported disabling normalization when we introduced black. But lately, black’s string normalization has seemed more appea

Re: [DISCUSS] string normalization in black

> > I'll weigh in on this most important of decisions :) OK but I must clear up one thing ... if we turn on string normalization, we do not get to choose single vs double -- with black, there is only one way, and it is double. Personally I have always liked single but yeah, I am in favor of tur

Re: [DISCUSS] string normalization in black

OK seems like all are in favor. Do we even need a vote? I guess lazy consensus still applies even when you don't formally call for a lazy consensus vote. So I reckon I won't bother. But the implementation to @Jed Cunningham 's point we can wait until closer to 2.5. And I'll connect with fo

Re: [DISCUSS] string normalization in black

> > Opinion: Adding a lazy consensus thread will not hurt and I think it > should happen Sure, can do >

[LAZY CONSENSUS] enable string normalization on black

Proposal: remove param --skip-string-normalization on black pre-commit hook Discussion thread: https://lists.apache.org/thread/0g887n3gyr611k908zsz27ccshb85z2n After 72 hours, unless there is an objection, this proposal will be adopted. What is lazy consensus? Read here

Re: [VOTE] September 2022 PR of the Month

One vote for https://github.com/apache/airflow/pull/26400 (improved test command) On Tue, Sep 27, 2022 at 10:50 AM Jed Cunningham wrote: > My write-in is ExternalPythonOperator: > https://github.com/apache/airflow/pull/25780 >

Re: [VOTE] Release Airflow 2.4.1 from 2.4.1rc1

+1 (binding) verified signatures and licenses, checksum and ran sample dag On Fri, Sep 30, 2022 at 11:21 AM Jed Cunningham wrote: > Kind bump, I need 1 more vote on this release. >

Re: [ANNOUNCEMENT] Breeze rewrite to Python finally completed !!! Congrats to Bowrna, Edith (and Elad as mentor)!

Nice, congrats 👏 On Mon, Oct 3, 2022 at 3:12 PM Jarek Potiuk wrote: > Hello Airflow community, > > I have just merged the last PR from the "Breeze rewrite to Python" > project, completing the project we started with Bowrna and Edith as > Outreachy interns and Elad co-mentoring it with me. Congr

Re: [VOTE] Airflow Providers prepared on October 04, 2022

+1 (binding) checked signatures checksums and licenses On Wed, Oct 5, 2022 at 3:31 PM Jed Cunningham wrote: > +1 (binding) > > Checked signatures, checksums, and licences. >

[DISCUSS] no extra prefix required

As of airflow 2.3, airflow hooks no longer need to name extra fields with the `extra__conn_type__field_name` convention (PR https://github.com/apache/airflow/pull/22607). I believe using the short name is much more intuitive and user-friendly. So I am currently working on updating our hooks so the

Re: Possible conflicts after string normalization in providers

Thanks Jarek There will be a couple more notices like this as we apply to the rest of the codebase. Non-providers / non-core is coming soon, and core will be applied closer to 2.5. And if you don't want to squash to one commit, I think you can also just do `pre-commit run black --all-files`, whi

Re: Possible conflicts after string normalization in providers

OK -- the non-providers/non-core (a.k.a. "other") group of string normalization has just been merged. So, if you already rebased, unlucky you -- you need to do it again. But fear not, you won't need to do it yet another time until closer to the release of airflow 2.5, when we'll apply string norm

Re: [PROPOSAL] Clarifications of triage team role including strenghtening importance of active triaging

> > 3. I am also getting a true sense of just how overwhelming the influx of > Issues, PRs and Discussions is. I have come across several folks who > submitted PRs and never got feedback and then left the community. Losing > these folks is a bad experience for them but also for us because we lost >

Re: Possible conflicts after string normalization in providers

Ok -- just merged normalization for core. Now we're fully normalized. Expect conflicts in your branches (if they touch core airflow). Apologies in advance.

Re: [LAZY CONSENSUS] Do not treat min-airflow-version bump in providers as breaking

I think it makes sense and I'm a +1. For the convenience of other readers I'll paste your rationale here: The rationale i have - that from the point of view of provider, it's just a > dependency change (which we generally consided non-breaking) and it does > not break people's workflows in genera

Re: [VOTE] Airflow Providers prepared on November 15, 2022

Verified signatures, licenses, checksums. +1 (binding) do want to mention though that it looks like yandex and jdbc release notes still both mention 4.0 (which was an accident related to recent change in policy re major bump for min airflow version bump) but i defer to your judgment whether these

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

Hi, Is it not possible to just have unicode dag_id with no distinct "name"? If you explored this route and encountered problems which caused you to abandon, can you share what were the problems? I think having just one ID for a dag is a nice thing, if we can keep it. On Wed, Jan 11, 2023 at 11:

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

flow/issues/23020> > > *Abdul Hadi Shakir* > > > On Thu, Jan 12, 2023 at 1:19 PM Daniel Standish > wrote: > >> Hi, >> >> Is it not possible to just have unicode dag_id with no distinct "name"? >> If you explored this route and encountered pro

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

g_id (which > currently IMHO is not an option) my vote goes to 2. > > We can also drop MySQL support :D > > J. > > > > > On Thu, Jan 12, 2023 at 9:56 AM Ash Berlin-Taylor wrote: > >> +1 to what Daniel said >> >> On 12 January 2023 08:32:29 GMT, Daniel

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

Following up here... that PR has been merged At some point we should probably have a vote on that, if it's meant to be actual binding policy. Maybe we're still feeling it out? But "what is public" is a pretty fundamental concern to the project. Maybe such a policy itself should be an AIP? Me

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

ed contains external type annotations for the Python standard > library and Python builtins, as well as third party packages as > contributed by people external to those projects. > This data can e.g. be used for static analysis, type checking or type > inference. > ""&quo

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

t; > I believe I must have missed some important points in the conversation, or > even misunderstood some points. But just summarizing what I have understood > as well as what I would prefer. > > > Regards, > XD > > On Thu, Jan 26, 2023 at 9:45 AM Daniel Standish > wrote

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

okie doke, took a crack at it https://github.com/apache/airflow/pull/29200

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

Thinking about this some more. As an airflow developer, a lot of our backcompat concerns (which takes up a substantial portion of our energy), is about the concern that we might break something for someone who has extended either built-in classes or provider classes. Maybe we are to permissive in

[DISCUSS] AIP-52 updates - setup / teardown tasks

I’m part of a group working on the implementation of AIP-52. We would like to update the community on some changes to the implementation approach, the planned roadmap, and give an opportunity to provide feedback. First though, let’s recap briefly what are the main benefits of adding setup and tea

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

Hi, would like to clarify, in this thread we're specifically hoping to get community feedback on the proposal to drop the "implicit" logic. In the original AIP, if you instantiate a setup task in a group, in effect it's made the setup task for all tasks in the group. And the proposal up for discu

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

re > 2. `task1 >> task2 >> teardown_task` to me falsely implies that teardown > depends on task2, But it doesn't. It only depends on the "scope being > exited". So that's not quite the case. With the proposed implementation, there's no such scope concept. They're just normal tasks, with specia

Re: [DISCUSS] a cache for Airflow Variables

It would not help with kubernetes executor. Did you try with local executor? Another option to consider is to implement this specifically on the AWS secrets manager secrets backend itself.

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

The key thing for me: setup and teardown tasks are not normal tasks, and > shouldn't be used, nor appear, as such in DAG code. > *** > > Crucially I want us to not let perfect be the enemy of good, and all this > confusion and discussion is exactly why I had originally placed "

Re: [DISCUSS] a cache for Airflow Variables

Surprised to hear that it doesn't work with celery. Is that right? I assumed that this was the main target. If it's really only a benefit in dag processor, it's surprising that it provides much benefit because it should be one call per var-file-parse; in worker it will be once per ti and I assume

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

Thanks Elad for the feedback. re 1. i don't really see a problem with the trigger rule being public. The way I see it, it's another trigger rule like any other trigger rule. Every trigger rule behaves differently, that's true here too. This one happens to be relied upon for teardown tasks. That

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

at is not something he sees unless he checks the code of > >teardown and setup decorators. > > > >This means that users of ShortCircuitOperator will not even know they need > >to take action (until it wont work as expexted) and they will propbably > >start as as

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

we are discussing it now, so I think it is cool. > > J. > > On Mon, Mar 27, 2023 at 8:43 AM Ash Berlin-Taylor wrote: > > > > If the set-up ran then the tear down _must_ run. No question. > > > > Nothing should be able to change this fact. If you can, then they

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

Happy to see the engagement on this one. Thanks to everyone for thinking it through and contributing their thoughts. re niko > - Context managers: > I found most of the context manager syntax proposals a little hard to > understand, but some better than others. Ultimately if I put my DAG author

Re: [DISCUSS] Exclude some providers that hold us back from releasing

It seems reasonable to me. On Mon, Mar 27, 2023 at 12:02 AM Jarek Potiuk wrote: > Hello Everyone, > > TL;DR; I wanted to raise a discussion and make a proposal about option > to skip some niche providers of our from releasing if they are holding > us back, regarding the dependencies > > We are g

Re: [VOTE] April 2023 PR of the Month

This is a tough one. Many PRs and contributors deserving of recognition here. And cool to see so much engagement in the voting. But my vote goes to #30375... It may seem like a small fix but I think it (at least hopefully) should alleviate a lot of frustration and it's an example of someone, whos

Re: [VOTE][RESULT] April 2023 PR of the Month

Feels very appropriate 👏 On Fri, Apr 28, 2023 at 1:33 PM Michael Robinson wrote: > Hello all, > > Thanks to all who participated in this month’s unusually competitive vote > for PR of the Month. By my count, we have a three-way tie. The winners: > > #30705 by @potiuk: “Optimize parallel test exe

Re: [DISCUSS] Preparing for dropping Python 3.7 support

Thanks for the heads up about the timing re 2.7 @Jarek. I too am eager for the walrus operator.

Re: Huge fependency bump in main (thanks to Google efforts)

nice

Re: [ANNOUNCEMENT] Python 3.11 support merged for Apache Airflow

Congrats 🎉 On Mon, May 22, 2023, 3:54 PM Jarek Potiuk wrote: > Hello everyone, > > I am glad to announce that I just merged > https://github.com/apache/airflow/pull/27264 that implements Python 3.11 > support for Airflow. > > Python 3.11 brings a number of speed improvements for single-threaded

Re: [VOTE] May 2023 PR of the Month

you can have any author you like, as long as it is potiuk https://github.com/apache/airflow/pull/27264 On Fri, May 26, 2023 at 2:09 PM Hussein Awala wrote: > I join Bas and I vote for #27264. > > On Sat 27 May 2023 at 00:03, Bas Harenslak > wrote: > > > My vote goes to https://github.com/apac

Re: Deferrable operators in system tests, how?

I just took a look and it turns out that DebugExecutor works fine with triggerer you just need to have one running. You could run one in a subprocess. I experimented with refactoring the subprocess hook for this purpose (so you can start the subprocess asynchronously) and then ran this dag with d

Re: Deferrable operators in system tests, how?

Don't think fixture would break that. It would just be test code not in the dag. It would just ensure that the triggerer is running before the tests that use the triggerer need it. But doing it in breeze makes more sense for sure. Although I suppose a combination approach could be considered EG, if

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

Ok want to pick back up setup / teardown discussion as we get closer to 2.7. For personal reasons I had to take some time off work just as we were wrapping up work on 2.6, and at least partly due to that, we punted setup / teardown to 2.7. But we've picked it back up and continued along the path

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

You're not too late, and thanks for engaging with the issue. I don't think anyone would dispute that users will sometimes want a setup without a teardown. But the question is should we require that users explicitly make the scope of a setup well-defined. Like Jarek, I have some ambivalence on th

Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider

Just on the question of semver, I am not convinced that semver prohibits this. As a user, your concerns and expectations regarding semver are about essentially how the code works, e.g. are you going to have to refactor all of your 500 dags. In other words the API.. But to me this is a lower leve

Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider

> I guess the question is not "can we do it" but more "should we do it" :D right :)

Re: [DISCUSS] Should we pre-install celery/k8s providers?

Yeah I think this is an important point: > > Core Airflow so far hasn't had the dependencies > required to make those executors functional either - users either had to > use the extra or install the provider directly. So that doesn't really > change if we choose not to preinstall the providers. > A

Re: [DISCUSS] Should we pre-install celery/k8s providers?

* *provider* extras or provider optional features

Re: [DISCUSS] Should we pre-install celery/k8s providers?

Yeah it ain't the easiest of decisions. @niko, when I think about the staged approach, it feels like it is maybe more disruptive than doing it all at once. stage 1: force everyone to install k8s and celery libs -- that's one disruption / risk stage 2: remove pre-install -- another disruption it

[LAZY CONSENSUS] Accept revisions to AIP-52 Setup and Teardown tasks

Calling for a vote by lazy consensus to accept the changes to AIP-52 Setup and Teardown Tasks. *Discussion thread:* https://lists.apache.org/thread/c4s0541nrjbjm1or8tpl08y4qtmjj4gd *Docs:* https://github.com/apache/airflow/blob/e3d82c6be0e0e1468ade053c37690aa1e0e4882d/docs/apache-airflow/howto/set

Re: [LAZY CONSENSUS] Accept revisions to AIP-52 Setup and Teardown tasks

The vote has passed. On Thu, Aug 3, 2023 at 6:30 PM Daniel Standish wrote: > Calling for a vote by lazy consensus to accept the changes to AIP-52 Setup > and Teardown Tasks. > > *Discussion thread:* > https://lists.apache.org/thread/c4s0541nrjbjm1or8tpl08y4qtmjj4gd >

[DISCUSS] move from semver to a more "rolling" release cycle for core

For whatever reason, I was reminded recently of snowflake's "behavior change" policy See here: https://docs.snowflake.com/en/release-notes/behavior-change-policy I think semver is problematic for core because basically you cannot implement behavior changes until the "mythical" major release. The

Re: [DISCUSS] move from semver to a more "rolling" release cycle for core

s, without it having to be anything more than that. CalVer <https://calver.org/> may be a good option. On Sat, Aug 26, 2023 at 11:22 PM Daniel Standish < daniel.stand...@astronomer.io> wrote: > For whatever reason, I was reminded recently of snowflake's "behavior > chang

Re: [DISCUSS] move from semver to a more "rolling" release cycle for core

to a MAJOR release - because our users will not be > able to upgrade and apply security fixes without - sometimes - majorly > breaking their installation. This is 100% against the spirit and idea of > the regulations. The regulations aim to force those who produce software to > make i

Re: [DISCUSS] move from semver to a more "rolling" release cycle for core

t; > > > To be honest, Airflow already faces challenges in improving the > > adoption > > > > of new features, in my personal opinion. For example, it took about a > > > year > > > > for deferrable operators to gain awareness and interest. I al

Re: [DISCUSS] move from semver to a more "rolling" release cycle for core

tent there is one, is potentially a better airflow. Personally, putting on my user hat, that feels like a worthy trade. On Wed, Aug 30, 2023 at 12:01 PM Daniel Standish < daniel.stand...@astronomer.io> wrote: > Yeah I agree completely with more liberal use of something like more >

Re: [DISCUSS] move from semver to a more "rolling" release cycle for core

gt; * Also I agree 'backwards compatible" does not mean "has this feature > enabled in new release". Take the infamous "SubDAG" as an example: If we > find a way to decouple it, I would be all for having a flag that ENABLES it > if set - but disabled by default.

Re: [DISCUSS] move from semver to a more "rolling" release cycle for core

> > would definitely be in favour of that approach and using it more > liberally. I think SemVer does not say anything about this case - "the > software still supports it but you need to flip a flg" sounds like a nice > way of introducing behavioural changes without breaking changes. This doesn'

Re: [DISCUSS] Executors docs should be published in Airflow core or providers?

Sounds reasonable.

Re: [DISCUSS] Mechanism of SLA

Some questions for you Sung. I tried looking to understand why we needed to remove behavior 3 discussed in AIP: *[remove]* Task-level SLA measured from DAG-run scheduled start time I'm just a little concerned that removing this would be a mistake because, in my mind, part of the essence of what

Re: [DISCUSS] Mechanism of SLA

reliability metrics that help us detect when the DAG runs > themselves are being scheduled late, or if we are reaching the limit of > worker parallelism. These are issues that will impact the delivery times of > all of the DAGs and the Tasks, if not resolved immediately. So to me, as >

Re: [DISCUSS] Mechanism of SLA

First of all, thanks for being so charitable in engaging in this dialogue, I appreciate it. Yeah I think that the notion that maybe Airflow is making really impractical promises with SLA, well that could be true. One question for you, as I continue to let this percolate. Can you help me understa

Re: [DISCUSS] Mechanism of SLA

OK so one difference here is, you're adding a new DAG SLA concept. Which is useful. One subtle difference from what I think is the existing "concept" of SLA is that you are evaluating it against when it started, as opposed to when it should have started, and evaluating it only in the course of ru

Re: [DISCUSS] Allow multiple pools for one task and require all corresponding pool slots to be available before a task can run

I share Jens's concerns about complexity.

Re: [DISCUSS] Mechanism of SLA

I was able to chat with a couple folks about this. Small sample, but the sentiment was, "this is just a timeout". In other words, if we're going to call this SLA, we really ought to evaluate against the "this thing should have run by" time and not the actual start time. And, ideally, we should al

Re: [DISCUSS] What we should do with DbApiHook.get_uri?

Yeah I think your proposal seems reasonable. Airflow conn URI is not same thing as sqlalchemy URI. That worked for some simple circumstances but it is definitely not true in general. Hooks that need it should generally implement it. On Tue, Sep 19, 2023 at 10:51 AM Andrey Anshin wrote: > Hist

Re: [DISCUSS] Mechanism of SLA

I don't think of it as really a question about accurate record keeping but more a question of what an SLA is, i.e. when do you want the warning, or what do you want the warning based on. I think that the idea has been that it really means, "if task not done by X time each day then warn". And the

Re: [VOTE] Release Apache Airflow Helm Chart 1.11.0 based on 1.11.0rc2

+1 binding Verified licence signature checksum, installed and ran a dag On Mon, Oct 2, 2023 at 8:00 AM Hussein Awala wrote: > Thank you for the clarification! > > > However, I just had an idea which we can try out for future releases 👍 > > I wonder if we can exclude the file from rat check and

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

I would also like to see it deprecated. That said, I am not convinced there is anything that we cannot encode using URI though. I think the problem is just when one tries to use the same URI to mean two different things, e.g. both airflow connection URI and sqlalchemy URI. They are different. H

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

The thing that makes *me* hesitant to deprecate is the sheer magnitude of breaking it would bring (even though we're only talking about a hypothetical 3.0 release), balanced against the actual pain it causes. I.e. it's confusing to use, and takes up space in docs (when, if removed, we could just sh

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

> > I still think different people have different expectations of the Airflow > URI format. My expectation is that this is only a SerDe mechanism around > Airflow Connection. If we have a look at the codebase of Airflow we found > that this is not always true and we try to use it as SA URI. Yes,

Re: [DISCUSS] Future of Pendulum in Airflow

Thanks Jarek On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk wrote: > I think we miss important insight - straight from the source. > > I believe it's time to be candid and simply ask questions for the future of > Pendulum directly where we should - ie. we should just ask maintainers. > > I've just

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

unds that the Connection class' get_uri is meant to be general, > > supporting the use case of serializing the Airflow connection on basis of > > host, port, schema, password, type etc. > > > > Thanks & Regards, > > Amogh Desai > > > > On Mon, Nov

Re: [DISCUSS] Using serde for PyODBC case/other cases [was Serialization in Apache Airflow]

> > Tried to get others some opportunity to comment, but I see it's mostly me > <> Bolke. It might help to start off this separate discussion thread with a simple / concise problem statement. (that might sound snarky but it ain't :) ) And I say *might help* cus you still might get crickets even

  1   2   3   4   >