I think passing the full TI to executors following a backwards compatible path
is perfectly fine and shouldn't get much push-back.
What I think we should really discuss is whether (and if yes, how) we want to
introduce task resourcing to executors. Should the Edge Executor just come up
with it'
+1 (binding) - Checked SVN, Check in Docker, Reproducible package build,
Licenses, Signatures
On 14.11.24 20:30, Elad Kalif wrote:
Hey all,
I have just cut the new wave Airflow Providers packages. This email is
calling a vote on the release,
which will last for 72 hours - which means that it wi
+1 non binding. All AWS system tests are running successfully using
apache-airflow-providers-amazon==9.1.0rc4. Only
example_comprehend_document_classifier is failing but is already failing in
main (AWS team is working on a fix). You can see the results here:
https://aws-mwaa.github.io/#/open-so
Hi,
not really surprising as I was talking with Marco about this at work to
further work on the EdgeWorker I am also in favor in option 2. It would
be a small breaking change in the API but would be well suited in
Airflow 3. But looking at the PR we could also keep the existing
signature and allo
> I think that if we have implemented async for an operator, we should
remove
the non-async version.
I think we all agree here. The question is only "when". I see two options:
1) When Airflow 3 is released
2) When Airflow 2 stops being supported
I'd be for 2) .
Why? Because it has the least imp
Hey all,
I have just cut the new wave Airflow Providers packages. This email is
calling a vote on the release,
which will last for 72 hours - which means that it will end on November 17,
2024 19:30 PM UTC and until 3 binding +1 votes have been received.
Consider this my (binding) +1.
Airflow Pr
To me, I don't really mind reschedule mode so much. But I *really *don't
like that we have *both* styles implemented in operators (talking about
providers here).
I think that if we have implemented async for an operator, we should remove
the non-async version.
And we should remove the `deferrabl
Hi all,
I´m currently working on an PR to enable the EdgeWorker with slot/resource
handling. PR: https://github.com/apache/airflow/pull/43737.
In the PR we decided to make a devlist discussion about how to get additional
task_instance data into the executor. This can be managed in different way
> Seems that option 2 gives the most while
still providing fallback for users - so why go straight from 1 to 3 if
1 looks impossible?
I think the problem with this is that while functionally it will work (i.e.
yes, there is fallback), it is not feasible and sustainable for many big
users. If you h
Not sure why most people choose between 1 and 3, when the
reschedule mode seems to be the biggest issue. Removing poke
does not remove a lot of complexity - while removing reschedule
simplifies the scheduler. Seems that option 2 gives the most while
still providing fallback for users - so why go st
Option 1 is a non-starter from the compatibility standpoint. Not every
reschedulable sensor can be deferrable as the retriable check operation
itself can have arbitrary execution time.
Option 2 is a way to go if we want to formally sunset sensors as the
dedicated mechanism. Option 3 is almost keep
11 matches
Mail list logo
