Re: [DISCUSS] AIP-78 scheduler-managed backfill

In my mind, there are two separate discussions here. 1. Locking and deadlocks This I believe will get better as part of the set of changes in this AIP as well as AIP-72. Of course, pending the "mini-scheduler" discussion. I am personally in two minds about that, having been both a proponent and a

More ruff style rules

Hey folks, There have been a number of projects over the years to enable more and more style rules in our linters. They serve as a great first project and help to standardize code which makes it easier for Future Us to jump into sections of the code we otherwise don't usually work in and for n

Re: [DISCUSS] AIP-78 scheduler-managed backfill

> > It's somehow related (but yes -it's more AIP-72 question). Mini scheduler > currently **actually** attempts to lock the DagRun table when it runs - > this is precisely what has been recently made as "optionally skipped" when > mini-scheduler could not obtain the lock immediately - because it wr

Re: [VOTE] Airflow Providers prepared on July 12, 2024

+1 non binding. All AWS system tests are running successfully against apache-airflow-providers-amazon==8.26.0rc2. You can see the results here: https://aws-mwaa.github.io/#/open-source/system-tests/version/a89514ec38d368efa9733c8376953024c8da9f1a_8.26.0rc2.html On 2024/07/12 14:21:54 Elad Kalif

Re: [DISCUSS] To AIP-44 or not to AIP-44

-0.25 to me continuing - I don't see the value long term in putting more effort into this, and any lessons from this have already been learnt, as development on Airflow 3 has already started, so this would literally be happening in parallel. Given we are planning, at most, 1 or two more release

[VOTE] Release Airflow 2.9.3 from 2.9.3rc1

Hey fellow Airflowers, I have cut Airflow 2.9.3rc1. This email is calling for a vote on the release, which will last at least 72 hours, from Friday, July 12, 2024 at 5:10 pm UTC until Monday, July 15, 2024, at 5:10 pm UTC

Re: [DISCUSS] AIP-78 scheduler-managed backfill

> I think we're the unfortunate victims of overloaded terminology. The deadlock I refer to in the AIP is not a database deadlock. I was absolutely certain it was about db deadlocks - also because they are often mentioned in the context of running backfill - and having separate scheduler loop and

Re: [DISCUSS] AIP-78 scheduler-managed backfill

> > But regarding the scheduler pressure, I think I have a bit of a > different observation and the deadlock problem which is a bit > downplayed in the current proposal is - IMHO - crucial problem to be > solved when we want to make backfilling more accessible > I think we're the unfortunate victi

Re: [VOTE] Airflow Providers prepared on July 12, 2024

Valid links for testing: The test procedure for PMC members is described in https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members

[VOTE] Airflow Providers prepared on July 12, 2024

Hey all, I have just cut rc2 for Amazon and Weaviate Providers packages. This email is calling a vote on the release, which will last for 24 hours - which means that it will end on July 13, 2024 14:20 PM UTC and until 3 binding +1 votes have been received. Consider this my (binding) +1. Airflow

Re: [DISCUSS] To AIP-44 or not to AIP-44

I think we should finish the AIP within Airflow 2 - it will take time until Airflow 3 is out, and I believe some learnings from finishing and running this AIP might be useful for Airflow 3. We plan to contribute to finishing this AIP. On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T

[ANNOUNCE] Apache Airflow Providers prepared on July 09, 2024 are released

Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages prepared on July 09, 2024 were just released. Full list of PyPI packages released is added at the end of the message. The source release, as well as the binary releases, are available here: https://airf

[RESULT][VOTE] Airflow Providers - release of July 09, 2024

Hello, Apache Airflow Providers prepared on July 09, 2024 have been accepted. providers amazon and weaviate are excluded from this release. 3 "+1" binding votes received: - Elad Kalif - Jarek Potiuk - Hussein Awala 4 "+1" non-binding votes received: - Wei Lee - Vincent Beck - Amogh Desai - Utkar

Re: [VOTE] Airflow Providers prepared on July 09, 2024

amazon and weaviate providers are excluded from this wave On Thu, Jul 11, 2024 at 12:20 PM Utkarsh Sharma wrote: > +1 (non-binding) ran a few examples of dags and they worked as expected. > > Thanks, > Utkarsh Sharma > > > On Thu, Jul 11, 2024 at 1:09 PM Freddy Demiane > > wrote: > > > +1 (non-

Re: [DISCUSS] AIP-78 scheduler-managed backfill

This has been long awaited :). And we definitely need to do it. But regarding the scheduler pressure, I think I have a bit of a different observation and the deadlock problem which is a bit downplayed in the current proposal is - IMHO - crucial problem to be solved when we want to make backfilling

Re: [DISCUSS] AIP-78 scheduler-managed backfill

I think it makes sense to orchestrate backfills in a more managed way than through a CLI command, especially that execution of tasks would happen through regular executors configured in the Airflow deployment. Once concern I have, also called out in the AIP, is the increased load on the scheduler.

Re: [VOTE] AIP-72: Task Execution Interface aka Task SDK

+1 non-binding In general I think the AIP makes sense, and there are of course details to clarify and iron out - but this can happen on the way. On Fri, Jul 12, 2024 at 11:20 AM Ephraim Anierobi wrote: > +1 binding > > On Fri, 12 Jul 2024 at 06:41, Scheffler Jens (XC-AS/EAE-ADA-T) > wrote: > >

Re: [VOTE] AIP-72: Task Execution Interface aka Task SDK

+1 binding On Fri, 12 Jul 2024 at 06:41, Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 binding > > Mit freundlichen Grüßen / Best regards > > Jens Scheffler > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > GERMANY | www.