itHub issue:
>> https://github.com/apache/airflow/issues/32342). It has led us to avoid
>> using Dynamic Task Mapping, as well as nesting of Task Groups, while we are
>> still on MySQL, which is a shame because these features would be a natural
>> fit to solve some of our proble
MySQL, which is a shame because these features would be a natural
> fit to solve some of our problems.
> >
> > Damian
> >
> >
> >
> > -Original Message-
> > From: Jigar Parekh
> > Sent: Monday, October 14, 2024 2:35 PM
> > To: dev@air
apache.org
Subject: Re: [DISCUSSION] Plans for Database backend
I have observed that allocating right amount of memory-cpu helps to avoid
issues with MySQL
Jigar
> On Oct 14, 2024, at 11:51 AM, Damian Shaw
> wrote:
>
> I can't speak to general performance between MySQL and PostgreS
14, 2024 2:35 PM
> To: dev@airflow.apache.org
> Subject: Re: [DISCUSSION] Plans for Database backend
>
> In my understanding, the slower performance of PostgreSQL is a known behavior
> for write intensive applications. PGBouncer used for connection pooling
> cannot change/impro
rom: Jigar Parekh
Sent: Monday, October 14, 2024 2:35 PM
To: dev@airflow.apache.org
Subject: Re: [DISCUSSION] Plans for Database backend
In my understanding, the slower performance of PostgreSQL is a known behavior
for write intensive applications. PGBouncer used for connection pooling cannot
chan
In my understanding, the slower performance of PostgreSQL is a known behavior
for write intensive applications. PGBouncer used for connection pooling cannot
change/improve that. And Airflow with multiple DAGs and/or dynamic tasks with
heavy workload will be write intensive. I have done extensive
MySQL is not going away. You can use it if you want. We have no plans to
remove it.
The advice did not change. Postgres is generally more stable that's why we
recommend it. MySQL has much worse locking behaviour that is somewhat
unpredictable and - especially when you use backfills - it is known t
Back in June’24, I had a discussion on Slack about a database issue. My
database backend for the Airflow instance is MySQL. It was recommended to
migrate to PostgreSQL to resolve such issues. I was also told that MySQL may
not be supported in the future versions. I configured PostgreSQL and perf
