In my understanding, the slower performance of PostgreSQL is a known behavior for write intensive applications. PGBouncer used for connection pooling cannot change/improve that. And Airflow with multiple DAGs and/or dynamic tasks with heavy workload will be write intensive. I have done extensive tests and have collected database statistics that show the bottleneck. I can share the details if you would like to see. I am wondering if there are any load tests or comparisons done so far by anyone in the community. Unless I am doing something totally wrong, PostgreSQL actually does not seem to be a right choice at all; slower performance and higher cost (due to PGBouncer process) compared to MySQL
Jigar > On Oct 14, 2024, at 2:41 AM, Jarek Potiuk <ja...@potiuk.com> wrote: > > MySQL is not going away. You can use it if you want. We have no plans to > remove it. > > The advice did not change. Postgres is generally more stable that's why we > recommend it. MySQL has much worse locking behaviour that is somewhat > unpredictable and - especially when you use backfills - it is known to > generate occasional deadlocks. This is likely why you got the advice (quite > likely by me). > > But in general it's fully supported and works and we have no plans to get > rid of it. > > Re: cost/price - it's up to you to choose and pay for services you use - we > do not compare pricing of all possible available services. > > Also if you follow the best practices for Postgres (use pgbouncer: > https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#setting-up-a-postgresql-database), > it's unlikely you will have slower postgres with similar machine/setup. > > J. > > >> On Mon, Oct 14, 2024 at 7:38 AM Jigar Parekh <ji...@vizeit.com> wrote: >> >> Back in June’24, I had a discussion on Slack about a database issue. My >> database backend for the Airflow instance is MySQL. It was recommended to >> migrate to PostgreSQL to resolve such issues. I was also told that MySQL >> may not be supported in the future versions. I configured PostgreSQL and >> performed few tests to compare both the DBs with the type of heavy workload >> I was expecting for my airflow instance in the production environment. The >> test results did not show a reason to switch from MySQL to PostgreSQL. In >> fact, PostgreSQL performed slower and airflow configuration cost more >> compared to MySQL. I wanted to start this discussion to find out if others >> have any similar observations about Postgres and what is Airflow community >> planning to do about MySQL support in the future versions? >> >> Jigar >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
