RE: [DISCUSSION] Plans for Database backend

2024-10-14 Thread Damian Shaw
Our MySQL DB is on bare metal, and we don't see any memory or cpu bottlenecks, so maybe it would help, but it's not something we have evidence for or can cheaply test. Damian -Original Message- From: Jigar Parekh Sent: Monday, October 14, 2024 3:19 PM To: dev@airflow.apache.org Subject

Re: [DISCUSSION] Plans for Database backend

2024-10-14 Thread Jigar Parekh
I have observed that allocating right amount of memory-cpu helps to avoid issues with MySQL Jigar > On Oct 14, 2024, at 11:51 AM, Damian Shaw > wrote: > > I can't speak to general performance between MySQL and PostgreSQL, but I can > tell you a real-world specific issue I we have faced bein

[ANNOUNCE] The "tests_comon" moved from "dev" to be top level directory

2024-10-14 Thread Jarek Potiuk
Hello everyone, I have just merged my PR https://github.com/apache/airflow/pull/42985 to move "tests_common". This was a follow up after Ash's splitting providers PR - solving a teething problem where the test_commons could not be modified by a non-contributor in their PR. Please rebase your PRs

Re: [DISCUSSION] Plans for Database backend

2024-10-14 Thread Jarek Potiuk
Hello Jigar, I think if someone wants to do a fair comparison of performance and cost, a detailed analysis of multiple cases, users, usage patterns and statistics if needed - rather than anecdotal evidence and single installation done not following the best practices (i.e. pgbouncer). The stateme

Re: [DISCUSSION] Plans for Database backend

2024-10-14 Thread Jarek Potiuk
MySQL is not going away. You can use it if you want. We have no plans to remove it. The advice did not change. Postgres is generally more stable that's why we recommend it. MySQL has much worse locking behaviour that is somewhat unpredictable and - especially when you use backfills - it is known t

Re: [DISCUSSION] Proposal to Add Couchbase Provider to Apache Airflow

2024-10-14 Thread Jarek Potiuk
On Mon, Oct 14, 2024 at 7:12 AM Shyam Rajamannar wrote: > Hi Amogh, Jarek > > Sure, we will go with the custom provider approach. Thanks for the > feedback. > > Also, I just need to create a PR to https://github.com/apache/airflow-site > to update the ecosystem docs, right? > Yes. Just click "Su

[ANNOUNCE] Apache Airflow Providers prepared on October 10, 2024 are released

2024-10-14 Thread Elad Kalif
Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages prepared on October 10, 2024 were just released. Full list of PyPI packages released is added at the end of the message. The source release, as well as the binary releases, are available here: https://a

[RESULT][VOTE] Airflow Providers - release of October 10, 2024

2024-10-14 Thread Elad Kalif
Hello, Apache Airflow Providers prepared on October 10, 2024 have been accepted. 5 "+1" binding votes received: - Elad Kalif - Jens Scheffler - Jarek Potiuk - Kaxil Naik - Hussein Awala 5 "+1" non binding votes received: - Freddy Demiane - Wei Lee - Amogh Desai - Pavankumar Gopidesu - Vishnu Ch

Re: [DISCUSSION] Plans for Database backend

2024-10-14 Thread Jigar Parekh
In my understanding, the slower performance of PostgreSQL is a known behavior for write intensive applications. PGBouncer used for connection pooling cannot change/improve that. And Airflow with multiple DAGs and/or dynamic tasks with heavy workload will be write intensive. I have done extensive

RE: [DISCUSSION] Plans for Database backend

2024-10-14 Thread Damian Shaw
I can't speak to general performance between MySQL and PostgreSQL, but I can tell you a real-world specific issue I we have faced being on MySQL for Airflow. MySQL locking and Airflow's rendering of task fields do not play nice, and we see many errors (the relevant GitHub issue: https://github.