Our MySQL DB is on bare metal, and we don't see any memory or cpu bottlenecks,
so maybe it would help, but it's not something we have evidence for or can
cheaply test.
Damian
-Original Message-
From: Jigar Parekh
Sent: Monday, October 14, 2024 3:19 PM
To: dev@airflow.apache.org
Subject
I have observed that allocating right amount of memory-cpu helps to avoid
issues with MySQL
Jigar
> On Oct 14, 2024, at 11:51 AM, Damian Shaw
> wrote:
>
> I can't speak to general performance between MySQL and PostgreSQL, but I can
> tell you a real-world specific issue I we have faced bein
Hello everyone,
I have just merged my PR https://github.com/apache/airflow/pull/42985 to
move "tests_common". This was a follow up after Ash's splitting providers
PR - solving a teething problem where the test_commons could not be
modified by a non-contributor in their PR.
Please rebase your PRs
Hello Jigar,
I think if someone wants to do a fair comparison of performance and cost, a
detailed analysis of multiple cases, users, usage patterns and statistics
if needed - rather than anecdotal evidence and single installation done not
following the best practices (i.e. pgbouncer).
The stateme
MySQL is not going away. You can use it if you want. We have no plans to
remove it.
The advice did not change. Postgres is generally more stable that's why we
recommend it. MySQL has much worse locking behaviour that is somewhat
unpredictable and - especially when you use backfills - it is known t
On Mon, Oct 14, 2024 at 7:12 AM Shyam Rajamannar
wrote:
> Hi Amogh, Jarek
>
> Sure, we will go with the custom provider approach. Thanks for the
> feedback.
>
> Also, I just need to create a PR to https://github.com/apache/airflow-site
> to update the ecosystem docs, right?
>
Yes. Just click "Su
Dear Airflow community,
I'm happy to announce that new versions of Airflow Providers packages
prepared on October 10, 2024 were just released. Full list of PyPI packages
released is added at the end of the message.
The source release, as well as the binary releases, are available here:
https://a
Hello,
Apache Airflow Providers prepared on October 10, 2024 have been accepted.
5 "+1" binding votes received:
- Elad Kalif
- Jens Scheffler
- Jarek Potiuk
- Kaxil Naik
- Hussein Awala
5 "+1" non binding votes received:
- Freddy Demiane
- Wei Lee
- Amogh Desai
- Pavankumar Gopidesu
- Vishnu Ch
In my understanding, the slower performance of PostgreSQL is a known behavior
for write intensive applications. PGBouncer used for connection pooling cannot
change/improve that. And Airflow with multiple DAGs and/or dynamic tasks with
heavy workload will be write intensive. I have done extensive
I can't speak to general performance between MySQL and PostgreSQL, but I can
tell you a real-world specific issue I we have faced being on MySQL for Airflow.
MySQL locking and Airflow's rendering of task fields do not play nice, and we
see many errors (the relevant GitHub issue:
https://github.
10 matches
Mail list logo
