GitHub user dschneider-wxs created a discussion: psycopg2.OperationalError: SSL error: decryption failed or bad record mac
Hoping someone can point us in the right direction. We are seeing this error sporadically. for example last night it happened twice in an hour (about 830 and 930) and then not again until 5am. The scheduler is crashing and restarting. Logs for the 30s of errors attached. We are running on ECS Fargate. We have a service for the scheduler and a service for the triggerer, webserver, dag_processor, db-init. We use RDS (postgres) db. We have RDS proxy service running (my devops colleague says this is like pgbouncer). We also setup keep alive in the sqlalchemy config per https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#setting-up-a-postgresql-database Additionally we set the proxy pool timeout to 59 min with the airflow pool limit at the default 30 min. Initially they were the same and we suspected a race condition but it has not solved the issue. Otherwise preping is enabled and I've seen both the keepalive and preping in the sqlalchemy logs. The DB is a small instance and Proxy service are both static. resource usage is not maxed out. We are using LocalExecutor, originally with 2 schedulers running in separate Tasks, but last night just 1 scheduler running. cpu/memory usage is well below 100% usually although last night at 430 the cpu starting increasing somewhat suddenly. this makes me think that the scheduler was slow about accessing the db or something and evetually tripped on a closed connection(?) we have 1 hourly dag, and 1 triggered on a asset from this dag. A 3rd dag that is asset triggered using sqs (which has about 20k msg in it to keep things running). We have made progress. we used to get this error multiple times an hour and now its more sparse, but what edge case could we be running into? [airflow-log.rtf](https://github.com/user-attachments/files/25801635/airflow-log.rtf) GitHub link: https://github.com/apache/airflow/discussions/63014 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
