kaxil commented on PR #42082: URL: https://github.com/apache/airflow/pull/42082#issuecomment-2400544374
> Can provide more details tomorrow, but I was able to reproduce an issue seen at Astronomer due to this aggressive locking. > > With a DAG with 1000 FileSensors set to `reschedule` mode, and 10 tries. Launching 5 runs of this DAG tanks the scheduler's heartbeat, as soon as the sensor tasks begin rescheduling. SQL statements back up, all waiting for tuple locks on the same `dag_run` record. The scheduler then also gets stuck waiting behind these lock statements, and it starts missing heartbeats. > > Planning to patch this PR in tomorrow morning to test its effects. @seanmuth Any update on the results of this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
