ephraimbuddy commented on issue #18041: URL: https://github.com/apache/airflow/issues/18041#issuecomment-1015159913
> @ephraimbuddy it does reduce the number of errors by a lot! Thank you! There are some errors where the jobs are stuck in `queued` state, but I think it should be solvable by finding the right interval configuration for our use cases. Just out of curiosity, how did you know that the problem was with the interval from the logs? And btw, even though it is under `AIRFLOW__KUBERNETES`, this configuration somehow works with Celery Executor with KEDA too. If you are not on kubernetes then I'm not sure how it worked for you(_I think it shouldn't work_). What formed my opinion was that your pods were taking time to start and queued tasks were being moved to scheduled. So I don't know how it worked for your case. Maybe @jedcunningham can explain better. Maybe you should increase `AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE` also in kubernetes 🙃 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
