> > Loading the DagBag takes around 40 seconds because of the number of tasks
this is suspicioius. it's not a given that a dag will take 40 seconds to parse due to 1000 or 2000 tasks. do you perhaps have network calls in your dag that are slowing things down? i would try to identify exactly what in the dag parse is slow and see if you can remove the slowness. 60 workers set to process 1 task at a time can you increase worker concurrency? i.e. num tasks per worker. default is 16 i think, and 1 is of course the lowest number possible. i presume you have heavy tasks that need the full resources of the worker?
