Hi again, everybody, based on the feedback here on the mailing list and on GitHub, and lots of digging into the docs, I have now changed snakemake's behaviour to do much fewer status queries and to do them in database-optimised batches via sacct. As there is a lot of user and admin knowledge around here, I wondered if any of you might find the time to comment on the suggested changes. These are in the open pull request here, with the most important infos in the initial comment starting the pull request:
https://github.com/snakemake/snakemake/pull/2167 We're aiming to merge and release this in the next couple of days, to get this out quickly and enable users to reduce strain on slurm clusters as soon as possible (by updating to the then latest version of snakemake). So if it's already merged by the time you manage to have a look, nevertheless don't hesitate to comment on the pull request, comment here or open a new issue on the snakemake repository (optimally pinging me in) -- we'll gladly further improve the situation, here. Also, in case the solutions provided for snakemake job status queries find approval here, it might make sense to point developers of other workflow managers to this pull request, whenever you find such workflow management systems overstraining your slurm setup. Otherwise, many thanks already to Tim Wickberg, Sean Maxwell, Bas van der Vlies and Brian Andrus for the particularly helpful feedback and suggestions that led to the current solution! cheers, david
