On Tue, 13 Aug 2019, Hector Oron wrote: > Hello, > > Missatge de Bastian Blank <wa...@debian.org> del dia dt., 13 d’ag. > 2019 a les 11:51: > > > > Hi folks > > > > salsa.debian.org is partially down for now. Especially everything that > > concerns Salsa CI. > > > > Someone decided to inject a few thousand CI jobs using Salsa CI > > yesterday evening. Since then the system is not longer at 50% load, but > > at a 100%. It turns out that the configured amount of concurrency in CI > > builds can't be handled by the current available system resources. > > > > This now affects all user access to salsa.debian.org. So we disabled > > access to all the Salsa CI stuff for now, which will make everything > > using it fail. > > > > We have to discuss first how we can go forward. > > Those are very bad news. I hope that can be recovered soon. If you > need more resources to be able to handle more jobs or longer timeouts, > please let DPL know so more resources can be added. It is already recovered. We will investigate where we can extend the ressources. But some misusages (like requesting >1300 merge requests via API on a big project, that in consequence run >1300 ci jobs, that...) can't be solved regardless on how many resources we add.
Alex
