Hello Philippe, Philippe Vaucher wrote: > I'm the author of https://hub.docker.com/r/silex/emacs, dockerized Emacs > images for a lot of versions. > > You can see one of the builds here: > https://gitlab.com/Silex777/docker-emacs/pipelines/148568952
Wow! I see 40 different builds there. Which is totally awesome btw. > The build thus builds a lot of images at once, and thus I have many > parallel "git clone https://git.sv.gnu.org/git/emacs.git";. That can certainly be a problem. Because Emacs is not small. And in conjunction with the rest of the Internet also cloning Emacs and cloning other projects from the server the effect can brown out the system. > For a while this worked fine, but since 4-5 days I get a lot of http > 502/504 and I found out that I can only do about 4 parralel git clones. > > Is there a limit and/or maintainance going? Was I but in some sort of > throttle-list? We do have rate limits in place. Because otherwise the general background radiation activity of the Internet will break things. However nothing has changed recently in regard to the rate limits for a long time. As I look at the logs the last rate limit change was Sat Dec 7 06:31:50 2019 -0500 which seems long enough ago that it isn't anything recent. Meaning that this is probably simply just you competing with other users on the Internet for resources. Cloning with the https transport that uses git-http-backend for the backend. We are using Nginx rate limiting. Which you can read about how the algorithm works here. It is basically a smoothing process. https://www.nginx.com/blog/rate-limiting-nginx/ How are you doing the clone? Is it a fresh clone into an empty directory? Because that would be the worst case. In that worst case scenario it would need to transport 100% of the repository every time. That's pretty heavy. The Emacs git repository is about 359MB in size. Seeding that with the previous version and updating it would make that transfer much more efficient. Whenever I have set up continuous integration builds I always set up a local mirror of all remote repositories. I poll with a cronjob to refresh those repositories. Since the update is incremental it is pretty efficient for keeping the local mirrors up to date. Then all of the continuous integration builds pull from the local mirror. This has a pretty good result in that the LAN is very robust and only shows infrastructure failures when there is something really catastrophic happening on the local network. Since 100% of everything is local in that case. Transient WAN glitches such as routing problems or server brownouts happen periodically and are unavoidable but then will catch up the local mirror image with the next cronjob refresh. Can refresh on a pretty quick cycle. I would do every four hours for example. Plus those errors are much more understandable as a network issue separate from the CI build errors. It's a good way to protect the CI builds and redunce noise from them to a minimum. Periodically the Savannah vcs server is pummeled by network abuse. People. It's why we can't have nice things. I have been seeing some zero-dark-thirty times when things have been hammered into failure. Which then will recover by the time I wake up and go peek at them. Do the failures are are seeing have a periodic time of day cycle when they are more likely to happen in the middle of the US nighttime? If so then that is probably related. I have been needing to look at the logs and improve the fail2ban rules to try to push back on the abuse a little more strongly. I do this periodically but the type of abuse keeps changing. For example right now I am seeing a new wp-content path (Wordpress attack). However note that there is no Wordpress running on these machines. Anti-abuse rule maintenance is always needed. Bob
