On 2024-10-27 17:10, Alfred M. Szmidt wrote:
the question of "how hard" is hard to answer, ironically.Yes; our Git server already can't cope with its current load, https://savannah.gnu.org/support/?110712 If some web pages migrate to Git, the load will increase. Then we should get new hardware that is capable, no matter if web pages are using git or not. CCing sysadmin@ and rms@ who can maybe do something about it.
AFAIK (from lurking around various communication channels) FSF sysadmins purchased 3 extra servers after a certain incident in July when Ian pulled an all-nighter at the datacenter. It would appears that for now, capacity issue can be solved without purchasing new hardware. On the other hand, there was an incident in early October where the breaker popped (tripped overcurrent protection) at the datacenter, so adding new hardware will increase the load of electricity that power supplies at the datacenter don't have the capacity to deliver now. And there is the last issue: freedom-respecting-blob-free hardware is very scarce now. FSF is using the last KGPE-D16 motherboards, designed and manufactured more than a decade ago, so breakages/damages/wear&tear happen naturally. The RYF certified Talos II was such a good candidate, but the price of the board has skyrocketed, and then the board was released in 2017, its support may end anytime soon.
Which brings us to the next question, how we can solve the load issue. I was informed that most git traffic comes from scrap bots and CI builds, pipelines (gitlab runner, github actions, jenkins pipelines, etc.). Legitimate human user traffic is not much, and Savannah users can always use ssh, which is authenticated and not pounded as hard as http(s). Even though rate-limiting and fail2ban are in place, the hostile internet is getting more and more aggressive, and existing measures might not be sufficient. A while ago, I suggested that we create new VMs just for readonly git/cvs/svn/hg access, load balanced on them. I believe my plan will solve the issue, and I have experience setting up nginx as the load balancer at home, since I designed and set up my homelab myself.
However, FSF infrastructure is not my homelab :D. Of course sysadmins need to do many things that only they have the access to do. This naturally will delay the plan. To give you an idea how long the delay can be: I donated a VM (in my infra) as ns3.gnu.org to FSF in July. I completed the trisquel install and initial setup in half a day. 3 months later, I'm still waiting for sysadmins to set up bind9 so it can actually start to serve dns queries. Oh well. This example was not high priority of course, but you get the idea. The two sysadmins are way too overloaded, especially with the unusually high amount of datacenter trip. Recently, they have office relocation while fighting off constant and continuous DDos attack on www.gnu.org. Clearly, we can draw the conclusion that the sysadmin team is understaffed, to the extent that FSF infrastructure is going to have chronic illness (I don't like the term "technical debt"). Earlier this year I wrote to both RMS and Zoë separately to address this, but it's clear that it was/is a problem very hard to be solved. FSF needs more donations and members. With the ongoing misinformation, FUD and smearing campaigns against FSF and GNU, I'm not optimistic.
So now it seems that the only solution for now is waiting for things to change. Maybe by this time the next year, our git service would be faster than ever. Or, if FSF wants to hire a few college students working part time at $7.25/hr (I would do it :D).
Lukewarm regards, -- Jing Luo About me: https://jing.rocks/about/ GPG Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC
signature.asc
Description: OpenPGP digital signature
