Just today I received an update from sysadmin that they have reduced the consumption of the backup job (again, this time in a different way), which I had requested back at the time of the previous discussions. We'll see if it helps.
I'm sure there is more to be done, in any case, as the load should not get that high on the host in the first place. At this point I believe it is load on the vcs backend host (that holds the version control repositories) that ripples through and causes requests to the frontend vhost to time out. My best guess at the moment is that there is simply too much disk contention, and there is no quick fix. The top processes at the time of high load tend to be Apache and web browsing VC CGI scripts. We've already systematically made those processes run with a low priority (so the machine doesn't get bogged down even worse) which has helped some but, as you've seen, not solved the problem. I don't have an answer. So, I don't have a good answer, but the above is the status. karl
