~ Jan 2013 it experience long delays with a good consistency when starting at 07:25 and 12:25 UTC (they are still there today).
That is good information. And the timing seems like it might relate. Ward, I wonder about using --bwlimit as well as nice on the backup job. That's the first guess that comes to mind. After all, if the backup takes four hours instead of two, it hardly matters, right? Whereas people definitely notice when their VC updates time out. It's just a guess, but ... do you have the --stats on what bandwidth the rsync is using now, so could try (say) halving it? Other than that ... for starters, I've added a cron entry to vcs:/etc/cron.d/sv to run top every 20 minutes, logging to /var/tmp/toplog for now. (I forgot -b originally so savannah-reports-private saw a bunch of failures, sorry about that. Hopefully working now.) I know Bob et al. were gathering stats on frontend, but I didn't see anything on vcs. karl