On 11.04.2014 02:24, Daniel Shahaf wrote: > stef...@apache.org wrote on Thu, Apr 10, 2014 at 18:08:46 -0000: >> +++ subversion/branches/thunder/notes/thundering-herd.txt Thu Apr 10 >> 18:08:45 2014 >> @@ -0,0 +1,50 @@ >> +The Problem >> +----------- >> + >> +In certain situations, such as the announcement of a milestone being >> +reched, the server gets hit by a large number of client requests for >> +the same data. Since updates or checkouts may take minutes, more of >> +the same requests come in before the first ones complete. The server >> +than gets progressively slowed down. > Does this have to be solved in svn? > > The solution sounds like you are reinventing the OS' disk cache. > Wouldn't it be better to make a syscall to hint the cache that "<these> > files are going to be a hot path for the next few minutes", so the OS > can then consider the size of the files v. the size of the CPU caches > and other processes' needs and make its own decisions? > > I'm not against having this branch, I just don't immediately see why > this is the best solution to the described problem.
I'd normally agree ... I'm pretty sure that in the described scenario, the disk cache is already as hot as it'll ever get. The fact is that that the FS layer caches data that are derived from disk contents in a rather expensive way, and no amount of disk caching can amortize that expense. However: I would argue that, long term, effort is better invested into changing the way we store data on disk to lower or, dare I say, eliminate the expense of deriving it. Currently the cached data falls into two distinct groups: * metadata (directories and nodes) * content (fulltext (and deltas, I think?) Retrieving the fulltext of a file is in general going to be expensive, because of deltification; but the thundering-herd effect isn't all that important for file contents, compared to metadata, since the latter gets the lions' share of accesses. Currently, the cost of retrieving metadata is directly dependent on the cost of retrieving content, since we naïvely store directories the same way as file contents. I'd argue that effort is better spent on implementing the new metadata index design which makes much better use of disk (or rather, database page) caching. FWIW, I agree with Ivan that the proposed block/retry approach leaves me less than enthusiastic ... I don't think it's even provable that it cannot cause deadlocks, because the blocking order depends on access patterns driven by clients that we have no control over. OK, nothing will actually deadlock since the locks are designed to expire; but I can easily imagine scenarios where the blocking actually slows things down overall. Limiting the number of allowed parallel requests on the HTTPd and svnserve level is a much better approach, IMO, and can be tweaked by the admin depending on server capabilities. I don't think anyone's actually ever researched how many parallel requests the repository can actually handle in these situations. I suspect looking into that would be time better spent. -- Brane -- Branko Čibej | Director of Subversion WANdisco // Non-Stop Data e. br...@wandisco.com
