On Thu, Jul 21, 2011 at 08:09, Philip Martin <philip.mar...@wandisco.com> wrote: > Greg Stein <gst...@gmail.com> writes: > >>> The first thing I tried leaks memory: >>> http://subversion.tigris.org/issues/show_bug.cgi?id=3967 >> >> Are you suggesting that my fix for 3888 caused this problem, or is >> incorrect in some way? Or is this a new, distinct, and undiscovered >> problem that you're bringing up? > > Issue 3888 was something to do with file transfers? I see 3967 with > empty revisions so it's probably different.
The root problem with 3888 is that N server requests would be queued up, where N could grow into the tens of thousands. There are various bits to try and minimize the memory space of each of those requests, but it still ended up needing a huge/unbounded amount of memory. The fix was to halt processing of the server response (that indicated *which* requests to make) when the outstanding request count gets too high. Once the count drops, then response processing resumes and more requests are queued. If the svnrdump process generates a hojillion independent requests, then it may be seeing a similar problem. Applying throttling similar to the update process should fix it. If the process is single-request based (eg. 'svn log' and a million log entries in a stream), then we'd have to look elsewhere. That would be an iterpool-like symptom such as we just found in the log processing. Cheers, -g
