On Tue, Mar 15, 2011 at 6:22 PM, Paul Burba <ptbu...@gmail.com> wrote: > On Mon, Mar 14, 2011 at 2:36 PM, Hyrum K Wright <hy...@hyrumwright.org> wrote: >> On Sat, Mar 12, 2011 at 6:47 AM, Stefan Sperling <s...@elego.de> wrote: >>> On Fri, Mar 11, 2011 at 10:43:46PM -0500, Greg Stein wrote: >>>> 2011/3/11 Branko Čibej <br...@e-reka.si>: >>>> >... >>>> > For the second task, I think the first order of business is to change >>>> > the wc-db tree crawler to do one query instead of zillions, or at least, >>>> > where several queries are required, to do them all in one transaction. >>>> >>>> stsp has been working this recently. Killing the node walker, and >>>> moving to table scans. >>> >>> Yes. So far, I've been working in the revision status code. >>> There are two problems left to fix before I'll move on to the next task: >>> >>> - There are API layering issues (wc_db.c calls into node.c). >>> This is related to the API discussions in the other thread >>> so I'll follow up there. >> >> Before reading this thread, I saw the call into node.c, and have >> subsequently removed it. >> >>> - The revision status code issues about 5 separate queries, >>> which aren't combined via a transaction and don't use temporary tables. >>> This is no worse than the previous code using the node walker, >>> obviously :) But I'll look at fixing this so that the results >>> returned correspond to the state of the DB as of the time the >>> svn_wc__db_revision_status() call was made. >> >> I wrapped this API in a txn in r1081510. >> >>> For others who want to jump in and help, here is a list of places >>> where the node walker is still being used. I'm not sure if we can >>> eliminate it everywhere before release, but each of these should >>> be looked at to see whether we can use an alternative approach to >>> increase performance: >>> >>> subversion/libsvn_client/changelist.c >>> subversion/libsvn_client/commit_util.c >>> subversion/libsvn_client/info.c >>> subversion/libsvn_client/merge.c >>> subversion/libsvn_client/mergeinfo.c >>> subversion/libsvn_client/prop_commands.c >>> (This should be propget and propset. Proplist is already using >>> queries involving temporary tables. Rewriting propget on top >>> of the proplist code would be easy. > > Is anyone working on propget? I'll take stab at that if not.
Done, see r1082658. Not surprisingly, this makes a big difference in the performance of svn propget -R. What is possibly a little(?) surprising is that it's faster than 1.6 too: Using a checkout of https://svn.apache.org/repos/asf/subversion/branches@1082531 at depth immediates and setting these branches to infinite depth, 1.0.x 1.1.x 1.2.x 1.3.x 1.4.x 1.5.x 1.6.x, gives us reasonably(?) large WC: 272 MB 14,044 files 1,428 Directories svn pg svn:mergeinfo -Rv on the root Version Elapsed Time Peak Working Set Memory ------- ------------ ----------------------- 1.6.x@1082537 00:00:03.859 14,400 K trunk@1082591 00:01:15.979 12,004 K trunk@1082591 (PATCHED) 00:00:00.581 15,448 K Paul > It > would be useful for dealing with merge.c:merge_reintegrate_locked's > use of svn_wc__node_walk_children(). > > Paul > >>> Propset needs more work.) >>> subversion/libsvn_client/ra.c >>> subversion/libsvn_wc/update_editor.c >> >> I'll take a gander at some of these, too. (But I'm not entirely sure >> what I' gandering at or for...) >> >> -Hyrum >> >
