My takeaway is that using 'svn ls' without recursion is OK. I've just done a check on member_apps and I get:
$ time svn up member_apps/ Updating 'member_apps': At revision 93890. real 0m1.573s user 0m0.055s sys 0m0.027s $ time svn ls https://svn.apache.org/repos/private/documents/member_apps | wc -l 882 real 0m0.943s user 0m0.053s sys 0m0.026s i.e. svn update is ~ 50% slower than svn ls However, when using recursion, svn ls is quite a bit slower: $ time svn ls https://svn.apache.org/repos/private/documents/member_apps -R | wc -l 931 real 0m3.161s user 0m0.062s sys 0m0.031s On Tue, 26 Nov 2019 at 00:36, Greg Stein <gst...@gmail.com> wrote: > One quick answer from earlier in the thread: > > > > > > What's most concerning is not just the elapsed time, but that this > > > > > likely means that the call is expensive on the server, which may > > > > > impact other users. > > > > > > > > > > > > > > I see this as premature optimisation; we don't know whether svn list > is > > > > more expensive than svn update overall. > > > > There may be other reasons why list is slower. Nor do we know if the > > > > request will impact other users. > > "svn list" must generate the listing of 12k+ files, recursively. That takes > some time to process and deliver over the network. I believe it is likely a > PROPFIND which introduces some overheads on both ends (XML construction on > server, parsing on client; sheer network size, too). > > "svn up" generates a diff report. "get these 3 files", and that's easily > extracted from the difference between revision-working-copy and > revision-server (plus some other concerns). > > So yes: an update is *way* faster, all around. > > > On Mon, Nov 25, 2019 at 5:15 PM Sam Ruby <ru...@intertwingly.net> wrote: > > > Actually copy Greg this time. > > > > ---------- Forwarded message --------- > > From: Sam Ruby <ru...@intertwingly.net> > > Date: Mon, Nov 25, 2019, 4:15 PM > > Subject: Re: Does Whimsy need to have a copy of Bills? > > To: Whimsy dev <dev@whimsical.apache.org> > > > > > > adding Greg to email. > > > > Recap: a change is being proposed whereas whimsy will do the > > equivalent of the following command after every icla is processed: > > > > svn ls https://svn.apache.org/repos/private/documents/iclas --depth > > infinity > > > > Do you really need to use depth=infinity? The directory name is likely > sufficient information. ? > > depth=immediates (the default for svn ls) is going to be just a few > seconds. > > Currently this appears to take around forty to sixty elapsed seconds to > > process. > > > > Questions for Greg: > > 1) Does this proposed workload present an unreasonable load on the svn > > server? > > > > This should be fine as long as you don't put a "-v" switch in there. > That'll take 10-15 minutes as it reconstructs all the files on the server > and measures their size. > > The listing is just a single thread on the server, a fetch of the directory > names, and then assembly/delivery of that result. There really shouldn't be > any contention with other users, or heavy use of the CPU. > > > > 2) Are there any faster alternatives which get us a list of names but no > > data > > > > So I experimented with a hack. I did a "full checkout" of the iclas > directory, but stopped it after a single file was checked out. This left a > partial checkout. Subversion will tell the server "I have $these. what am I > missing?" when you run "svn status -u". You'll get a listing of the 12k > missing files. Takes about 7 seconds or so. > > Specifically: > $ # kill after reading/printing one line (the first file checked out) > $ svn co https://svn.apache.org/repos/private/documents/iclas | python -c > 'import signal,os,sys ; print sys.stdin.readline() ; > os.killpg(os.getpgrp(), signal.SIGHUP)' > A iclas/jiwei-guo > > svn: E200015: Caught signal > svn: E200042: Additional errors: > svn: E200015: Caught signal > Hangup > $ # get a recursive listing via status > $ time svn st -u iclas | wc -l > 12818 > > real 0m6.782s > user 0m3.461s > sys 0m2.358s > > The status output should be easy to parse (it is designed as a fixed-width > set of codes, then filename). > > Even if you do a full/normal checkout, note that "svn status -u" may be > useful. Depending on whether you need the content, or just the names, you > may want to migrate to the status-based approach. > > Oh! Just realized a better way, to avoid the hack/partial checkout. Even > better, just check out the "iclas" directory for the revision it was > created. It is an empty directory in that revision (a sibling directory > received a bunch of Member applications, but those won't be in this > checkout). > > $ svn -r 9696 co https://svn.apache.org/repos/private/documents/iclas > Checked out revision 9696. > > The "svn status" works the same against the above (empty) working copy. > Also at about 8 seconds. > > So. In summary, use "svn status" against a HEAD checkout, or against r9696 > for those who don't want the gigabytes of ICLA forms. > > A similar technique can be used for any of the other Whimsy data > directories, of course. To find when a particular directory was created: > > $ # run an "svn log" in reverse order, and limit/stop at the first log > entry. > $ svn log --stop-on-copy --limit 1 -r0:HEAD > https://svn.apache.org/repos/private/documents/iclas > ------------------------------------------------------------------------ > r9696 | jim | 2006-11-17 10:35:28 -0600 (Fri, 17 Nov 2006) | 3 lines > > Start loading of scanned docs. Start with creating > the dirs and upload the member apps > > ------------------------------------------------------------------------ > > Note that sometimes a directory is created with content, in that revision. > The "iclas" directory just happened to be created empty. But I imagine most > directories will be much smaller at their creation, than they are today. > (iow, don't expect them to always be empty at creation) > > Hope that helps, > -g > > > > > - Sam Ruby > > > > On Mon, Nov 25, 2019 at 3:44 PM sebb <seb...@gmail.com> wrote: > > > > > > On Mon, 25 Nov 2019 at 19:48, Sam Ruby <ru...@intertwingly.net> wrote: > > > > > > > On Mon, Nov 25, 2019 at 2:29 PM sebb <seb...@gmail.com> wrote: > > > > > > > > > > On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net> > > wrote: > > > > > > > > > > > On Sun, Nov 24, 2019 at 1:17 PM sebb <seb...@gmail.com> wrote: > > > > > > > > > > > > > > I was thinking of using svn ls to create a listing file which > > would > > > > be > > > > > > > cached locally. > > > > > > > > > > > > Unfortunately, some observations (numbers below are approximate): > > > > > > > > > > > > svn up on a populated iclas directory: one second > > > > > > > > > > > > svn ls on iclas: two seconds, but only returns depth one > > > > > > > > > > > > svn ls on iclas --depth infinity: one minute > > > > > > > > > > > > What's most concerning is not just the elapsed time, but that > this > > > > > > likely means that the call is expensive on the server, which may > > > > > > impact other users. > > > > > > > > > > > > > > > > > I see this as premature optimisation; we don't know whether svn > list > > is > > > > > more expensive than svn update overall. > > > > > There may be other reasons why list is slower. Nor do we know if > the > > > > > request will impact other users. > > > > > > > > > > Besides, if the code checks SVN info first, it will only need to > > fetch > > > > the > > > > > updated listing when there has been a change. > > > > > Those directories are not busy. > > > > > > > > > > Furthermore, every time a new test installation is set up, there is > > > > > definitely a large load on the server and network. > > > > > This network load in particular must be orders of magnitude greater > > than > > > > > for a listing. > > > > > > > > Sorry for not being clear. I would be very concerned if whimsy-vm4 > > > > were invoking svn ls --depth infinity every 10 minutes as the current > > > > cron job does. > > > > > > > > > That would not be the case. > > > > > > The job would use 'svn info' on the remote repo and only fetch the > > listing > > > if necessary. > > > > > > For the repos in question, changes are rare. > > > > > > > > > > Before any such change is deployed, it would be wise > > > > for us to check both with the infrastructure team and the subversion > > > > team (Greg likely can help with both). > > > > > > > > I'm less concerned about the overhead on development machines, and > > > > there I suspect that most users would be happy with a svn checkout > > > > --depth empty. > > > > > > > > > > > This would not allow testing of the functions that need to know the > list > > of > > > file names. > > > > > > > > > > - Sam Ruby > > > > > > >
