On Tue, Sep 22, 2015 at 12:57 AM, Johan Corveleyn <jcor...@gmail.com> wrote: > Last week I performed some performance tests on various types of > backends (FSFS formats 6 and 7, and FSX) served by 1.9, on machines > that are similar to our company's production svn setup. This is a > Solaris Sparc t5 with back-end on a Netapp cluster mounted via NFS; > client is a Windows client over a 100 Mbit LAN. > > Disclaimer: this is not a scientific test. Most parts were not fully > isolated from production, with normal fluctuations in available > bandwidth (netapp cluster, network, ...). YMMV. > > > == Summary == > > FSFS f7 (packed) and FSFS f6 without dir-deltification (created by > 1.8, unpacked) go head to head. Large log/blame-requests and export > are somewhat faster on f7, smaller log/blame/list-requests are > somewhat faster on f6_1.8. Disk storage: f7 is 40% smaller than f6_1.8 > (probably because of dir-deltification). It seems that f7 is faster > when disk caches are cold (but this is hard to test / quantify (how > cold is "cold")). I think I'll go with f7 for the cold cache > performance on large requests, the smaller footprint (and better > admin-ability because of the packed form), and the faster performance > on 'export'. > > Further: > > * f6 with dir-deltification (the default when creating a > --compatible-version=1.8 repository with svnadmin 1.9) is a no-go: > it's 2 to 3 times slower than f7 and f6_1.8 for log and blame. > > * The --block-read option makes f7 even faster for large requests > (record numbers for export, huge log and huge blame requests), but > also much slower for small requests. With block-size=8 the advantage > for large requests mostly remains, while small requests get a bit > faster (but still slower than f7 without block-read, or especially > f6_1.8). That leaves me with a dilemma: which use-case is more > important. I think I won't use block-read atm ... would be nice if > those regressions for small requests could be improved though > (ideally, the system would be able to guess whether block-read is > interesting for a particular high-level request). > > * FSX is rather slow in most tests (specifically: it's fine when disks > are cold, but doesn't speed up much when disk caches become hot -- cpu > usage is permanently high though, so I think FSX is cpu bound in many > of my tests). Also: not included in the numbers below, but export with > FSX was significantly faster with --cache-revprops *disabled* (up to > around 120 seconds vs 165 seconds when disk caches were hot). In other > words: although I did run all FSX tests with cache-revprops enabled, > that seems to slow down export. > > * Unpacked f7 and FSX are rather slow (didn't fully test them, it was > immediately obvious) -- these back-ends are best used in packed form. > OTOH, f6 gest best performance in unpacked form (pack manifest lookup > is overhead). > > > == Details & Numbers == > > Backend variations: > * f6_1.8 > repos created & loaded with 1.8 (no dir/prop-deltif) > on-disk size: 10 GB > server: svnserve -d --foreground -M128 -c0 -r repos_1.8 > * f6_1.9 > repos created with 1.9-beta1 with --compatible-version 1.8 > & loaded with 1.9-beta1 (has dir/prop-deltif enabled by default) > on-disk size: 6.1 GB > server: svnserve -d --foreground -M128 -c0 -r repos_1.9_f6 > * f7_pkd > repos created & loaded with 1.9-beta1 + packed > on-disk size: 5.8 GB > server: svnserve -d --foreground -M128 -c0 -r repos_1.9_f7_packed > * f7_pkd_br > same as above, but with block-read enabled > server: svnserve -d --foreground -M128 -c0 --block-read yes -r > repos_1.9_f7_packed > * f7_pkd_br8 > same as above, but with block-read enabled, and block-size=8 (fsfs.conf) > server: svnserve -d --foreground -M128 -c0 --block-read yes -r > repos_1.9_f7_packed > * fsx_pkd > repos created with --fs-type fsx & loaded with 1.9-beta1 + packed > on-disk size: 5.3 GB > server: svnserve -d --foreground -M128 -c0 --cache-revprops yes -r > repos_1.9_fsx_packed > > svnserve: 1.9.0-beta1 (provided by philip - Wandisco) on Solaris Sparc > client: svnbench 1.9.1 x64 (Sliksvn) on Windows 7 > Network between client and server: 100 Mbit LAN > > Hardware: > * Server: Solaris t5 sparc x64 (16 x sparcv9 3600 Mhz) 24 GB RAM > * Back-end: Netapp cluster (don't know the details), mounted via NFS > (The volume used here is relatively slow -- we have faster arrays for > other systems, but this one is what we have for SVN right now) > > Test Method: > * Per backend variation: > * Per type of test (export, log subtree, blame-huge, list, log file, > blame-normal): > * Start svnserve > * Run test from client 4 times > * Stop svnserve > > Infrastructure observations: > * 'iostat' on the backend volume showed high busy-percentages for each > first run, the three subsequent iterations consistently showed 0 % busy. > First run is still interesting, showing "cold" results. > * 'top' showed low cpu usage for each first run (waiting for I/O), high cpu > percentages for the three subsequent iterations. > > Times in seconds. The tables below are best viewed with a fixed-width font. > > null-export subtree (2,638 dirs; 21,713 files = 178 MB; 129,864 props = 2 MB) > =================== > run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd | > ----------------------------------------------------------------------------- > 1st (~cold) | 1053 | 147 | 374 | 162 | 73 | 227 | > 2nd | 81 | 71 | 67 | 85 | 56 | 165 | > 3rd | 80 | 81 | 65 | 86 | 56 | 167 | > 4th | 79 | 65 | 65 | 85 | 57 | 167 | > > null-log -v subtree (62,458 revs; 98,900 msg lines; 299,171 changes) > =================== > run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd | > ----------------------------------------------------------------------------- > 1st (~cold) | 1558 | 802 | 611 | 176 | 152 | 342 | > 2nd | 291 | 865 | 252 | 126 | 136 | 211 | > 3rd | 272 | 742 | 229 | 130 | 134 | 213 | > 4th | 266 | 722 | 232 | 133 | 134 | 208 | > > null-blame huge-deep-history (13,998 revs; 13,997 deltas; 33873 MB in deltas) > ============================ > run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd | > ----------------------------------------------------------------------------- > 1st (~cold) | 531 | 968 | 351 | 254 | 381 | 308 | > 2nd | 342 | 585 | 340 | 230 | 293 | 293 | > 3rd | 332 | 555 | 349 | 232 | 241 | 298 | > 4th | 332 | 582 | 308 | 233 | 239 | 319 | > > null-list -R subtree (2,639 dirs; 21,713 files; 0 locks) > ==================== > run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd | > ----------------------------------------------------------------------------- > 1st (~cold) | 6.8 | 11.5 | 6.7 | 10.7 | 7.0 | 16.2 | > 2nd | 5.3 | 7.0 | 6.3 | 10.1 | 6.9 | 15.0 | > 3rd | 4.9 | 6.2 | 6.3 | 9.9 | 6.8 | 14.1 | > 4th | 5.0 | 6.4 | 6.3 | 10.2 | 6.8 | 15.2 | > > null-log -v file (434 revs; 957 msg lines; 13,955 changes) > ================ > run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd | > ----------------------------------------------------------------------------- > 1st (~cold) | 17.5 | 30.7 | 15.3 | 23.7 | 8.0 | 16.5 | > 2nd | 3.8 | 10.2 | 3.9 | 14.1 | 6.8 | 11.6 | > 3rd | 3.6 | 8.4 | 3.9 | 13.9 | 7.1 | 11.4 | > 4th | 3.5 | 8.6 | 3.9 | 13.7 | 6.8 | 11.3 | > > null-blame normal (542 revs; 542 deltas; 22 MB in deltas) > ================= > run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd | > ----------------------------------------------------------------------------- > 1st (~cold) | 6.9 | 27.6 | 19.9 | 38.1 | 15.7 | 29.7 | > 2nd | 4.1 | 15.8 | 5.3 | 32.1 | 14.2 | 27.6 | > 3rd | 4.1 | 13.3 | 5.3 | 33.3 | 14.6 | 28.0 | > 4th | 4.1 | 13.3 | 5.5 | 32.6 | 14.5 | 28.0 | >
For those who are interested: a couple of months ago we did the final migration of our production repository to FSFS format 7 (dump/load-ed from old format 3 (svn 1.5) repository). It's running very smoothly ever since. Before migrating I tested different variations for some fsfs.conf parameters, to measure performance a bit. See numbers below (may not be directly comparable to the old numbers of previous tests above -- history has grown since then of course). These are not scientific tests, and I only did a limited number of iterations due to lack of time, so YMMV. TL;DR: We chose default settings for FSFS format 7 (packed), except for dir/props-deltification, which we disabled. As it turned out, dir/props-deltification in our case had a large performance cost for export and blame (with huge history). Other knobs we tested (max-linear-deltification and compression-level) didn't have that much of an impact. Repository variations: * f3: FSFS format 3 (svn 1.5 compatible): 15 GB * f7: FSFS format 7 packed (default settings): 6.8 GB * f7c1: f7 with compression-level=1: 6.9 GB * f7c0: f7 with compression-level=0: 10 GB * sd: f7 with max-linear-deltification=1: 8.7 GB * sdndp: f7 with max-linear-deltification=1 and enable-dir/props-deltification=false (forgot to note disk space) * ndp: f7 with enable-dir/props-deltification=false: 12 GB * ndpc1: f7 with enable-dir/props-deltification=false and compression-level=1: 12 GB Client: SVN 1.9.3 / Windows 7 64bit - Intel i7-3740QM 2.7 GHz 16 GB RAM Server: SVN 1.9.3 / Solaris 10 - t5 sparc x64 (32 x sparcv9 3600 Mhz) 64 GB RAM Server config: HTTPS / SVNInMemoryCacheSize 128 MB / SVNPathAuthz off Back-end filesystem: NetApp filer, mounted via NFS Network between client and server: 100 Mbit LAN Times in seconds null-export large-subtree (2,984 dirs; 22,602 files = 188 MB; 157,695 props = 4 MB) ====================== run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 442 | 112 | 247 | 258 | 256 | 319 | 125 | 132 | 2nd | 73 | 114 | 103 | 102 | 83 | 85 | 75 | 72 | 3rd | 73 | 124 | 95 | 101 | 107 | 75 | 73 | 75 | 4th | 72 | 112 | 93 | 98 | 150 | 74 | 71 | 61 | null-log -v large-subtree (72,731 revs; 110,870 msg lines; 365,983 changes) ====================== run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 3165 | 362 | 688 | 618 | 756 | 888 | 439 | 454 | 2nd | 1454 | 367 | 378 | 376 | 396 | 351 | 330 | 331 | 3rd | 1551 | 366 | 374 | 502 | 395 | 350 | 334 | 335 | 4th | 1398 | 364 | 371 | 380 | 392 | 351 | 357 | 335 | null-blame large-xmlfile-huge-history (15,183 revs; 15,179 deltas; 38266 MB in deltas) ====================== run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 2135 | 678 | 775 | 785 | 2114 | 2019 | 536 | 535 | 2nd | 1850 | 800 | 718 | 684 | 1698 | 1599 | 667 | 502 | 3rd | 1927 | 1048 | 796 | 693 | 1733 | 1597 | 516 | 605 | 4th | 1854 | 704 | 700 | 687 | 1696 | 1665 | 515 | 492 | null-list -R large-subtree (2,985 dirs; 22,602 files; 0 locks (?)) ======================= run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 24 | 13 | 13 | 13 | 13 | 12 | 12 | 11 | 2nd | 16 | 12 | 13 | 13 | 13 | 12 | 7 | 7 | 3rd | 15 | 12 | 7 | 13 | 13 | 12 | 12 | 11 | 4th | 15 | 12 | 7 | 7 | 13 | 11 | 11 | 11 | null-log -v file-avghistory (449 revs; 977 msg lines; 14,064 changes) ======================== run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 35 | 9 | 19 | 21 | 30 | 28 | 9 | 9 | 2nd | 20 | 12 | 8 | 8 | 6 | 5 | 6 | 5 | 3rd | 2 | 15 | 12 | 8 | 6 | 5 | 1 | 5 | 4th | 19 | 8 | 8 | 8 | 1 | 5 | 6 | 5 | null-blame average-file (602 revs; 602 deltas; 26 MB in deltas) =================== run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 8.3 | 10.6 | 14.1 | 16.9 | 24.5 | 25.3 | 9.7 | 8.6 | 2nd | 6.1 | 9.4 | 9.2 | 19.1 | 8.7 | 7.4 | 6.6 | 6.4 | 3rd | 6.2 | 10.0 | 10.1 | 2.6 | 1.6 | 7.1 | 6.8 | 6.3 | 4th | 6.0 | 2.1 | - | - | - | - | 1.3 | 6.5 | null-export small-tree (48 dirs; 394 files = 2 MB; 2,650 props = 70 KB) ========================== run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 9.2 | 2.3 | 5.1 | 10.2 | 6.4 | 5.5 | 2.9 | 2.9 | 2nd | 1.9 | 2.4 | 2.3 | 3.8 | 2.2 | 2.3 | 2.1 | 2.0 | 3rd | 1.9 | 2.2 | 2.0 | 5.8 | 1.7 | 1.9 | 1.5 | 1.7 | 4th | 1.6 | 1.5 | 1.6 | 3.3 | 1.5 | 1.8 | 1.7 | 1.4 | null-list branches/ (337 dirs; 0 files; 0 locks) ======================= run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 0.51 | 0.48 | 0.66 | 0.98 | 0.64 | 0.53 | 0.46 | 0.48 | 2nd | 0.46 | 0.53 | 0.52 | 0.58 | 0.48 | 0.45 | 0.49 | 0.46 | 3rd | 0.47 | 0.44 | 0.43 | 0.56 | 0.50 | 0.46 | 0.50 | 0.54 | 4th | 0.39 | 0.41 | 0.42 | 0.54 | 0.47 | 0.47 | 0.48 | 0.55 | null-log -v file-small-history (30 revs; 44 msg lines; 939 changes) ======================== run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 2.7 | 0.92 | 2.54 | 3.92 | 3.03 | 2.98 | 1.08 | 1.08 | 2nd | 1.3 | 0.84 | 0.91 | 1.86 | 0.81 | 0.74 | 0.76 | 0.76 | 3rd | 1.3 | 0.88 | 0.91 | 1.75 | 0.80 | 0.76 | 0.79 | 0.84 | 4th | 1.3 | 0.83 | 0.90 | 1.11 | 0.79 | 0.71 | 0.80 | 0.76 | null-export trunk (43,650 dirs; 204,837 files = 5.7 GB; 1,412,000 props = 36 MB) ================= run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1| --------------------------------------------------------------------- 1st (~cold) | 2249 | 979 | 1078 | 1213 | 1087 | 1205 | 929 | 896 | 2nd | 1048 | 969 | 962 | 954 | 901 | 961 | 974 | 844 | 3rd | 974 | 1084 | 1000 | 974 | 884 | 845 | 893 | 967 | 4th | 956 | 987 | 961 | 1059 | 882 | 840 | 839 | 841 | -- Johan
