On 03/20/2018 04:54 AM, Aaron Lu wrote:
This series is meant to improve zone->lock scalability for order 0 pages.
With will-it-scale/page_fault1 workload, on a 2 sockets Intel Skylake
server with 112 CPUs, CPU spend 80% of its time spinning on zone->lock.
Perf profile shows the most time consuming part under zone->lock is the
cache miss on "struct page", so here I'm trying to avoid those cache
misses.
I ran page_fault1 comparing 4.16-rc5 to your recent work, these four
patches plus the three others from your github branch zone_lock_rfc_v2.
Out of curiosity I also threw in another 4.16-rc5 with the pcp batch
size adjusted so high (10922 pages) that we always stay in the pcp lists
and out of buddy completely. I used your patch[*] in this last kernel.
This was on a 2-socket, 20-core broadwell server.
There were some small regressions a bit outside the noise at low process
counts (2-5) but I'm not sure they're repeatable. Anyway, it does
improve the microbenchmark across the board.
[*] lkml.kernel.org/r/20170919072342.GB7263 () intel ! com
,,586305.0,747,587731.0,1766
4.0,3.4,609505.0,1563,608007.0,1170
8.0,5.9,633145.0,1752,622690.0,1287
,,1131428.0,7397,1022890.0,7334
-1.0,-0.3,1119974.0,2558,1020102.0,5707
3.0,3.3,1165004.0,6232,1056689.0,6411
,,1590413.0,6412,1346900.0,8925
-0.7,1.0,1579816.0,7217,1360376.0,4418
2.3,3.4,1626925.0,5321,1392515.0,8180
,,2064476.0,5035,1656475.0,14714
-1.0,0.9,2043240.0,9070,1672036.0,5342
1.2,3.9,2089959.0,8287,1721614.0,7797
,,2486090.0,11178,1878085.0,15286
-0.1,1.1,2483021.0,15100,1898459.0,9295
1.3,4.0,2517602.0,13717,1952995.0,7481
,,2869756.0,9194,2058398.0,20580
0.4,3.5,2882220.0,20584,2129444.0,9689
2.4,6.0,2937618.0,14126,2182859.0,9650
,,3242589.0,15354,2231977.0,20188
0.9,3.2,3270796.0,15125,2303780.0,6607
2.0,6.7,3306528.0,17683,2381279.0,16507
,,3598209.0,10765,2361819.0,13509
1.2,4.5,3642407.0,14894,2469191.0,16250
2.0,8.1,3671834.0,17501,2552786.0,12112
,,3974345.0,12605,2511565.0,31986
2.3,4.8,4067553.0,11070,2632608.0,8158
2.5,9.8,4073111.0,12464,2758075.0,31433
,,4333026.0,12187,2636914.0,15065
2.4,5.8,4435852.0,21692,2789949.0,16400
3.2,10.3,4470666.0,23663,2907263.0,15052
,,4932423.0,12184,2675769.0,23925
2.7,3.6,5064666.0,18600,2771476.0,22438
3.6,7.9,5110434.0,21460,2888419.0,18181
,,5461255.0,14704,2631232.0,24390
1.7,2.5,5554957.0,20979,2697554.0,19370
3.1,5.8,5629143.0,22781,2782902.0,20347
,,5924367.0,11835,2445607.0,25821
1.4,4.9,6004723.0,19071,2566547.0,26031
2.9,8.4,6094087.0,17676,2651793.0,20051
,,6381611.0,16792,2277611.0,39558
1.2,4.4,6459693.0,18869,2377795.0,25094
2.4,12.4,6534837.0,24991,2560085.0,12638
,,6804737.0,13654,2232121.0,19409
1.1,4.8,6881730.0,18995,2338868.0,18923
2.4,12.5,6970318.0,28677,2510594.0,25891
,,7197145.0,17500,2313168.0,16694
1.2,4.5,7287072.0,28727,2418232.0,23120
2.6,7.6,7383613.0,17992,2489382.0,28385
,,7550498.0,15101,2226427.0,24769
1.6,4.4,7667641.0,24306,2324675.0,16265
2.8,5.3,7761917.0,29855,2345195.0,20660
,,7902794.0,12579,2188399.0,37454
1.7,6.0,8033876.0,21158,2320641.0,13053
2.8,8.9,8126732.0,27620,2383083.0,25173
,,8277506.0,15448,2198021.0,33075
2.1,6.4,8453255.0,17411,2339395.0,20221
3.0,7.7,8529130.0,29853,2366800.0,20139
,,8651034.0,19706,2239626.0,25694
2.2,3.2,8840988.0,24387,2311966.0,25694
3.1,9.3,8918721.0,34762,2448589.0,26228
,,8777023.0,11833,2259622.0,32155
2.5,5.7,8993348.0,25481,2389000.0,29464
3.5,6.3,9085319.0,27791,2401713.0,28706
,,8855202.0,27455,2268030.0,35914
3.0,4.1,9123705.0,34289,2361876.0,31917
4.2,11.9,9228843.0,33484,2536867.0,26001
,,8952897.0,21601,2280539.0,30530
3.5,10.1,9268365.0,30804,2510883.0,26312
4.6,10.3,9367048.0,30681,2514740.0,32897
,,9036582.0,19483,2374728.0,42892
3.7,2.2,9369541.0,33253,2425993.0,45953
5.0,4.5,9489640.0,28066,2482179.0,33821
,,9136041.0,18233,2336090.0,30037
4.0,6.1,9497501.0,34409,2478602.0,31960
4.7,10.5,9563832.0,34507,2581056.0,38516
,,9226630.0,17998,2326070.0,33782
3.7,7.8,9570547.0,27052,2508396.0,41848
4.4,10.4,9634192.0,40218,2566842.0,31116
,,9305784.0,24574,2391261.0,29548
3.8,6.1,9656252.0,39164,2536624.0,45738
4.5,5.2,9720210.0,31172,2516447.0,32348
,,9381004.0,19378,2442774.0,35745
2.6,1.3,9626125.0,65187,2474560.0,29978
4.1,4.4,9766045.0,54298,2549227.0,31200
,,9401844.0,27746,2456372.0,40550
2.7,3.8,9652161.0,51629,2549004.0,39991
3.5,6.9,9731681.0,51852,2625589.0,27822
,,9428320.0,17562,2509119.0,39752
2.1,6.1,9630472.0,50106,2662447.0,44347
3.1,5.7,9722152.0,50349,2651891.0,28519
,,9561062.0,21910,2392883.0,24181
2.0,7.1,9755774.0,60382,2563573.0,32132
3.5,15.0,9894735.0,45967,2752506.0,28517
,,9624859.0,30462,2480667.0,27055
2.7,5.5,9883943.0,61851,2618326.0,36656
4.5,15.4,10057320.0,46352,2863788.0,34022
,,9739896.0,35436,2476666.0,30301
3.1,8.2,10043706.0,60944,2680570.0,42346
4.6,16.8,10191082.0,51348,2893385.0,32717
,,9833955.0,39366,2628480.0,36567
3.5,2.7,10180871.0,50050,2699941.0,42805
5.0,7.3,10323136.0,50768,2820628.0,30552
,,9908832.0,20826,2666415.0,51379
3.5,0.5,10251385.0,58551,2679925.0,49144
5.1,5.7,10418155.0,51726,2817192.0,34043
,,9969311.0,20378,2563399.0,36720
3.5,4.8,10314449.0,60867,2686176.0,42926
5.4,9.1,10504881.0,53101,2796816.0,37461
,,10077169.0,36182,2584728.0,32672
3.1,7.4,10393453.0,63048,2775523.0,39745
4.7,11.9,10549870.0,45281,2893000.0,39102
,,10115997.0,25835,2653036.0,33259
2.7,5.7,10388901.0,63402,2803290.0,36021
4.6,11.2,10580796.0,63517,2949834.0,31422
,,10162757.0,33119,2681195.0,30592
2.5,3.0,10413010.0,76720,2761752.0,32472
4.0,9.5,10568061.0,65614,2935127.0,38463
,,10223472.0,41882,2670421.0,26049
2.4,5.0,10470977.0,58009,2803478.0,37111
4.1,7.4,10646450.0,54810,2868986.0,52724
kernel (#) ntask proc thr proc stdev thr
stdev
speedup speedup pgf/s pgf/s
4.16-rc5 (1) 1 586,305 747 587,731
1,766
lu-zone (2) 1 4.0% 3.4% 609,505 1,562 608,007
1,169
4.16-rc5-nz (3) 1 8.0% 5.9% 633,145 1,752 622,690
1,286
4.16-rc5 (1) 2 1,131,428 7,396 1,022,890
7,333
lu-zone (2) 2 -1.0% -0.3% 1,119,974 2,557 1,020,102
5,706
4.16-rc5-nz (3) 2 3.0% 3.3% 1,165,004 6,232 1,056,689
6,411
4.16-rc5 (1) 3 1,590,413 6,411 1,346,900
8,924
lu-zone (2) 3 -0.7% 1.0% 1,579,816 7,216 1,360,376
4,418
4.16-rc5-nz (3) 3 2.3% 3.4% 1,626,925 5,321 1,392,515
8,180
4.16-rc5 (1) 4 2,064,476 5,034 1,656,475
14,713
lu-zone (2) 4 -1.0% 0.9% 2,043,240 9,069 1,672,036
5,342
4.16-rc5-nz (3) 4 1.2% 3.9% 2,089,959 8,287 1,721,614
7,796
4.16-rc5 (1) 5 2,486,090 11,178 1,878,085
15,286
lu-zone (2) 5 -0.1% 1.1% 2,483,021 15,100 1,898,459
9,295
4.16-rc5-nz (3) 5 1.3% 4.0% 2,517,602 13,717 1,952,995
7,481
4.16-rc5 (1) 6 2,869,756 9,194 2,058,398
20,580
lu-zone (2) 6 0.4% 3.5% 2,882,220 20,583 2,129,444
9,689
4.16-rc5-nz (3) 6 2.4% 6.0% 2,937,618 14,126 2,182,859
9,650
4.16-rc5 (1) 7 3,242,589 15,354 2,231,977
20,188
lu-zone (2) 7 0.9% 3.2% 3,270,796 15,124 2,303,780
6,607
4.16-rc5-nz (3) 7 2.0% 6.7% 3,306,528 17,683 2,381,279
16,507
4.16-rc5 (1) 8 3,598,209 10,764 2,361,819
13,508
lu-zone (2) 8 1.2% 4.5% 3,642,407 14,893 2,469,191
16,250
4.16-rc5-nz (3) 8 2.0% 8.1% 3,671,834 17,501 2,552,786
12,112
4.16-rc5 (1) 9 3,974,345 12,605 2,511,565
31,986
lu-zone (2) 9 2.3% 4.8% 4,067,553 11,069 2,632,608
8,158
4.16-rc5-nz (3) 9 2.5% 9.8% 4,073,111 12,463 2,758,075
31,432
4.16-rc5 (1) 10 4,333,026 12,187 2,636,914
15,064
lu-zone (2) 10 2.4% 5.8% 4,435,852 21,691 2,789,949
16,399
4.16-rc5-nz (3) 10 3.2% 10.3% 4,470,666 23,663 2,907,263
15,052
4.16-rc5 (1) 11 4,932,423 12,183 2,675,769
23,924
lu-zone (2) 11 2.7% 3.6% 5,064,666 18,600 2,771,476
22,438
4.16-rc5-nz (3) 11 3.6% 7.9% 5,110,434 21,459 2,888,419
18,180
4.16-rc5 (1) 12 5,461,255 14,704 2,631,232
24,390
lu-zone (2) 12 1.7% 2.5% 5,554,957 20,978 2,697,554
19,369
4.16-rc5-nz (3) 12 3.1% 5.8% 5,629,143 22,781 2,782,902
20,346
4.16-rc5 (1) 13 5,924,367 11,835 2,445,607
25,821
lu-zone (2) 13 1.4% 4.9% 6,004,723 19,070 2,566,547
26,031
4.16-rc5-nz (3) 13 2.9% 8.4% 6,094,087 17,676 2,651,793
20,050
4.16-rc5 (1) 14 6,381,611 16,791 2,277,611
39,557
lu-zone (2) 14 1.2% 4.4% 6,459,693 18,869 2,377,795
25,093
4.16-rc5-nz (3) 14 2.4% 12.4% 6,534,837 24,990 2,560,085
12,638
4.16-rc5 (1) 15 6,804,737 13,653 2,232,121
19,408
lu-zone (2) 15 1.1% 4.8% 6,881,730 18,995 2,338,868
18,922
4.16-rc5-nz (3) 15 2.4% 12.5% 6,970,318 28,677 2,510,594
25,890
4.16-rc5 (1) 16 7,197,145 17,499 2,313,168
16,694
lu-zone (2) 16 1.2% 4.5% 7,287,072 28,727 2,418,232
23,120
4.16-rc5-nz (3) 16 2.6% 7.6% 7,383,613 17,991 2,489,382
28,385
4.16-rc5 (1) 17 7,550,498 15,101 2,226,427
24,768
lu-zone (2) 17 1.6% 4.4% 7,667,641 24,305 2,324,675
16,265
4.16-rc5-nz (3) 17 2.8% 5.3% 7,761,917 29,854 2,345,195
20,659
4.16-rc5 (1) 18 7,902,794 12,578 2,188,399
37,453
lu-zone (2) 18 1.7% 6.0% 8,033,876 21,158 2,320,641
13,053
4.16-rc5-nz (3) 18 2.8% 8.9% 8,126,732 27,619 2,383,083
25,172
4.16-rc5 (1) 19 8,277,506 15,448 2,198,021
33,074
lu-zone (2) 19 2.1% 6.4% 8,453,255 17,411 2,339,395
20,220
4.16-rc5-nz (3) 19 3.0% 7.7% 8,529,130 29,852 2,366,800
20,139
4.16-rc5 (1) 20 8,651,034 19,705 2,239,626
25,694
lu-zone (2) 20 2.2% 3.2% 8,840,988 24,387 2,311,966
25,693
4.16-rc5-nz (3) 20 3.1% 9.3% 8,918,721 34,761 2,448,589
26,227
4.16-rc5 (1) 21 8,777,023 11,833 2,259,622
32,154
lu-zone (2) 21 2.5% 5.7% 8,993,348 25,480 2,389,000
29,464
4.16-rc5-nz (3) 21 3.5% 6.3% 9,085,319 27,790 2,401,713
28,706
4.16-rc5 (1) 22 8,855,202 27,455 2,268,030
35,914
lu-zone (2) 22 3.0% 4.1% 9,123,705 34,288 2,361,876
31,917
4.16-rc5-nz (3) 22 4.2% 11.9% 9,228,843 33,483 2,536,867
26,000
4.16-rc5 (1) 23 8,952,897 21,601 2,280,539
30,530
lu-zone (2) 23 3.5% 10.1% 9,268,365 30,803 2,510,883
26,312
4.16-rc5-nz (3) 23 4.6% 10.3% 9,367,048 30,681 2,514,740
32,896
4.16-rc5 (1) 24 9,036,582 19,482 2,374,728
42,891
lu-zone (2) 24 3.7% 2.2% 9,369,541 33,253 2,425,993
45,952
4.16-rc5-nz (3) 24 5.0% 4.5% 9,489,640 28,066 2,482,179
33,820
4.16-rc5 (1) 25 9,136,041 18,232 2,336,090
30,036
lu-zone (2) 25 4.0% 6.1% 9,497,501 34,408 2,478,602
31,959
4.16-rc5-nz (3) 25 4.7% 10.5% 9,563,832 34,506 2,581,056
38,516
4.16-rc5 (1) 26 9,226,630 17,998 2,326,070
33,782
lu-zone (2) 26 3.7% 7.8% 9,570,547 27,052 2,508,396
41,848
4.16-rc5-nz (3) 26 4.4% 10.4% 9,634,192 40,217 2,566,842
31,115
4.16-rc5 (1) 27 9,305,784 24,573 2,391,261
29,547
lu-zone (2) 27 3.8% 6.1% 9,656,252 39,164 2,536,624
45,738
4.16-rc5-nz (3) 27 4.5% 5.2% 9,720,210 31,171 2,516,447
32,347
4.16-rc5 (1) 28 9,381,004 19,377 2,442,774
35,745
lu-zone (2) 28 2.6% 1.3% 9,626,125 65,187 2,474,560
29,977
4.16-rc5-nz (3) 28 4.1% 4.4% 9,766,045 54,298 2,549,227
31,199
4.16-rc5 (1) 29 9,401,844 27,746 2,456,372
40,549
lu-zone (2) 29 2.7% 3.8% 9,652,161 51,629 2,549,004
39,990
4.16-rc5-nz (3) 29 3.5% 6.9% 9,731,681 51,852 2,625,589
27,821
4.16-rc5 (1) 30 9,428,320 17,561 2,509,119
39,752
lu-zone (2) 30 2.1% 6.1% 9,630,472 50,106 2,662,447
44,347
4.16-rc5-nz (3) 30 3.1% 5.7% 9,722,152 50,348 2,651,891
28,518
4.16-rc5 (1) 31 9,561,062 21,909 2,392,883
24,180
lu-zone (2) 31 2.0% 7.1% 9,755,774 60,381 2,563,573
32,132
4.16-rc5-nz (3) 31 3.5% 15.0% 9,894,735 45,966 2,752,506
28,516
4.16-rc5 (1) 32 9,624,859 30,462 2,480,667
27,055
lu-zone (2) 32 2.7% 5.5% 9,883,943 61,850 2,618,326
36,655
4.16-rc5-nz (3) 32 4.5% 15.4% 10,057,320 46,352 2,863,788
34,021
4.16-rc5 (1) 33 9,739,896 35,435 2,476,666
30,301
lu-zone (2) 33 3.1% 8.2% 10,043,706 60,943 2,680,570
42,346
4.16-rc5-nz (3) 33 4.6% 16.8% 10,191,082 51,348 2,893,385
32,717
4.16-rc5 (1) 34 9,833,955 39,366 2,628,480
36,567
lu-zone (2) 34 3.5% 2.7% 10,180,871 50,050 2,699,941
42,804
4.16-rc5-nz (3) 34 5.0% 7.3% 10,323,136 50,767 2,820,628
30,551
4.16-rc5 (1) 35 9,908,832 20,826 2,666,415
51,379
lu-zone (2) 35 3.5% 0.5% 10,251,385 58,551 2,679,925
49,143
4.16-rc5-nz (3) 35 5.1% 5.7% 10,418,155 51,726 2,817,192
34,042
4.16-rc5 (1) 36 9,969,311 20,377 2,563,399
36,720
lu-zone (2) 36 3.5% 4.8% 10,314,449 60,867 2,686,176
42,925
4.16-rc5-nz (3) 36 5.4% 9.1% 10,504,881 53,100 2,796,816
37,461
4.16-rc5 (1) 37 10,077,169 36,182 2,584,728
32,672
lu-zone (2) 37 3.1% 7.4% 10,393,453 63,048 2,775,523
39,745
4.16-rc5-nz (3) 37 4.7% 11.9% 10,549,870 45,280 2,893,000
39,102
4.16-rc5 (1) 38 10,115,997 25,835 2,653,036
33,259
lu-zone (2) 38 2.7% 5.7% 10,388,901 63,402 2,803,290
36,020
4.16-rc5-nz (3) 38 4.6% 11.2% 10,580,796 63,516 2,949,834
31,421
4.16-rc5 (1) 39 10,162,757 33,118 2,681,195
30,591
lu-zone (2) 39 2.5% 3.0% 10,413,010 76,719 2,761,752
32,471
4.16-rc5-nz (3) 39 4.0% 9.5% 10,568,061 65,614 2,935,127
38,463
4.16-rc5 (1) 40 10,223,472 41,882 2,670,421
26,049
lu-zone (2) 40 2.4% 5.0% 10,470,977 58,008 2,803,478
37,111
4.16-rc5-nz (3) 40 4.1% 7.4% 10,646,450 54,810 2,868,986
52,724
