Regarding split & merge, I have default values filestore_merge_threshold = 10 filestore_split_multiple = 2
according to https://bugzilla.redhat.com/show_bug.cgi?id=1219974 the recommended values are filestore_merge_threshold = 40 filestore_split_multiple = 8 Is it something that I can easily change to default or lower values than proposed in case of further performance degradation ? I did tests of 4 pools: 2 replicated pools (x3 ) and 2 EC pools (k=6,m=3) The pool with the lowest bandwidth has osd tree structure like ├── 20.115s1_head │ └── DIR_5 │ └── DIR_1 │ ├── DIR_1 │ │ ├── DIR_0 │ │ ├── DIR_1 │ │ ├── DIR_2 │ │ │ ├── DIR_0 │ │ │ ├── DIR_1 │ │ │ ├── DIR_2 │ │ │ ├── DIR_3 │ │ │ ├── DIR_4 │ │ │ ├── DIR_5 │ │ │ ├── DIR_6 │ │ │ ├── DIR_7 │ │ │ ├── DIR_8 │ │ │ ├── DIR_9 │ │ │ ├── DIR_A │ │ │ ├── DIR_B │ │ │ ├── DIR_C │ │ │ ├── DIR_D │ │ │ ├── DIR_E │ │ │ └── DIR_F Tests results # rados bench -p default.rgw.buckets.data 10 write hints = 1 Maintaining 16 concurrent writes of 4194432 bytes to objects of size 4194432 for up to 10 seconds or 0 objects Object prefix: benchmark_data_sg08-09_180679 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 129 113 451.975 452.014 0.0376714 0.128027 2 16 209 193 385.964 320.01 0.119609 0.138517 3 16 235 219 291.974 104.003 0.0337624 0.13731 4 16 235 219 218.981 0 - 0.13731 5 16 266 250 199.983 62.0019 0.111673 0.238424 6 16 317 301 200.649 204.006 0.0340569 0.298489 7 16 396 380 217.124 316.01 0.0379956 0.283458 8 16 444 428 213.981 192.006 0.0304383 0.274193 9 16 485 469 208.426 164.005 0.391956 0.283421 10 16 496 480 191.983 44.0013 0.104497 0.292074 11 16 497 481 174.894 4.00012 0.999985 0.293545 12 16 497 481 160.32 0 - 0.293545 13 16 497 481 147.987 0 - 0.293545 14 16 497 481 137.417 0 - 0.293545 Total time run: 14.493353 Total writes made: 497 Write size: 4194432 Object size: 4194432 Bandwidth (MB/sec): 137.171 Stddev Bandwidth: 147.001 Max bandwidth (MB/sec): 452.014 Min bandwidth (MB/sec): 0 Average IOPS: 34 Stddev IOPS: 36 Max IOPS: 113 Min IOPS: 0 Average Latency(s): 0.464281 Stddev Latency(s): 1.09388 Max latency(s): 6.3723 Min latency(s): 0.023835 Cleaning up (deleting benchmark objects) Removed 497 objects Clean up completed and total clean up time :10.622382 # # rados bench -p benchmark_erasure_coded 10 write hints = 1 Maintaining 16 concurrent writes of 4202496 bytes to objects of size 4202496 for up to 10 seconds or 0 objects Object prefix: benchmark_data_sg08-09_180807 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 424 408 1635.11 1635.19 0.0490434 0.0379616 2 16 828 812 1627.03 1619.16 0.0616501 0.0388467 3 16 1258 1242 1659.06 1723.36 0.0304412 0.0384537 4 16 1659 1643 1646.03 1607.13 0.0155402 0.0387351 5 16 2053 2037 1632.61 1579.08 0.0453354 0.0390236 6 16 2455 2439 1629 1611.14 0.0485313 0.0392376 7 16 2649 2633 1507.34 777.516 0.0148972 0.0393161 8 16 2858 2842 1423.61 837.633 0.0157639 0.0449088 9 16 3245 3229 1437.75 1551.02 0.0200845 0.0444847 10 16 3629 3613 1447.85 1539 0.0654451 0.0441569 Total time run: 10.229591 Total writes made: 3630 Write size: 4202496 Object size: 4202496 Bandwidth (MB/sec): 1422.18 Stddev Bandwidth: 341.609 Max bandwidth (MB/sec): 1723.36 Min bandwidth (MB/sec): 777.516 Average IOPS: 354 Stddev IOPS: 85 Max IOPS: 430 Min IOPS: 194 Average Latency(s): 0.0448612 Stddev Latency(s): 0.0712224 Max latency(s): 1.08353 Min latency(s): 0.0134629 Cleaning up (deleting benchmark objects) Removed 3630 objects Clean up completed and total clean up time :2.321669 # # rados bench -p volumes 10 write hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_sg08-09_180651 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 336 320 1279.89 1280 0.0309006 0.0472524 2 16 653 637 1273.84 1268 0.0465151 0.0495701 3 16 956 940 1253.17 1212 0.0337327 0.0504146 4 16 1256 1240 1239.85 1200 0.0177263 0.0509145 5 16 1555 1539 1231.05 1196 0.0364991 0.0516724 6 16 1868 1852 1234.51 1252 0.0260964 0.0510236 7 16 2211 2195 1254.13 1372 0.040738 0.050847 8 16 2493 2477 1238.35 1128 0.0228582 0.0514979 9 16 2838 2822 1254.07 1380 0.0265224 0.0508641 10 16 3116 3100 1239.85 1112 0.0160151 0.0513104 Total time run: 10.192091 Total writes made: 3117 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1223.3 Stddev Bandwidth: 89.9383 Max bandwidth (MB/sec): 1380 Min bandwidth (MB/sec): 1112 Average IOPS: 305 Stddev IOPS: 22 Max IOPS: 345 Min IOPS: 278 Average Latency(s): 0.0518144 Stddev Latency(s): 0.0529575 Max latency(s): 0.663523 Min latency(s): 0.0122169 Cleaning up (deleting benchmark objects) Removed 3117 objects Clean up completed and total clean up time :0.212296 # # rados bench -p benchmark_replicated 10 write hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_sg08-09_180779 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 309 293 1171.94 1172 0.0233267 0.0508877 2 16 632 616 1231.87 1292 0.0258237 0.049612 3 16 959 943 1257.19 1308 0.0335615 0.0483464 4 16 1276 1260 1259.85 1268 0.031461 0.0504689 5 16 1643 1627 1301.44 1468 0.0274032 0.0489651 6 16 1991 1975 1316.51 1392 0.0408116 0.0483596 7 16 2328 2312 1320.98 1348 0.0242298 0.048175 8 16 2677 2661 1330.33 1396 0.097513 0.047962 9 16 3042 3026 1344.72 1460 0.0196724 0.0474078 10 16 3384 3368 1347.03 1368 0.0426199 0.0472573 Total time run: 10.482871 Total writes made: 3384 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1291.25 Stddev Bandwidth: 90.4861 Max bandwidth (MB/sec): 1468 Min bandwidth (MB/sec): 1172 Average IOPS: 322 Stddev IOPS: 22 Max IOPS: 367 Min IOPS: 293 Average Latency(s): 0.048763 Stddev Latency(s): 0.0547666 Max latency(s): 0.938211 Min latency(s): 0.0121556 Cleaning up (deleting benchmark objects) Removed 3384 objects Clean up completed and total clean up time :0.239684 # Luis why did you advise against increasing pg_num pgp_num ? I'm wondering which option is better: increasing pg_num or filestore_merge_threshold and filestore_split_multiple ? Thanks Jakub On Thu, Feb 1, 2018 at 9:38 AM, Jaroslaw Owsiewski < jaroslaw.owsiew...@allegro.pl> wrote: > Hi, > > maybe "split is on the floor"? > > Regards > -- > Jarek >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
