Hi Something that's been bugging me for a while is I am trying to diagnose iowait time within KVM guests. Guests doing reads or writes tend do about 50% to 90% iowait but the host itself is only doing about 1% to 2% iowait. So the result is the guests are extremely slow.
I currently run 3x hosts each with a single SSD and single HDD OSD in cache-teir writeback mode. Although the SSD (Samsung 850 EVO 120GB) is not a great one it should at least perform reasonably compared to a hard disk and doing some direct SSD tests I get approximately 100MB/s write and 200MB/s read on each SSD. When I run rados bench though, the benchmark starts with a not great but okay speed and as the benchmark progresses it just gets slower and slower till it's worse than a USB hard drive. The SSD cache pool is 120GB in size (360GB RAW) and in use at about 90GB. I have tried tuning the XFS mount options as well but it has had little effect. Understandably the server spec is not great but I don't expect performance to be that bad. OSD config: [osd] osd crush update on start = false osd mount options xfs = "rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M" Servers spec: Dual Quad Core XEON E5410 and 32GB RAM in each server 10GBE @ 10G speed with 8000byte Jumbo Frames. Rados bench result: (starts at 50MB/s average and plummets down to 11MB/s) sudo rados bench -p rbd 50 write --no-cleanup -t 1 Maintaining 1 concurrent writes of 4194304 bytes for up to 50 seconds or 0 objects Object prefix: benchmark_data_osc-mgmt-1_10007 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 1 14 13 51.9906 52 0.0671911 0.074661 2 1 27 26 51.9908 52 0.0631836 0.0751152 3 1 37 36 47.9921 40 0.0691167 0.0802425 4 1 51 50 49.9922 56 0.0816432 0.0795869 5 1 56 55 43.9934 20 0.208393 0.088523 6 1 61 60 39.994 20 0.241164 0.0999179 7 1 64 63 35.9934 12 0.239001 0.106577 8 1 66 65 32.4942 8 0.214354 0.122767 9 1 72 71 31.55 24 0.132588 0.125438 10 1 77 76 30.3948 20 0.256474 0.128548 11 1 79 78 28.3589 8 0.183564 0.138354 12 1 82 81 26.9956 12 0.345809 0.145523 13 1 85 84 25.842 12 0.373247 0.151291 14 1 86 85 24.2819 4 0.950586 0.160694 15 1 86 85 22.6632 0 - 0.160694 16 1 90 89 22.2466 8 0.204714 0.178352 17 1 94 93 21.8791 16 0.282236 0.180571 18 1 98 97 21.5524 16 0.262566 0.183742 19 1 101 100 21.0495 12 0.357659 0.187477 20 1 104 103 20.597 12 0.369327 0.192479 21 1 105 104 19.8066 4 0.373233 0.194217 22 1 105 104 18.9064 0 - 0.194217 23 1 106 105 18.2582 2 2.35078 0.214756 24 1 107 106 17.6642 4 0.680246 0.219147 25 1 109 108 17.2776 8 0.677688 0.229222 26 1 113 112 17.2283 16 0.29171 0.230487 27 1 117 116 17.1828 16 0.255915 0.231101 28 1 120 119 16.9976 12 0.412411 0.235122 29 1 120 119 16.4115 0 - 0.235122 30 1 120 119 15.8645 0 - 0.235122 31 1 120 119 15.3527 0 - 0.235122 32 1 122 121 15.1229 2 0.319309 0.262822 33 1 124 123 14.9071 8 0.344094 0.266201 34 1 127 126 14.8215 12 0.33534 0.267913 35 1 129 128 14.6266 8 0.355403 0.269241 36 1 132 131 14.5536 12 0.581528 0.274327 37 1 132 131 14.1603 0 - 0.274327 38 1 133 132 13.8929 2 1.43621 0.28313 39 1 134 133 13.6392 4 0.894817 0.287729 40 1 134 133 13.2982 0 - 0.287729 41 1 135 134 13.0714 2 1.87878 0.299602 42 1 138 137 13.0459 12 0.309637 0.304601 43 1 140 139 12.9285 8 0.302935 0.304491 44 1 141 140 12.7256 4 1.5538 0.313415 45 1 142 141 12.5317 4 0.352417 0.313691 46 1 145 144 12.5201 12 0.322063 0.317458 47 1 145 144 12.2537 0 - 0.317458 48 1 145 144 11.9984 0 - 0.317458 49 1 145 144 11.7536 0 - 0.317458 50 1 146 145 11.5985 1 3.79816 0.341463
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
