Hello, On Wed, 4 Jun 2014 22:36:00 +0800 Indra Pramana wrote:
> Hi Christian, > > Good day to you, and thank you for your reply. > > Just now I managed to identify 3 more OSDs which were slow and needed to > be trimmed. Here is a longer (1 minute) result of rados bench after the > trimming: > This is the second time I see you mentioning needing to trim OSDs. Does that mean your actual storage is on SDDs? If only your journals are on SSDs (and nothing else) a trim (how do you trim them?) should have no effect at all. > http://pastebin.com/YFTbLyHA > You should run atop on all of your storage nodes and watch all OSD disks when your cluster stalls. If you have too many nodes/OSDs to watch them all at the same time, use the logging functionality of atop (probably with a lower interval than the standard 10 seconds) and review things after a bench run. I have a hard time believing that your entire cluster just stopped processing things for 10 seconds there, but I bet an OSD or node stalled. > ==== > Total time run: 69.441936 > Total writes made: 3773 > Write size: 4096000 > Bandwidth (MB/sec): 212.239 > > Stddev Bandwidth: 247.672 > Max bandwidth (MB/sec): 921.875 > Min bandwidth (MB/sec): 0 > Average Latency: 0.58602 > Stddev Latency: 2.39341 > Max latency: 32.1121 > Min latency: 0.04847 > ==== > > When I run this for 60 seconds, I noted some slow requests message when I > monitor using ceph -w, near the end of the 60-second period. > > I have verified that all OSDs have I/O speed of > 220 MB/s after I > trimmed the remaining slow ones just now. I noted that some SSDs are > having 250 MB/s of I/O speed when I take it out of cluster, but then > drop to 150 MB/s -ish after I put back into the cluster. > Also having to trim SSDs to regain performance suggests that you probably aren't using Intel DC ones. Some (most really)) SSDs are known to have massive delays (latencies) when having to do a garbage collection or other internal functions. > Could it be due to the latency? You mentioned that average latency of 0.5 > is pretty horrible. How can I find what contributes to the latency and > how to fix the problem? Really at loss now. :( > The latency is a combination of all delays, in your case I'm sure it is storage related. Christian > Looking forward to your reply, thank you. > > Cheers. > > > > > On Mon, Jun 2, 2014 at 4:56 PM, Christian Balzer <ch...@gol.com> wrote: > > > > > Hello, > > > > On Mon, 2 Jun 2014 16:15:22 +0800 Indra Pramana wrote: > > > > > Dear all, > > > > > > I have managed to identify some slow OSDs and journals and have since > > > replaced them. RADOS benchmark of the whole cluster is now fast, much > > > improved from last time, showing the cluster can go up to 700+ MB/s. > > > > > > ===== > > > Maintaining 16 concurrent writes of 4194304 bytes for up to 10 > > > seconds or 0 objects > > > Object prefix: benchmark_data_hv-kvm-01_6931 > > > sec Cur ops started finished avg MB/s cur MB/s last lat > > > avg lat 0 0 0 0 0 0 > > > - 0 1 16 214 198 791.387 792 > > > 0.260687 0.074689 2 16 275 259 517.721 244 > > > 0.079697 0.0861397 3 16 317 301 401.174 > > > 168 0.209022 0.115348 4 16 317 301 > > > 300.902 0 - 0.115348 5 16 356 340 > > > 271.924 78 0.040032 0.172452 6 16 389 373 > > > 248.604 132 0.038983 0.221213 7 16 411 395 > > > 225.662 88 0.048462 0.211686 8 16 441 425 > > > 212.454 120 0.048722 0.237671 9 16 474 458 > > > 203.513 132 0.041285 0.226825 10 16 504 > > > 488 195.161 120 0.041899 0.224044 11 16 > > > 505 489 177.784 4 0.622238 0.224858 12 > > > 16 505 489 162.97 0 - 0.224858 Total > > > time run: 12.142654 Total writes made: 505 > > > Write size: 4194304 > > > Bandwidth (MB/sec): 166.356 > > > > > > Stddev Bandwidth: 208.41 > > > Max bandwidth (MB/sec): 792 > > > Min bandwidth (MB/sec): 0 > > > Average Latency: 0.384178 > > > Stddev Latency: 1.10504 > > > Max latency: 9.64224 > > > Min latency: 0.031679 > > > ===== > > > > > This might be better than the last result, but it still shows the same > > massive variance in latency and a pretty horrible average latency. > > > > Also you want to run this test for a lot longer, looking at the > > bandwidth progression it seems to drop over time. > > I'd expect the sustained bandwidth over a minute or so be below > > 100MB/s. > > > > > > > However, dd test result on guest VM is still slow. > > > > > > ===== > > > root@test1# dd bs=1M count=256 if=/dev/zero of=test conv=fdatasync > > > oflag=direct > > > 256+0 records in > > > 256+0 records out > > > 268435456 bytes (268 MB) copied, 17.1829 s, 15.6 MB/s > > > ===== > > > > > You're kinda comparing apples to oranges here. > > Firstly, the block size isn't the same, running a rbd bench with 1MB > > blocks shows about a 25% decrease in bandwidth. > > > > Secondly, is the VM user space, kernel space, what FS, etc. > > Mounting a RBD image formatted the same way in kernelspace on a host > > and doing the the dd test there would be a better comparison. > > > > Christian > > > I thought I have fixed the problem by replacing all those bad OSDs > > > and journals but apparently it doesn't resolve the problem. > > > > > > Is there any throttling settings which prevents the guest VMs to get > > > the I/O write speed that it's entitled to? > > > > > > Looking forward to your reply, thank you. > > > > > > Cheers. > > > > > > > > > > > > > > > On Tue, Apr 29, 2014 at 8:54 PM, Christian Balzer <ch...@gol.com> > > > wrote: > > > > > > > On Thu, 24 Apr 2014 13:51:49 +0800 Indra Pramana wrote: > > > > > > > > > Hi Christian, > > > > > > > > > > Good day to you, and thank you for your reply. > > > > > > > > > > On Wed, Apr 23, 2014 at 11:41 PM, Christian Balzer > > > > > <ch...@gol.com> > > > > wrote: > > > > > > > > > > > > > > Using 32 concurrent writes, result is below. The speed > > > > > > > > > really fluctuates. > > > > > > > > > > > > > > > > > > Total time run: 64.31704964.317049 > > > > > > > > > Total writes made: 1095 > > > > > > > > > Write size: 4194304 > > > > > > > > > Bandwidth (MB/sec): 68.100 > > > > > > > > > > > > > > > > > > Stddev Bandwidth: 44.6773 > > > > > > > > > Max bandwidth (MB/sec): 184 > > > > > > > > > Min bandwidth (MB/sec): 0 > > > > > > > > > Average Latency: 1.87761 > > > > > > > > > Stddev Latency: 1.90906 > > > > > > > > > Max latency: 9.99347 > > > > > > > > > Min latency: 0.075849 > > > > > > > > > > > > > > > > > That is really weird, it should get faster, not slower. ^o^ > > > > > > > > I assume you've run this a number of times? > > > > > > > > > > > > > > > > Also my apologies, the default is 16 threads, not 1, but > > > > > > > > that still isn't enough to get my cluster to full speed: > > > > > > > > --- > > > > > > > > Bandwidth (MB/sec): 349.044 > > > > > > > > > > > > > > > > Stddev Bandwidth: 107.582 > > > > > > > > Max bandwidth (MB/sec): 408 > > > > > > > > --- > > > > > > > > at 64 threads it will ramp up from a slow start to: > > > > > > > > --- > > > > > > > > Bandwidth (MB/sec): 406.967 > > > > > > > > > > > > > > > > Stddev Bandwidth: 114.015 > > > > > > > > Max bandwidth (MB/sec): 452 > > > > > > > > --- > > > > > > > > > > > > > > > > But what stands out is your latency. I don't have a 10GBE > > > > > > > > network to compare, but my Infiniband based cluster (going > > > > > > > > through at least one switch) gives me values like this: > > > > > > > > --- > > > > > > > > Average Latency: 0.335519 > > > > > > > > Stddev Latency: 0.177663 > > > > > > > > Max latency: 1.37517 > > > > > > > > Min latency: 0.1017 > > > > > > > > --- > > > > > > > > > > > > > > > > Of course that latency is not just the network. > > > > > > > > > > > > > > > > > > > > > > What else can contribute to this latency? Storage node load, > > > > > > > disk speed, anything else? > > > > > > > > > > > > > That and the network itself are pretty much it, you should know > > > > > > once you've run those test with atop or iostat on the storage > > > > > > nodes. > > > > > > > > > > > > > > > > > > > > > I would suggest running atop (gives you more information > > > > > > > > at one glance) or "iostat -x 3" on all your storage nodes > > > > > > > > during these tests to identify any node or OSD that is > > > > > > > > overloaded in some way. > > > > > > > > > > > > > > > > > > > > > > Will try. > > > > > > > > > > > > > Do that and let us know about the results. > > > > > > > > > > > > > > > > I have done some tests using iostat and noted some OSDs on a > > > > > particular storage node going up to the 100% limit when I run the > > > > > rados bench test. > > > > > > > > > Dumping lots of text will make people skip over your mails, you > > > > need to summarize and preferably understand yourself what these > > > > numbers mean. > > > > > > > > The iostat output is not too conclusive, as the numbers when > > > > reaching 100% utilization are not particular impressive. > > > > The fact that it happens though should make you look for anything > > > > different with these OSDs, from smartctl checks to PG > > > > distribution, as in "ceph pg dump" and then tallying up each PG. > > > > Also look at "ceph osd tree" and see if those OSDs or node have a > > > > higher weight than others. > > > > > > > > The atop line indicates that sdb was being read at a rate of > > > > 100MB/s and assuming that your benchmark was more or less the only > > > > thing running at that time this would mean something very odd is > > > > going on, as all the other OSDs were have no significant reads > > > > going on and all were being written at about the same speed. > > > > > > > > Christian > > > > > > > > > ==== > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > > 1.09 0.00 0.92 21.74 0.00 76.25 > > > > > > > > > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > > > > > avgrq-sz avgqu-sz await r_await w_await svctm %util > > > > > sda 0.00 0.00 4.33 42.00 73.33 6980.00 > > > > > 304.46 0.29 6.22 0.00 6.86 1.50 6.93 > > > > > sdb 0.00 0.00 0.00 17.67 0.00 6344.00 > > > > > 718.19 59.64 854.26 0.00 854.26 56.60 *100.00* > > > > > sdc 0.00 0.00 12.33 59.33 70.67 18882.33 > > > > > 528.92 36.54 509.80 64.76 602.31 10.51 75.33 > > > > > sdd 0.00 0.00 3.33 54.33 24.00 15249.17 > > > > > 529.71 1.29 22.45 3.20 23.63 1.64 9.47 > > > > > sde 0.00 0.33 0.00 0.67 0.00 4.00 > > > > > 12.00 0.30 450.00 0.00 450.00 450.00 30.00 > > > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > > 1.38 0.00 1.13 7.75 0.00 89.74 > > > > > > > > > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > > > > > avgrq-sz avgqu-sz await r_await w_await svctm %util > > > > > sda 0.00 0.00 5.00 69.00 30.67 19408.50 > > > > > 525.38 4.29 58.02 0.53 62.18 2.00 14.80 > > > > > sdb 0.00 0.00 7.00 63.33 41.33 20911.50 > > > > > 595.82 13.09 826.96 88.57 908.57 5.48 38.53 > > > > > sdc 0.00 0.00 2.67 30.00 17.33 6945.33 > > > > > 426.29 0.21 6.53 0.50 7.07 1.59 5.20 > > > > > sdd 0.00 0.00 2.67 58.67 16.00 20661.33 > > > > > 674.26 4.89 79.54 41.00 81.30 2.70 16.53 > > > > > sde 0.00 0.00 0.00 1.67 0.00 6.67 > > > > > 8.00 0.01 3.20 0.00 3.20 1.60 0.27 > > > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > > 0.97 0.00 0.55 6.73 0.00 91.75 > > > > > > > > > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > > > > > avgrq-sz avgqu-sz await r_await w_await svctm %util > > > > > sda 0.00 0.00 1.67 15.33 21.33 120.00 > > > > > 16.63 0.02 1.18 0.00 1.30 0.63 1.07 > > > > > sdb 0.00 0.00 4.33 62.33 24.00 13299.17 > > > > > 399.69 2.68 11.18 1.23 11.87 1.94 12.93 > > > > > sdc 0.00 0.00 0.67 38.33 70.67 7881.33 > > > > > 407.79 37.66 202.15 0.00 205.67 13.61 53.07 > > > > > sdd 0.00 0.00 3.00 17.33 12.00 166.00 > > > > > 17.51 0.05 2.89 3.11 2.85 0.98 2.00 > > > > > sde 0.00 0.00 0.00 0.00 0.00 0.00 > > > > > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > > 1.29 0.00 0.92 24.10 0.00 73.68 > > > > > > > > > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > > > > > avgrq-sz avgqu-sz await r_await w_await svctm %util > > > > > sda 0.00 0.00 0.00 45.33 0.00 4392.50 > > > > > 193.79 0.62 13.62 0.00 13.62 1.09 4.93 > > > > > sdb 0.00 0.00 0.00 8.67 0.00 3600.00 > > > > > 830.77 63.87 1605.54 0.00 1605.54 115.38 *100.00* > > > > > sdc 0.00 0.33 8.67 42.67 37.33 5672.33 > > > > > 222.45 16.88 908.78 1.38 1093.09 7.06 36.27 > > > > > sdd 0.00 0.00 0.33 31.00 1.33 629.83 > > > > > 40.29 0.06 1.91 0.00 1.94 0.94 2.93 > > > > > sde 0.00 0.00 0.00 0.33 0.00 1.33 > > > > > 8.00 0.12 368.00 0.00 368.00 368.00 12.27 > > > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > > 1.59 0.00 0.88 4.82 0.00 92.70 > > > > > > > > > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > > > > > avgrq-sz avgqu-sz await r_await w_await svctm %util > > > > > sda 0.00 0.00 0.00 29.00 0.00 235.00 > > > > > 16.21 0.06 1.98 0.00 1.98 0.97 2.80 > > > > > sdb 0.00 6.00 4.33 114.67 38.67 6422.33 > > > > > 108.59 9.19 513.19 265.23 522.56 2.08 24.80 > > > > > sdc 0.00 0.00 0.00 20.67 0.00 124.00 > > > > > 12.00 0.04 2.00 0.00 2.00 1.03 2.13 > > > > > sdd 0.00 5.00 1.67 81.00 12.00 546.17 > > > > > 13.50 0.10 1.21 0.80 1.22 0.39 3.20 > > > > > sde 0.00 0.00 0.00 0.00 0.00 0.00 > > > > > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > > > > ==== > > > > > > > > > > And the high utilisation is randomly affecting other OSDs as well > > > > > within the same node, and not only affecting one particular OSD. > > > > > > > > > > atop result on the node: > > > > > > > > > > ==== > > > > > ATOP - > > > > > ceph-osd-07 > > > > > 2014/04/24 > > > > > 13:49:12 > > > > > ------ > > > > > 10s elapsed > > > > > PRC | sys 1.77s | user 2.11s | > > > > > | | #proc 164 | | #trun 2 | > > > > > #tslpi 2817 | #tslpu 0 | | #zombie 0 | > > > > > clones 4 | | | #exit 0 | > > > > > CPU | sys 14% | user 20% | | irq 1% > > > > > | | | idle 632% | wait 133% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 1.79GHz | avgscal 54% | > > > > > cpu | sys 6% | user 7% | | irq 0% > > > > > | | | idle 19% | cpu006 w 68% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 2.42GHz | avgscal 73% | > > > > > cpu | sys 2% | user 3% | | irq 0% > > > > > | | | idle 88% | cpu002 w 7% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 1.68GHz | avgscal 50% | > > > > > cpu | sys 2% | user 2% | | irq 0% > > > > > | | | idle 86% | cpu003 w 10% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 1.67GHz | avgscal 50% | > > > > > cpu | sys 2% | user 2% | | irq 0% > > > > > | | | idle 75% | cpu001 w 21% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 1.83GHz | avgscal 55% | > > > > > cpu | sys 1% | user 2% | | irq 1% > > > > > | | | idle 70% | cpu000 w 26% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 1.85GHz | avgscal 56% | > > > > > cpu | sys 1% | user 2% | | irq 0% > > > > > | | | idle 97% | cpu004 w 1% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 1.64GHz | avgscal 49% | > > > > > cpu | sys 1% | user 1% | | irq 0% > > > > > | | | idle 98% | cpu005 w 0% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 1.60GHz | avgscal 48% | > > > > > cpu | sys 0% | user 1% | | irq 0% > > > > > | | | idle 98% | cpu007 w 0% > > > > > | | | steal 0% | guest 0% > > > > > | | avgf 1.60GHz | avgscal 48% | > > > > > CPL | avg1 1.12 | | avg5 0.90 > > > > > | | avg15 0.72 | | > > > > > | | csw 103682 | | intr 34330 > > > > > | | | | numcpu 8 | > > > > > MEM | tot 15.6G | | free 158.2M | cache 13.7G > > > > > | | dirty 101.4M | buff 18.2M | | > > > > > slab 574.6M | | | | > > > > > | | | > > > > > SWP | tot 518.0M | | free 489.6M | > > > > > | | | | > > > > > | | | | | > > > > > vmcom 5.2G | | vmlim 8.3G | > > > > > PAG | scan 327450 | | | stall 0 > > > > > | | | | > > > > > | | | swin 0 | > > > > > | | | swout 0 | > > > > > DSK | sdb | | busy 90% | read 8115 > > > > > | | write 695 | KiB/r 130 | | > > > > > KiB/w 194 | MBr/s 103.34 | | MBw/s 13.22 | > > > > > avq 4.61 | | avio 1.01 ms | > > > > > DSK | sdc | | busy 32% | read 23 > > > > > | | write 431 | KiB/r 6 | | > > > > > KiB/w 318 | MBr/s 0.02 | | MBw/s 13.41 | > > > > > avq 34.86 | | avio 6.95 ms | > > > > > DSK | sda | | busy 32% | read 25 > > > > > | | write 674 | KiB/r 6 | | > > > > > KiB/w 193 | MBr/s 0.02 | | MBw/s 12.76 | > > > > > avq 41.00 | | avio 4.48 ms | > > > > > DSK | sdd | | busy 7% | read 26 > > > > > | | write 473 | KiB/r 7 | | > > > > > KiB/w 223 | MBr/s 0.02 | | MBw/s 10.31 | > > > > > avq 14.29 | | avio 1.45 ms | > > > > > DSK | sde | | busy 2% | read 0 > > > > > | | write 5 | KiB/r 0 | | > > > > > KiB/w 5 | MBr/s 0.00 | | MBw/s 0.00 | avq > > > > > 1.00 | | avio 44.8 ms | > > > > > NET | transport | tcpi 21326 | | tcpo > > > > > 27479 | udpi 0 | udpo 0 | tcpao 0 > > > > > | | tcppo 2 | tcprs 3 | tcpie 0 | > > > > > tcpor 0 | | udpnp 0 | udpip 0 | > > > > > NET | network | | ipi 21326 | ipo 14340 > > > > > | | ipfrw 0 | deliv 21326 | > > > > > | | | | | > > > > > icmpi 0 | | icmpo 0 | > > > > > NET | p2p2 ---- | pcki 12659 | | pcko > > > > > 20931 | si 124 Mbps | | so 107 Mbps | coll > > > > > 0 | mlti 0 | | erri 0 | erro 0 > > > > > | | drpi 0 | drpo 0 | > > > > > NET | p2p1 ---- | pcki 8565 | | pcko > > > > > 6443 | si 106 Mbps | | so 7911 Kbps | coll > > > > > 0 | mlti 0 | | erri 0 | erro 0 > > > > > | | drpi 0 | drpo 0 | > > > > > NET | lo ---- | pcki 108 | | pcko > > > > > 108 | si 8 Kbps | | so 8 Kbps | coll > > > > > 0 | mlti 0 | | erri 0 | erro 0 > > > > > | | drpi 0 | drpo 0 | > > > > > > > > > > PID RUID EUID THR > > > > > SYSCPU USRCPU VGROW RGROW > > > > > RDDSK WRDSK ST EXC S > > > > > CPUNR CPU CMD 1/1 > > > > > 6881 root root 538 > > > > > 0.74s 0.94s 0K 256K > > > > > 1.0G 121.3M -- - S > > > > > 3 17% ceph-osd > > > > > 28708 root root 720 > > > > > 0.30s 0.69s 512K -8K > > > > > 160K 157.7M -- - S > > > > > 3 10% ceph-osd > > > > > 31569 root root 678 > > > > > 0.21s 0.30s 512K -584K > > > > > 156K 162.7M -- - S > > > > > 0 5% ceph-osd > > > > > 32095 root root 654 > > > > > 0.14s 0.16s 0K 0K > > > > > 60K 105.9M -- - S > > > > > 0 3% ceph-osd > > > > > 61 root root 1 > > > > > 0.20s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 3 2% kswapd0 > > > > > 10584 root root 1 > > > > > 0.03s 0.02s 112K 112K > > > > > 0K 0K -- - R > > > > > 4 1% atop > > > > > 11618 root root 1 > > > > > 0.03s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 6 0% kworker/6:2 > > > > > 10 root root 1 > > > > > 0.02s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 0 0% rcu_sched > > > > > 38 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 6 0% ksoftirqd/6 > > > > > 1623 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 6 0% kworker/6:1H > > > > > 1993 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 2 0% flush-8:48 > > > > > 2031 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 2 0% flush-8:0 > > > > > 2032 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 0 0% flush-8:16 > > > > > 2033 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 2 0% flush-8:32 > > > > > 5787 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 4K 0K -- - S > > > > > 3 0% kworker/3:0 > > > > > 27605 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 1 0% kworker/1:2 > > > > > 27823 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 0 0% kworker/0:2 > > > > > 32511 root root 1 > > > > > 0.01s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 2 0% kworker/2:0 > > > > > 1536 root root 1 > > > > > 0.00s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 2 0% irqbalance > > > > > 478 root root 1 > > > > > 0.00s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 3 0% usb-storage > > > > > 494 root root 1 > > > > > 0.00s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 1 0% jbd2/sde1-8 > > > > > 1550 root root 1 > > > > > 0.00s 0.00s 0K 0K > > > > > 400K 0K -- - S > > > > > 1 0% xfsaild/sdb1 > > > > > 1750 root root 1 > > > > > 0.00s 0.00s 0K 0K > > > > > 128K 0K -- - S > > > > > 2 0% xfsaild/sdd1 > > > > > 1994 root root 1 > > > > > 0.00s 0.00s 0K 0K > > > > > 0K 0K -- - S > > > > > 1 0% flush-8:64 > > > > > ==== > > > > > > > > > > I have tried to trim the SSD drives but the problem seems to > > > > > persist. Last time trimming the SSD drives can help to improve > > > > > the performance. > > > > > > > > > > Any advice is greatly appreciated. > > > > > > > > > > Thank you. > > > > > > > > > > > > -- > > > > Christian Balzer Network/Systems Engineer > > > > ch...@gol.com Global OnLine Japan/Fusion Communications > > > > http://www.gol.com/ > > > > > > > > > > -- > > Christian Balzer Network/Systems Engineer > > ch...@gol.com Global OnLine Japan/Fusion Communications > > http://www.gol.com/ > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Christian Balzer Network/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
