Hi,
Can you post the commands you ran for both benchmarks, without knowing the block size, write pattern and queue depth, it’s hard to determine where the bottleneck might be. I can see OSD sde has a very high service time, which could be a sign of a problem, does it always show up high during tests, or is this just a one off spike? The service times as a whole also look slightly high, I’m assuming there is a write back cache on the HP controller? It might also be worth checking the max queue depths of both target and initiator to make sure you are not hitting any limits. Nick From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of ano nym Sent: 11 December 2014 17:40 To: ceph-users@lists.ceph.com Subject: [ceph-users] tgt / rbd performance there is a ceph pool on a hp dl360g5 with 25 sas 10k (sda-sdy) on a msa70 which gives me about 600 MB/s continous write speed with rados write bench. tgt on the server with rbd backend uses this pool. mounting local(host) with iscsiadm, sdz is the virtual iscsi device. As you can see, sdz max out with 100%util at ~55MB/s when writing to it. I know that tgt-rbd is more a proof-of-concept then production-ready. Anyway, is someone using it and/or are there any hints to speed it up? avg-cpu: %user %nice %system %iowait %steal %idle 18.31 0.00 26.01 53.28 0.00 2.40 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 5.00 0.00 35.00 0.00 8316.50 475.23 2.36 68.74 0.00 68.74 15.11 52.90 sdb 0.00 0.00 0.00 24.00 0.00 3718.50 309.88 0.87 49.54 0.00 49.54 14.96 35.90 sdc 0.00 9.00 0.00 64.00 0.00 19739.00 616.84 3.33 51.12 0.00 51.12 9.20 58.90 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde 0.00 0.00 0.00 2.00 0.00 520.00 520.00 0.66 86.00 0.00 86.00 78.00 15.60 sdf 0.00 8.00 0.00 103.00 0.00 34638.00 672.58 7.45 78.69 0.00 78.69 9.46 97.40 sdg 0.00 3.00 0.00 74.00 0.00 25472.00 688.43 4.47 60.50 0.00 60.50 9.62 71.20 sdh 0.00 0.00 0.00 17.00 0.00 5760.00 677.65 0.62 45.65 0.00 45.65 14.29 24.30 sdi 0.00 3.00 0.00 43.00 0.00 14724.00 684.84 2.92 67.91 0.00 67.91 12.98 55.80 sdj 0.00 4.00 1.00 45.00 4.00 14152.00 615.48 3.18 58.52 19.00 59.40 13.59 62.50 sdk 0.00 6.00 0.00 75.00 0.00 29047.50 774.60 6.35 61.08 0.00 61.08 10.23 76.70 sdm 0.00 3.00 0.00 48.00 0.00 16488.00 687.00 1.75 36.40 0.00 36.40 9.08 43.60 sdl 0.00 1.00 0.00 46.00 0.00 17412.00 757.04 4.94 108.63 0.00 108.63 9.48 43.60 sdn 0.00 0.00 0.00 51.00 0.00 17692.00 693.80 4.49 88.10 0.00 88.10 9.90 50.50 sdo 0.00 8.00 0.00 55.00 0.00 13574.00 493.60 3.44 62.89 0.00 62.89 8.55 47.00 sdp 0.00 0.00 0.00 40.00 0.00 14488.00 724.40 2.82 69.72 0.00 69.72 16.88 67.50 sdq 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdr 0.00 1.00 0.00 32.00 0.00 8856.00 553.50 0.78 24.22 0.00 24.22 9.53 30.50 sds 0.00 7.00 1.00 51.00 4.00 16132.50 620.63 3.14 58.98 79.00 58.59 16.48 85.70 sdt 0.00 2.00 0.00 50.00 0.00 19040.00 761.60 3.56 71.12 0.00 71.12 9.54 47.70 sdu 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdw 0.00 0.00 0.00 16.00 0.00 6388.00 798.50 0.95 77.62 0.00 77.62 11.56 18.50 sdx 0.00 1.00 0.00 28.00 0.00 9840.00 702.86 1.40 49.86 0.00 49.86 12.43 34.80 sdy 0.00 2.00 0.00 54.00 0.00 20168.00 746.96 5.64 72.65 0.00 72.65 9.72 52.50 sdv 0.00 0.00 0.00 15.00 0.00 4300.00 573.33 0.49 33.07 0.00 33.07 19.00 28.50 sdz 0.00 0.00 0.00 115.00 0.00 57468.00 999.44 143.50 736.48 0.00 736.48 8.70 100.00 b.r. Michael
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
