Hi,

 

Can you post the commands you ran for both benchmarks, without knowing the 
block size, write pattern and queue depth, it’s hard to determine where the 
bottleneck might be.

 

I can see OSD sde has a very high service time, which could be a sign of a 
problem, does it always show up high during tests, or is this just a one off 
spike?

 

The service times as a whole also look slightly high, I’m assuming there is a 
write back cache on the HP controller?

 

It might also be worth checking the max queue depths of both target and 
initiator to make sure you are not hitting any limits.

 

Nick

 

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of ano nym
Sent: 11 December 2014 17:40
To: ceph-users@lists.ceph.com
Subject: [ceph-users] tgt / rbd performance

 

 

there is a ceph pool on a hp dl360g5 with 25 sas 10k (sda-sdy) on a msa70 which 
gives me about 600 MB/s continous write speed with rados write bench. tgt on 
the server with rbd backend uses this pool. mounting local(host) with iscsiadm, 
sdz is the virtual iscsi device. As you can see, sdz max out with 100%util at 
~55MB/s when writing to it. 

 

I know that tgt-rbd is more a proof-of-concept then production-ready. 

 

Anyway, is someone using it and/or are there any hints to speed it up?

 

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

          18.31    0.00   26.01   53.28    0.00    2.40

 

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util

sda               0.00     5.00    0.00   35.00     0.00  8316.50   475.23     
2.36   68.74    0.00   68.74  15.11  52.90

sdb               0.00     0.00    0.00   24.00     0.00  3718.50   309.88     
0.87   49.54    0.00   49.54  14.96  35.90

sdc               0.00     9.00    0.00   64.00     0.00 19739.00   616.84     
3.33   51.12    0.00   51.12   9.20  58.90

sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     
0.00    0.00    0.00    0.00   0.00   0.00

sde               0.00     0.00    0.00    2.00     0.00   520.00   520.00     
0.66   86.00    0.00   86.00  78.00  15.60

sdf               0.00     8.00    0.00  103.00     0.00 34638.00   672.58     
7.45   78.69    0.00   78.69   9.46  97.40

sdg               0.00     3.00    0.00   74.00     0.00 25472.00   688.43     
4.47   60.50    0.00   60.50   9.62  71.20

sdh               0.00     0.00    0.00   17.00     0.00  5760.00   677.65     
0.62   45.65    0.00   45.65  14.29  24.30

sdi               0.00     3.00    0.00   43.00     0.00 14724.00   684.84     
2.92   67.91    0.00   67.91  12.98  55.80

sdj               0.00     4.00    1.00   45.00     4.00 14152.00   615.48     
3.18   58.52   19.00   59.40  13.59  62.50

sdk               0.00     6.00    0.00   75.00     0.00 29047.50   774.60     
6.35   61.08    0.00   61.08  10.23  76.70

sdm               0.00     3.00    0.00   48.00     0.00 16488.00   687.00     
1.75   36.40    0.00   36.40   9.08  43.60

sdl               0.00     1.00    0.00   46.00     0.00 17412.00   757.04     
4.94  108.63    0.00  108.63   9.48  43.60

sdn               0.00     0.00    0.00   51.00     0.00 17692.00   693.80     
4.49   88.10    0.00   88.10   9.90  50.50

sdo               0.00     8.00    0.00   55.00     0.00 13574.00   493.60     
3.44   62.89    0.00   62.89   8.55  47.00

sdp               0.00     0.00    0.00   40.00     0.00 14488.00   724.40     
2.82   69.72    0.00   69.72  16.88  67.50

sdq               0.00     0.00    0.00    0.00     0.00     0.00     0.00     
0.00    0.00    0.00    0.00   0.00   0.00

sdr               0.00     1.00    0.00   32.00     0.00  8856.00   553.50     
0.78   24.22    0.00   24.22   9.53  30.50

sds               0.00     7.00    1.00   51.00     4.00 16132.50   620.63     
3.14   58.98   79.00   58.59  16.48  85.70

sdt               0.00     2.00    0.00   50.00     0.00 19040.00   761.60     
3.56   71.12    0.00   71.12   9.54  47.70

sdu               0.00     0.00    0.00    0.00     0.00     0.00     0.00     
0.00    0.00    0.00    0.00   0.00   0.00

sdw               0.00     0.00    0.00   16.00     0.00  6388.00   798.50     
0.95   77.62    0.00   77.62  11.56  18.50

sdx               0.00     1.00    0.00   28.00     0.00  9840.00   702.86     
1.40   49.86    0.00   49.86  12.43  34.80

sdy               0.00     2.00    0.00   54.00     0.00 20168.00   746.96     
5.64   72.65    0.00   72.65   9.72  52.50

sdv               0.00     0.00    0.00   15.00     0.00  4300.00   573.33     
0.49   33.07    0.00   33.07  19.00  28.50

 

sdz               0.00     0.00    0.00  115.00     0.00 57468.00   999.44   
143.50  736.48    0.00  736.48   8.70 100.00

 

b.r. Michael




_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to