Thanks for reply!
not very good, but seems acceptable, how do you think the possible reasons?
osd erf counters helpful for this?
sudo fio --filename=/dev/sda2 --direct=1 --sync=1 --rw=write --bs=4k
--numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting
--name=journal-test
journal-test: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync,
iodepth=1
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/49928KB/0KB /s] [0/12.5K/0 iops]
[eta 00m:00s]
journal-test: (groupid=0, jobs=1): err= 0: pid=247168: Fri Feb 12 16:08:12
2016
write: io=2944.1MB, bw=50259KB/s, iops=12564, runt= 60001msec
clat (usec): min=43, max=1503, avg=77.47, stdev=17.37
lat (usec): min=43, max=1503, avg=77.75, stdev=17.42
clat percentiles (usec):
| 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 63],
| 30.00th=[ 67], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 79],
| 70.00th=[ 86], 80.00th=[ 91], 90.00th=[ 100], 95.00th=[ 105],
| 99.00th=[ 122], 99.50th=[ 129], 99.90th=[ 147], 99.95th=[ 155],
| 99.99th=[ 167]
bw (KB /s): min=44200, max=57680, per=100.00%, avg=50274.42,
stdev=2662.04
lat (usec) : 50=4.64%, 100=84.85%, 250=10.51%, 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%
cpu : usr=6.34%, sys=32.72%, ctx=1507971, majf=0, minf=98
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=753897/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=2944.1MB, aggrb=50258KB/s, minb=50258KB/s, maxb=50258KB/s,
mint=60001msec, maxt=60001msec
Disk stats (read/write):
sda: ios=0/1506216, merge=0/0, ticks=0/39449, in_queue=39025, util=65.04%
2016-02-12 15:41 GMT+08:00 Huan Zhang <huan.zhang...@gmail.com>:
> Hi,
>
> ceph VERY SLOW with 24 osd(SAMSUNG ssd).
> fio /dev/rbd0 iodepth=1 direct=1 IOPS only ~200
> fio /dev/rbd0 iodepth=32 direct=1 IOPS only ~3000
>
> But test single ssd deive with fio:
> fio iodepth=1 direct=1 IOPS ~15000
> fio iodepth=32 direct=1 IOPS ~30000
>
> Why ceph SO SLOW? Could you give me some help?
> Appreciated!
>
>
> My Enviroment:
> [root@szcrh-controller ~]# ceph -s
> cluster eb26a8b9-e937-4e56-a273-7166ffaa832e
> health HEALTH_WARN
> 1 mons down, quorum 0,1,2,3,4 ceph01,ceph02,ceph03,ceph04,
> ceph05
> monmap e1: 6 mons at {ceph01=
>
> 10.10.204.144:6789/0,ceph02=10.10.204.145:6789/0,ceph03=10.10.204.146:6789/0,ceph04=10.10.204.147:6789/0,ceph05=10.10.204.148:6789/0,ceph06=0.0.0.0:0/5
> }
> election epoch 6, quorum 0,1,2,3,4
> ceph01,ceph02,ceph03,ceph04,ceph05
> osdmap e114: 24 osds: 24 up, 24 in
> flags sortbitwise
> pgmap v2213: 1864 pgs, 3 pools, 49181 MB data, 4485 objects
> 144 GB used, 42638 GB / 42782 GB avail
> 1864 active+clean
>
> [root@ceph03 ~]# lsscsi
> [0:0:6:0] disk ATA SAMSUNG MZ7KM1T9 003Q /dev/sda
> [0:0:7:0] disk ATA SAMSUNG MZ7KM1T9 003Q /dev/sdb
> [0:0:8:0] disk ATA SAMSUNG MZ7KM1T9 003Q /dev/sdc
> [0:0:9:0] disk ATA SAMSUNG MZ7KM1T9 003Q /dev/sdd
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
