I am running ceph jewel on 5 nodes with SSD OSDs.
I have an LVM image on a local RAID of spinning disks.
I have an RBD image on in a pool of SSD disks.
Both disks are used to run an almost identical CentOS 7 system.
Both systems were installed with the same kickstart, though the disk
partitioning is different.
I want to make writes on the the ceph image faster. For example, lots of
writes to MySQL (via MySQL replication) on a ceph SSD image are about 10x
slower than on a spindle RAID disk image. The MySQL server on ceph rbd
image has a hard time keeping up in replication.
So I wanted to test writes on these two systems
I have a 10GB compressed (gzip) file on both servers.
I simply gunzip the file on both systems, while running iostat.
The primary difference I see in the results is the average size of the
request to the disk.
CentOS7-lvm-raid-sata writes a lot faster to disk, and the size of the
request is about 40x, but the number of writes per second is about the same
This makes me want to conclude that the smaller size of the request for
CentOS7-ceph-rbd-ssd system is the cause of it being slow.
How can I make the size of the request larger for ceph rbd images, so I can
increase the write throughput?
Would this be related to having jumbo packets enabled in my ceph storage
network?
Here is a sample of the results:
[CentOS7-lvm-raid-sata]
$ gunzip large10gFile.gz &
$ iostat -x vg_root-lv_var -d 5 -m -N
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
...
vg_root-lv_var 0.00 0.00 30.60 452.20 13.60 222.15 1000.04
8.69 14.05 0.99 14.93 2.07 100.04
vg_root-lv_var 0.00 0.00 88.20 182.00 39.20 89.43 974.95
4.65 9.82 0.99 14.10 3.70 100.00
vg_root-lv_var 0.00 0.00 75.45 278.24 33.53 136.70 985.73
4.36 33.26 1.34 41.91 0.59 20.84
vg_root-lv_var 0.00 0.00 111.60 181.80 49.60 89.34 969.84
2.60 8.87 0.81 13.81 0.13 3.90
vg_root-lv_var 0.00 0.00 68.40 109.60 30.40 53.63 966.87
1.51 8.46 0.84 13.22 0.80 14.16
...
[CentOS7-ceph-rbd-ssd]
$ gunzip large10gFile.gz &
$ iostat -x vg_root-lv_data -d 5 -m -N
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
...
vg_root-lv_data 0.00 0.00 46.40 167.80 0.88 1.46
22.36 1.23 5.66 2.47 6.54 4.52 96.82
vg_root-lv_data 0.00 0.00 16.60 55.20 0.36 0.14
14.44 0.99 13.91 9.12 15.36 13.71 98.46
vg_root-lv_data 0.00 0.00 69.00 173.80 1.34 1.32
22.48 1.25 5.19 3.77 5.75 3.94 95.68
vg_root-lv_data 0.00 0.00 74.40 293.40 1.37 1.47
15.83 1.22 3.31 2.06 3.63 2.54 93.26
vg_root-lv_data 0.00 0.00 90.80 359.00 1.96 3.41
24.45 1.63 3.63 1.94 4.05 2.10 94.38
...
[iostat key]
w/s == The number (after merges) of write requests completed per second for
the device.
wMB/s == The number of sectors (kilobytes, megabytes) written to the device
per second.
avgrq-sz == The average size (in kilobytes) of the requests that were
issued to the device.
avgqu-sz == The average queue length of the requests that were issued to
the device.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
