Hi Igor I suspect you have very much the same problem as me.
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg22260.html Basically Samsung drives (like many SATA SSD's) are very much hit and miss so you will need to test them like described here to see if they are any good. http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ To give you an idea my average performance went from 11MB/s (with Samsung SSD) to 30MB/s (without any SSD) on write performance. This is a very small cluster. Pieter On Aug 12, 2015, at 04:33 PM, Voloshanenko Igor <igor.voloshane...@gmail.com> wrote: Hi all, we have setup CEPH cluster with 60 OSD (2 diff types) (5 nodes, 12 disks on each, 10 HDD, 2 SSD) Also we cover this with custom crushmap with 2 root leaf ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -100 5.00000 root ssd -102 1.00000 host ix-s2-ssd 2 1.00000 osd.2 up 1.00000 1.00000 9 1.00000 osd.9 up 1.00000 1.00000 -103 1.00000 host ix-s3-ssd 3 1.00000 osd.3 up 1.00000 1.00000 7 1.00000 osd.7 up 1.00000 1.00000 -104 1.00000 host ix-s5-ssd 1 1.00000 osd.1 up 1.00000 1.00000 6 1.00000 osd.6 up 1.00000 1.00000 -105 1.00000 host ix-s6-ssd 4 1.00000 osd.4 up 1.00000 1.00000 8 1.00000 osd.8 up 1.00000 1.00000 -106 1.00000 host ix-s7-ssd 0 1.00000 osd.0 up 1.00000 1.00000 5 1.00000 osd.5 up 1.00000 1.00000 -1 5.00000 root platter -2 1.00000 host ix-s2-platter 13 1.00000 osd.13 up 1.00000 1.00000 17 1.00000 osd.17 up 1.00000 1.00000 21 1.00000 osd.21 up 1.00000 1.00000 27 1.00000 osd.27 up 1.00000 1.00000 32 1.00000 osd.32 up 1.00000 1.00000 37 1.00000 osd.37 up 1.00000 1.00000 44 1.00000 osd.44 up 1.00000 1.00000 48 1.00000 osd.48 up 1.00000 1.00000 55 1.00000 osd.55 up 1.00000 1.00000 59 1.00000 osd.59 up 1.00000 1.00000 -3 1.00000 host ix-s3-platter 14 1.00000 osd.14 up 1.00000 1.00000 18 1.00000 osd.18 up 1.00000 1.00000 23 1.00000 osd.23 up 1.00000 1.00000 28 1.00000 osd.28 up 1.00000 1.00000 33 1.00000 osd.33 up 1.00000 1.00000 39 1.00000 osd.39 up 1.00000 1.00000 43 1.00000 osd.43 up 1.00000 1.00000 47 1.00000 osd.47 up 1.00000 1.00000 54 1.00000 osd.54 up 1.00000 1.00000 58 1.00000 osd.58 up 1.00000 1.00000 -4 1.00000 host ix-s5-platter 11 1.00000 osd.11 up 1.00000 1.00000 16 1.00000 osd.16 up 1.00000 1.00000 22 1.00000 osd.22 up 1.00000 1.00000 26 1.00000 osd.26 up 1.00000 1.00000 31 1.00000 osd.31 up 1.00000 1.00000 36 1.00000 osd.36 up 1.00000 1.00000 41 1.00000 osd.41 up 1.00000 1.00000 46 1.00000 osd.46 up 1.00000 1.00000 51 1.00000 osd.51 up 1.00000 1.00000 56 1.00000 osd.56 up 1.00000 1.00000 -5 1.00000 host ix-s6-platter 12 1.00000 osd.12 up 1.00000 1.00000 19 1.00000 osd.19 up 1.00000 1.00000 24 1.00000 osd.24 up 1.00000 1.00000 29 1.00000 osd.29 up 1.00000 1.00000 34 1.00000 osd.34 up 1.00000 1.00000 38 1.00000 osd.38 up 1.00000 1.00000 42 1.00000 osd.42 up 1.00000 1.00000 50 1.00000 osd.50 up 1.00000 1.00000 53 1.00000 osd.53 up 1.00000 1.00000 57 1.00000 osd.57 up 1.00000 1.00000 -6 1.00000 host ix-s7-platter 10 1.00000 osd.10 up 1.00000 1.00000 15 1.00000 osd.15 up 1.00000 1.00000 20 1.00000 osd.20 up 1.00000 1.00000 25 1.00000 osd.25 up 1.00000 1.00000 30 1.00000 osd.30 up 1.00000 1.00000 35 1.00000 osd.35 up 1.00000 1.00000 40 1.00000 osd.40 up 1.00000 1.00000 45 1.00000 osd.45 up 1.00000 1.00000 49 1.00000 osd.49 up 1.00000 1.00000 52 1.00000 osd.52 up 1.00000 1.00000 Then create 2 pools, 1 on HDD (platters), 1 on SSD/ and put SSD pul in from of HDD pool (cache tier) now we receive very bad performance results from cluster. Even with rados bench we received very unstable performance with even zero speed. So it's create very big issues for our clients. I try to tune all possible values, including OSD, but still no luck. Also very unbelievble situation, when i do ceph tell... bench on SSD OSD - i receive about 20MB/s If for HDD - 67 MB/s... I don;t understand why cache pools which consist of SSD works so bad... We used Samsung 850 Pro 256 Gb as SSDs Can you guys give me advice please... Also very idiotic thing, when i set cache-mode to forward and try to flush-evict all object (not all object evicted, some busy (locked on KVM sides). but now i receive quite stable results for rados bench Total time run: 30.275871 Total writes made: 2076 Write size: 4194304 Bandwidth (MB/sec): 274.278 Stddev Bandwidth: 75.1445 Max bandwidth (MB/sec): 368 Min bandwidth (MB/sec): 0 Average Latency: 0.232892 Stddev Latency: 0.240356 Max latency: 2.01436 Min latency: 0.0716344 Without zeros, etc... So i don't understand how it's possible. Also interesting thing, when i disable overlay for pool, rados bench become around 70MB/s as for ordinary HDD, but in same time rados bench for SSD pool, which not used anymore show same bad results... So please, give me some direction to deeg... _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
