Hi Indra, >I'd probably start by looking at your nodes and check if the SSDs are > saturated > >or if they have high write access times. > > Any recommended benchmark tool to do this? Especially those specific to > Ceph OSDs and will not cause any impact on overall performance? >
A simple way would be using iostat. I think something like 'iostat 1 -x' will show both total disk utilization (rightmost column) and the 3 columns just before should depict read/write io-access time in realtime. For ssd they should be low (< 6-8). Column 3 and 4 (r/s, w/s - iops) might be of interest as well. Just 'iostat -x' should show avg since uptime iirc. A better way is prob. collecting and graphing it - I think something like collectd or munin can do that or you. We use munin to keep historical data about all nodes performance - we can see if performance drops over time, iops increase or wait/latency time for disks explode as well. I think the default munin-node on ubuntu will capture this data automatically. > > >Maybe test them individually directly on the cluster. > > Is it possible to test I/O speed from client directly to certain OSD on > the cluster? From what I understand, the PGs are being randomly mapped to > any of the OSDs (based on the crush map?). > I was thinking more lowlvl - like fio or bonnie++ or the like :). I think with fio you can get the most detailed picture of how your ssd perform in terms of throughput and iops. > > > >At some point in time we accidentially had a node being reinstalled with > a non-LTS > >image (13.04 I think) - and the kernel (3.5.something) had a > bug/'feature' which > >caused lots of tcp segments to be retransmittet (approx. 1/100). > > Do you have any information about this bug? We are using Ubuntu 13.04 for > all our Ceph nodes. If you can refer me to any documentation on this bug > and how to resolve this issue, I will appreciate it very much. > To be honest, I can't recall - the server in question was hosted @SoftLayer in Dallas and it was their techs asking us to upgrade the kernel after we found the issue with the high retransmit count and reported it. It was easy to just upgrade the kernel and test - and the issue went away. I didn't dig any deeper; if I remember, I'll try accessing the ticket monday to get all the details if it is still there. Cheers, Martin > > Looking forward to your reply, thank you. > > Cheers. > > > On Fri, Mar 7, 2014 at 6:10 PM, Martin B Nielsen <mar...@unity3d.com>wrote: > >> Hi, >> >> I'd probably start by looking at your nodes and check if the SSDs are >> saturated or if they have high write access times. If any of that is true, >> does that account for all SSD or just some of them? Maybe some of the disks >> needs a trim. Maybe test them individually directly on the cluster. >> >> If you can't find anything with the disks, then try and look further up >> the stack. Network, interrupts etc. At some point in time we accidentially >> had a node being reinstalled with a non-LTS image (13.04 I think) - and the >> kernel (3.5.something) had a bug/'feature' which caused lots of tcp >> segments to be retransmittet (approx. 1/100). This one node slowed down our >> entire cluster and caused high access time across the board. 'Upgrading' to >> LTS fixed it. >> >> As you say, it can just be that the increased utilization of the the >> cluster causes it and that you'll 'just' have to add more nodes. >> >> Cheers, >> Martin >> >> >> On Fri, Mar 7, 2014 at 10:50 AM, Indra Pramana <in...@sg.or.id> wrote: >> >>> Hi, >>> >>> I have a Ceph cluster, currently with 5 osd servers and around 22 OSDs >>> with SSD drives and I noted that the I/O speed, especially write access to >>> the cluster is degrading over time. When we first started the cluster, we >>> can get up to 250-300 MB/s write speed to the SSD cluster but now we can >>> only get up to half the mark. Furthermore, it now fluctuates so sometimes I >>> can get slightly better speed but on another time I get very bad result. >>> >>> We started with 3 osd servers and 12 OSDs and gradually add more >>> servers. We are using KVM hypervisors as the Ceph clients and connection >>> between clients and servers and between the servers are through 10 GBps >>> switch with jumbo frames enabled on all interfaces. >>> >>> Any advice on how can I start to troubleshoot what might have caused the >>> degradation of the I/O speed? Does utilisation contributes to it (since now >>> we have more users compared to last time when we started)? Any optimisation >>> we can do to improve the I/O performance? >>> >>> Appreciate any advice, thank you. >>> >>> Cheers. >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
