Hello Dan, A couple of quick things... * Latency is shown as a sum of all measured latencies over a period of time, and the count of operations included. So to calculate the average per op latency, you must divide the sum by the count. The result will be in milliseconds.
* The latency values you're showing there are from 'recoverystate_perf',
meaning they're not relevant to normal operations of the OSDs. For that,
I'd recommend doing a perf dump against the OSD admin socket and looking at
the latency values under the "osd" section.
* I've not seen any documentation on each counter, aside from occasional
mailing list posts about specific counters..
Hope this helps!
Michael J. Kidd
Sr. Storage Consultant
Inktank Professional Services
On Fri, Mar 7, 2014 at 11:39 AM, Dan Ryder (daryder) <dary...@cisco.com>wrote:
> Hello,
>
>
>
> I'm working with two different Ceph clusters, and in both clusters, I'm
> seeing very high latency values.
>
>
>
> Here's part of a sample perf dump:
>
>
>
> "recoverystate_perf": { "initial_latency": { "avgcount": 338,
>
> "sum": 0.069851000},
>
> "started_latency": { "avgcount": 1647,
>
> "sum": 322317122.940019000},
>
> "reset_latency": { "avgcount": 1985,
>
> "sum": 195.935076000},
>
> "start_latency": { "avgcount": 1985,
>
> "sum": 0.234355000},
>
> "primary_latency": { "avgcount": 266,
>
> "sum": 10819570.688122000},
>
>
>
> You can see both started latency and primary latency have extremely high
> values.
>
>
>
> Some info about the cluster:
>
> All nodes are on the same subnet - 2 VMs, 1 physical node
>
> VM1 is just a Monitor, VM2 is Monitor and OSD, Physical node is just an
> OSD.
>
>
>
>
>
> One additional question, are these latency values in milliseconds? Is
> there any documentation on the units for perf dump command?
>
> I've looked around but haven't seen anything.
>
>
>
> Thanks,
>
> Dan
>
>
>
>
>
> [image: http://www.cisco.com/web/europe/images/email/signature/logo05.jpg]
>
> *Dan Ryder*
> ENGINEER.SOFTWARE ENGINEERING
> CSMTG Performance/Analytics
> dary...@cisco.com
> Phone: *+1 919 392 7438 <%2B1%20919%20392%207438>*
>
> *Cisco Systems, Inc.*
> 7100-8 Kit Creek Road
> PO Box 14987
> 27709-4987
> Research Triangle Park
> United States
> Cisco.com <http://www.cisco.com/>
>
>
>
> [image: Think before you print.] Think before you print.
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/index.html
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
<<inline: image006.png>>
<<inline: image005.jpg>>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
