Hi Edwin,
On 07/12/2013 08:03 AM, Edwin Peer wrote:
Hi there,
We've been noticing nasty multi-second cluster wide latencies if an OSD
drops out of an active cluster (due to power failure, or even being
stopped cleanly). We've also seen this problem occur when an OSD is
inserted back into the cluster.
You will probably see that Peering Groups (PGs) go into a different
state then active+clean.
What does ceph -s tell you in such a case?
Obviously, this has the effect of freezing all VMs doing I/O across the
cluster for several seconds when a single node fails. Is this behaviour
expected? Or have I perhaps got something configured wrong?
Not really the expected behavior, but it could be CPU power limitations
on the OSDs. I notice this latency with a Atom cluster as well, but
that's mainly due to the fact that the Atoms aren't fast enough to
figure out what's happening.
Faster AMD or Intel CPUs don't suffer from this. There will be a very
short I/O stall for certain PGs when an OSD goes down, but that should
be very short and not every VM should suffer.
How many OSDs do you have with how many PGs per pool?
Wido
We're trying very hard to eliminate all single points of failure in our
architecture, is there anything that can be done about this?
Regards,
Edwin Peer
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
