>How responsive generally is the machine under load? Is there available CPU?
The machine works well, and the issued OSDs are likely the same, seems
because they have relative slower disk( disk type are the same but the latency
is a bit higher ,8ms -> 10ms).
Top show no idle % but still have 30+% of io_wait, my colleague
educate me that io_wait can be treated as free.
Another information is offload the heartbeat to 1Gb nic doesn't solve the
problem, what's more, when we doing random write test, we can still see this
flipping happen. So I would like to say it may related with CPU scheduler ? The
heartbeat thread (in busy OSD ) failed to get enough cpu cycle.
-----Original Message-----
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Sage Weil
Sent: 2013年5月15日 7:23
To: Chen, Xiaoxi
Cc: Mark Nelson; ceph-de...@vger.kernel.org; ceph-us...@ceph.com
Subject: Re: [ceph-users] OSD state flipping when cluster-network in high
utilization
On Tue, 14 May 2013, Chen, Xiaoxi wrote:
> I like the idea to leave ping in cluster network because it can help
> us detect switch?nic failure.
>
> What confuse me is I keep pinging every ceph node's cluster ip?it is
> OK during the whole run with less than 1 ms latency?why the heartbeat
> still suffer? TOP show my cpu not 100% utilized?with ?30% io
> wait?.Enabling jumbo frame **seems** make things worth.(just
> feeling.no data supports)
I say "ping" in teh general sense.. it's not using ICMP, but sending small
messages over a TCP session, doing some minimal processing on the other end,
and sending them back. If the machine is heavily loaded and that thread
doesn't get scheduled or somehow blocks, it may be problematic.
How responsive generally is the machine under load? Is there available CPU?
We can try to enable debugging to see what is going on.. 'debug ms = 1'
and 'debug osd = 20' is everything we would need, but will incur additoinal
load itself and may spoil the experiment...
sage
>
> ???? iPhone
>
> ? 2013-5-14?23:36?"Mark Nelson" <mark.nel...@inktank.com> ???
>
> > On 05/14/2013 10:30 AM, Sage Weil wrote:
> >> On Tue, 14 May 2013, Chen, Xiaoxi wrote:
> >>>
> >>> Hi
> >>>
> >>> We are suffering our OSD flipping between up and down ( OSD X be
> >>> voted to down due to 3 missing ping, and after a while it tells
> >>> the monitor ?map xxx wrongly mark me down? ). Because we are
> >>> running sequential write performance test on top of RBDs, and the
> >>> cluster network nics is really in high utilization (8Gb/s+ for a 10Gb
> >>> network).
> >>>
> >>> Is this a expected behavior ? or how can I prevent this happen?
> >>
> >> You an increase the heartbeat grace period. The pings are handled
> >> by a separate thread on the backside interface (if there is one).
> >> If you are missing pings then the network or scheduler is
> >> preventing those (small) messages from being processed (there is
> >> almost no lock contention in that path). Which means it really is
> >> taking ~20 seconds or wahtever to handle those messages. It's
> >> really a questin of how unresponsive you want to permit the OSDs to be
> >> before you consider it a failure..
> >>
> >> sage
> >>
> >>
> >
> > It might be worth testing out how long pings or other network traffic are
> > taking during these tests. There may be some tcp tunning you can do here,
> > or even consider using a separate network for the mons.
> >
> > Mark
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
body of a message to majord...@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
