Does anyone have any insight into this? For further datapoints, I built
the 2.0 release and much more current openvswitch snapshots (most
recently to commit bdeadfdd) which exhibited the same problems. The
CentOS 6 kernel is 2.6.32. Because of presumed incompatibility with the
Linux bridge module, I made sure bridge.o wasn't being loaded. On a host
where ovsdb-server had not yet become unresponsive, ovs-vswitchd was
unkillable, in state R<L. Could my problem be related to vwsitchd
becoming unresponsive under load, taking ovsdb-server with it?
I've received further confirmation that this is involved in some way
with load, as a node inadvertently disconnected from the rest of the
Ceph cluster had a record uptime with openvswitch. If anyone can give me
pointers on getting a backtrace I'm happy to run things until failure
and get better data. I've had trouble with this at least as far as using
strace is concerned. As it is, I've cron'd a restart of openvswitch
every minute - obviously an incredibly unideal situation.
Thanks for any help,
Jeff
On 02/20/2014 12:54 AM, Jeff Bachtel wrote:
I'm running OpenVSwitch 1.11 from the RDO Havana repository. In
addition, I'm running OpenStack Havana, Neutron, and Ceph Emperor, all
on some CentOS 6.5 machines.
After installing Bacula on the previous openstack version (grizzly), I
noticed the networking had become somewhat load sensitive.
ovsdb-server was freezing - not responding to queries on its unix
socket and becoming unkillable in process state R< . Believing that it
was probably due to being behind in ovs version, I pushed ahead with
an upgrade only to find my stability problems become much much worse.
Every 20-30 minutes I can count on an ovsdb-server process freezing.
At
https://drive.google.com/folderview?id=0B-wx2_T_hW-_OXZJWGJNc0l0MzQ&usp=sharing
please find a folder with shared copies of diagnostic files from a
machine with hung ovsdb-server. There is a process list (.ps,
apologies forgot postscript until upload was done), strace, dmesg, and
/var/log/messages.
The strace didn't reveal anything suspicious to me. To mitigate I
tried lowering log verbosity, completely recreating conf.db, as well
as frequent compacting (every minute) and putting the db on a ramdisk,
nothing worked as a solution.
The ovsdb-server processes most likely to succumb to locking run on
ceph hosts running osd - meaning they can see a lot of network
traffic, as well as disk i/o.
I don't understand what a simple database RPC server could be doing
that would cause it to become unkillable, especially with the attempt
at minimizing disk i/o by putting the db file on a ramdisk.
I hope someone has some ideas of what I might do to test or mitigate
the situation. Not running ceph osd on the hosts is, unfortunately,
not a solution I can use.
Thanks,
Jeff
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
