We had this issue in Havana and applying the patch from that gist fixed the issue for us. We just noticed last week that it has returned in our upgraded Trusty/Icehouse cloud since we neglected to repatch the new eventlet package, and patching *that* fixed it again for us.
I reported an issue against eventlet for this a while ago, but looks like it has since been closed: https://github.com/eventlet/eventlet/issues/63 Cheers, Kieran On 14 October 2014 02:17, Joe Topjian <j...@topjian.net> wrote: > That's really interesting - thanks for the link. > > We've been able to narrow down why didn't see this with Cinder or Swift: in > short, Swift isn't using Oslo (AFAIK) and we had some previous logging > issues with Cinder in Havana so we altered the logging setup a bit. > > > On Mon, Oct 13, 2014 at 3:05 AM, Francois Deppierraz > <franc...@ctrlaltdel.ch> wrote: >> >> Hi Joe, >> >> Yes, same problem here running Ubuntu 14.04. >> >> The symptom is nova-api, nova-conductor and glance-api eating all CPU >> without responding to API requests anymore. >> >> It is possible to reproduce it thanks to the following script. >> >> https://gist.github.com/dbishop/7a2e224f3aafea1a1fc3 >> >> François >> >> On 11. 10. 14 00:40, Joe Topjian wrote: >> > Hello, >> > >> > This morning we noticed various nova, glance, and keystone services (not >> > cinder or swift) not working in two different clouds and required a >> > restart. >> > >> > We thought it was a network issue since one of the only commonalities >> > between the two clouds was that they are on the same network. >> > >> > Then later in the day I logged into a test cloud on a totally separate >> > network and had the same problem. >> > >> > Looking at all three environments, the commonality is now that they have >> > Ubuntu security updates automatically applied in the morning and this >> > morning rsyslog was patched and restarted. >> > >> > I found this oslo bug that kind of sounds like the issue we saw: >> > >> > https://bugs.launchpad.net/oslo.log/+bug/1076466 >> > >> > Doing further investigation, log files do indeed show a lack of entries >> > for various services/daemons until they were restarted. >> > >> > Has anyone else run into this? Maybe even this morning, too? :) >> > >> > Thanks, >> > Joe >> > >> > >> > >> > >> > _______________________________________________ >> > OpenStack-operators mailing list >> > OpenStack-operators@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> > >> >> >> _______________________________________________ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
