Basically the issue is seen in the following three lines of nova compute log. For that port even though it received the vif plugging event 2 mins before it waits for it and blocks and times out Is there a race condition in the code that basically gets the events to wait for and the one where it registers for this callback Any comments?
2016-10-09 07:35:24.869 6 DEBUG nova.compute.manager [req-afbc9c31-757a-478e-af8f-a4937e787a57 53a2bd71435c43e3953c37ffb9b38e21 56b04dfbb802430face997791412a62a - - -] [instance: 59b4f72b-5d1d-41ee-bc57-c57a8204383a] Received event network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 external_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:6509 2016-10-09 07:35:24.871 6 DEBUG nova.compute.manager [req-afbc9c31-757a-478e-af8f-a4937e787a57 53a2bd71435c43e3953c37ffb9b38e21 56b04dfbb802430face997791412a62a - - -] [instance: 59b4f72b-5d1d-41ee-bc57-c57a8204383a] No waiting events found dispatching network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 pop_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:545 2016-10-09 07:37:36.668 6 DEBUG nova.compute.manager [req-31f88ed2-ab3d-4b1d-a75b-b4e7b80b9202 28450924ed064558948f98eaec39fe6e 54e6da5850104d3682ef3500f3eb7a57 - - -] [instance: 59b4f72b-5d1d-41ee-bc57-c57a8204383a] Preparing to wait for external event network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 prepare_for_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:507 From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Tuesday, October 4, 2016 at 10:56 AM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Subject: [openstack-dev] [nova] VIF plugin issue _get_neutron_events Hi When booting Vms with 3 interfaces and when port binding happens quick enough I sometimes see that the first VM booted on a compute node gets stuck waiting for VIF plugging event but looking at the neutron logs the notification is sent and also nova responded with a 200 OK. When looking at the code in nova/virt/libvirt/driver.py we see in spawn() there is a call to _create_domain_and_network which calls _get_neutron_events So in what scenarios can this notification happen to soon or is there a race condition somewhere in this code wherby even though neutron sends the notification and nova responds the code here blocks for 300 seconds since it missed the event. Any clues on what maybe going on here. Its almost like the neutron notification happened too quickly while nova code was busy performing create_image This happens only with the first VM on a compute node Any ideas on how to fix this Ajay
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
