Public bug reported:
Description
===========
If the network fails during post live-migration tasks, the host/hypervisor
property of the instance isn't updated to reflect the *new* compute host.
The migrated instance appears to function properly, but issuing 'nova
show <instance_id>' still shows the origin compute node along with a
stack trace.
Per the trace below, a failure occurs inside
post_live_migration_at_destination() during a call to
network_api.migrate_instance_finish() which attempts to update the
neutron port mappings. This code exists outside the try/except/finally
block that will update the database regardless of failure. Should
network connectivity be lost, or the neutron api doesn't answer, the
instance properties aren't saved.
Steps to reproduce
==================
$ nova live-migration 0386d6cf-edf8-42b8-998f-54103ba675fe node1
$ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
+--------------------------------------+----------------------------------------------------------+
| Property | Value
|
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host | node1
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | node1.example.org
|
| OS-EXT-SRV-ATTR:instance_name | instance-00000034
|
| OS-EXT-STS:power_state | 1
|
| OS-EXT-STS:task_state | migrating
|
| OS-EXT-STS:vm_state | active
A Few minutes later, vm_state is in 'error' and a stack trace can be seen
$ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
...
...SNIP
........
fault {"message": "Unable to establish connection to
http://192.168.1.34:9696/v2.0/ports/dd042ee1-77c0-4c52-b5ea-9b5fbdd02860.json";,
"code": 500, "details": " File
\"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 375, in
decorated_function
return function(self, context, *args, **kwargs)
File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\",
line 5571, in post_live_migration_at_destination
migration)
File
\"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1577,
in migrate_instance_finish
migration['dest_compute'])
File
\"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1886,
in _update_port_binding_for_instance
p['id'], instance=instance)
File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\",
line 220, in __exit__
self.force_reraise()
File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\",
line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File
\"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1882,
in _update_port_binding_for_instance
{'port': {'binding:host_id': host}})
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 97, in
with_params
ret = self.function(instance, *args, **kwargs)
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 659, in
update_port
return self.put(self.port_path % (port), body=body)
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 367, in
put
headers=headers, params=params)
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 335, in
retry_request
headers=headers, params=params)
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 286, in
do_request
resp, replybody = self.httpclient.do_request(action, method,
body=body)
File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\",
line 306, in do_request
return self.request(url, method, **kwargs)
File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\",
line 294, in request
resp = super(SessionClient, self).request(*args, **kwargs)
File \"/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py\",
line 98, in request
return self.session.request(url, method, **kwargs)
File \"/usr/lib/python2.7/dist-packages/positional/__init__.py\",
line 94, in inner
return func(*args, **kwargs)
File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\",
line 452, in request
resp = send(**kwargs)
File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\",
line 496, in _send_request
raise exceptions.ConnectFailure(msg)
", "created": "2016-10-26T22:54:10Z"}
Expected result
===============
'host' and 'hypervisor_hostname' should be updated to the destination compute
node.
Environment
===========
Ubuntu 16.04
Openstack Mitaka
Nova 13.1.1
Neutron 8.1.2
+ Calico driver 1.3.0
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1638582
Title:
network failure during post live-migrate fails to update hypervisor
host
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
If the network fails during post live-migration tasks, the host/hypervisor
property of the instance isn't updated to reflect the *new* compute host.
The migrated instance appears to function properly, but issuing 'nova
show <instance_id>' still shows the origin compute node along with a
stack trace.
Per the trace below, a failure occurs inside
post_live_migration_at_destination() during a call to
network_api.migrate_instance_finish() which attempts to update the
neutron port mappings. This code exists outside the try/except/finally
block that will update the database regardless of failure. Should
network connectivity be lost, or the neutron api doesn't answer, the
instance properties aren't saved.
Steps to reproduce
==================
$ nova live-migration 0386d6cf-edf8-42b8-998f-54103ba675fe node1
$ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
+--------------------------------------+----------------------------------------------------------+
| Property | Value
|
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host | node1
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | node1.example.org
|
| OS-EXT-SRV-ATTR:instance_name | instance-00000034
|
| OS-EXT-STS:power_state | 1
|
| OS-EXT-STS:task_state | migrating
|
| OS-EXT-STS:vm_state | active
A Few minutes later, vm_state is in 'error' and a stack trace can be seen
$ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
...
...SNIP
........
fault {"message": "Unable to establish connection to
http://192.168.1.34:9696/v2.0/ports/dd042ee1-77c0-4c52-b5ea-9b5fbdd02860.json";,
"code": 500, "details": " File
\"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 375, in
decorated_function
return function(self, context, *args, **kwargs)
File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\",
line 5571, in post_live_migration_at_destination
migration)
File
\"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1577,
in migrate_instance_finish
migration['dest_compute'])
File
\"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1886,
in _update_port_binding_for_instance
p['id'], instance=instance)
File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\",
line 220, in __exit__
self.force_reraise()
File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\",
line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File
\"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1882,
in _update_port_binding_for_instance
{'port': {'binding:host_id': host}})
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 97, in
with_params
ret = self.function(instance, *args, **kwargs)
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 659, in
update_port
return self.put(self.port_path % (port), body=body)
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 367, in
put
headers=headers, params=params)
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 335, in
retry_request
headers=headers, params=params)
File
\"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 286, in
do_request
resp, replybody = self.httpclient.do_request(action, method,
body=body)
File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\",
line 306, in do_request
return self.request(url, method, **kwargs)
File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\",
line 294, in request
resp = super(SessionClient, self).request(*args, **kwargs)
File \"/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py\",
line 98, in request
return self.session.request(url, method, **kwargs)
File \"/usr/lib/python2.7/dist-packages/positional/__init__.py\",
line 94, in inner
return func(*args, **kwargs)
File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\",
line 452, in request
resp = send(**kwargs)
File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\",
line 496, in _send_request
raise exceptions.ConnectFailure(msg)
", "created": "2016-10-26T22:54:10Z"}
Expected result
===============
'host' and 'hypervisor_hostname' should be updated to the destination compute
node.
Environment
===========
Ubuntu 16.04
Openstack Mitaka
Nova 13.1.1
Neutron 8.1.2
+ Calico driver 1.3.0
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1638582/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
