Public bug reported:
Release: 2023.1
Steps to reproduce:
- Precreate ports for ironic node
- Attempt to launch multiple instances at once
- You need some failure that prevents the nodes for provisioning (in my case
the firewall broke PXE for one controller)
- Observe instances getting stuck in BUILD state indefinitely.
Some relevant logs:
[instance: 88d96c7a-aca6-49e4-b083-a67ed687275f] Failure
prepping block device:
ironicclient.common.apiclient.exceptions.InternalServerError: Unable to
attach VIF 5c60151c-f9ee-42cb-b2a8-f94a6a2336c8 because Ironic can not
update Neutron port 5c60151c-f9ee-42cb-b2a8-f94a6a2336c8 MAC address to
match physical MAC address 0c:42:a1:02:42:8b (HTTP 500)
Failed to update MAC address on Neutron port
5c60151c-f9ee-42cb-b2a8-f94a6a2336c8.:
openstack.exceptions.ConflictException: ConflictException: 409: Client
Error for url:
https://172.25.4.254:9696/v2.0/ports/5c60151c-f9ee-42cb-b2a8-f94a6a2336c8,
Unable to complete operation for network 6c96c4e1-7495-41ee-
ba80-a12386819bc1. The mac address 0c:42:a1:02:42:8b is in use.
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron Traceback (most
recent call last): 2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/ironic/common/neutron.py", line 176, in update_port_address
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
update_neutron_port(context, port_id, port_attrs) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/ironic/common/neutron.py", line 109, in update_neutron_port
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
client.update_port(port_id, **attrs) 2024-10-16 17:44:44.716 7 ERROR
ironic.common.neutron File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/network/v2/_proxy.py", line 2324, in update_port
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
self._update(_port.Port, port, if_revision=if_revision, 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/proxy.py",
line 64, in check 2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
return method(self, expected, actual, *args, **kwargs) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/network/v2/_proxy.py", line 150, in _update
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
res.commit(self, base_path=base_path, if_revision=if_revision)
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1742, in commit 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron return self._commit(
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1787, in _commit 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron
self._translate_response(response, has_body=has_body) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1253, in _translate_response
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
exceptions.raise_from_response(response, error_message=error_message)
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/exceptions.py", line 240, in raise_from_response
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron raise cls(
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
openstack.exceptions.ConflictException: ConflictException: 409: Client
Error for url:
https://172.25.4.254:9696/v2.0/ports/5c60151c-f9ee-42cb-b2a8-f94a6a2336c8,
Unable to complete operation for network 6c96c4e1-7495-41ee-
ba80-a12386819bc1. The mac address 0c:42:a1:02:42:8b is in use.
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
Then the during teardown:
(openstack-venv) [stackhpc_admin@SVN3-DR08-U34 openstack-config]$ openstack
server event show 8c6baec9-fb74-4f68-aaa7-a203d8198251
req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value
|
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | create
|
| events | details='InvalidRequest',
event='compute__do_build_and_run_instance',
finish_time='2024-10-16T17:34:25.000000', host='svn1-dr07-u2-ironic',
|
| |
host_id='c04cb592b2146dc1ff0c6f0ca708218365527c67885df43ebec22320',
result='Error', start_time='2024-10-16T17:34:21.000000', traceback=' File
|
| |
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/utils.py", line
1439, in decorated_function
|
| | return function(self, context, *args, **kwargs)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py",
line 214, in decorated_function
|
| | compute_utils.add_instance_fault_from_exc(context,
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
|
| | self.force_reraise()
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
|
| | raise self.value
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py",
line 203, in decorated_function
|
| | return function(self, context, *args, **kwargs)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py",
line 2456, in _do_build_and_run_instance
|
| | self._cleanup_allocated_networks(context, instance,
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py",
line 2996, in _cleanup_allocated_networks
|
| | self.driver.unplug_vifs(instance, network_info)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/virt/ironic/driver.py",
line 1629, in unplug_vifs
|
| | node = self._get_node(instance.node)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/virt/ironic/driver.py",
line 221, in _get_node
|
| | node = self.ironic_connection.get_node(node_id,
fields=_NODE_FIELDS)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/baremetal/v1/_proxy.py",
line 328, in get_node
|
| | return self._get_with_fields(_node.Node, node,
fields=fields)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/baremetal/v1/_proxy.py",
line 63, in _get_with_fields
|
| | return res.fetch(
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line
1633, in fetch
|
| | request = self._prepare_request(
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line
1229, in _prepare_request
|
| | raise exceptions.InvalidRequest(
|
| | '
|
| id | req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90
|
| message | Error
|
| project_id | 931d1604d56c45698f602ffcee907e3c
|
| request_id | req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90
|
| start_time | 2024-10-16T17:34:20.000000
|
| user_id | 8072931a57a94c0f96e45a544eb0393c
|
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Sorry for the bad formatting.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2085432
Title:
Ironic builds stuck after "The mac address is in use conflict"
Status in OpenStack Compute (nova):
New
Bug description:
Release: 2023.1
Steps to reproduce:
- Precreate ports for ironic node
- Attempt to launch multiple instances at once
- You need some failure that prevents the nodes for provisioning (in my case
the firewall broke PXE for one controller)
- Observe instances getting stuck in BUILD state indefinitely.
Some relevant logs:
[instance: 88d96c7a-aca6-49e4-b083-a67ed687275f] Failure
prepping block device:
ironicclient.common.apiclient.exceptions.InternalServerError: Unable
to attach VIF 5c60151c-f9ee-42cb-b2a8-f94a6a2336c8 because Ironic can
not update Neutron port 5c60151c-f9ee-42cb-b2a8-f94a6a2336c8 MAC
address to match physical MAC address 0c:42:a1:02:42:8b (HTTP 500)
Failed to update MAC address on Neutron port
5c60151c-f9ee-42cb-b2a8-f94a6a2336c8.:
openstack.exceptions.ConflictException: ConflictException: 409: Client
Error for url:
https://172.25.4.254:9696/v2.0/ports/5c60151c-f9ee-42cb-b2a8-f94a6a2336c8,
Unable to complete operation for network 6c96c4e1-7495-41ee-
ba80-a12386819bc1. The mac address 0c:42:a1:02:42:8b is in use.
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron Traceback (most
recent call last): 2024-10-16 17:44:44.716 7 ERROR
ironic.common.neutron File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/ironic/common/neutron.py", line 176, in update_port_address
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
update_neutron_port(context, port_id, port_attrs) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/ironic/common/neutron.py", line 109, in update_neutron_port
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
client.update_port(port_id, **attrs) 2024-10-16 17:44:44.716 7 ERROR
ironic.common.neutron File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/network/v2/_proxy.py", line 2324, in update_port
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
self._update(_port.Port, port, if_revision=if_revision, 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/proxy.py", line 64, in check 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron return method(self,
expected, actual, *args, **kwargs) 2024-10-16 17:44:44.716 7 ERROR
ironic.common.neutron File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/network/v2/_proxy.py", line 150, in _update
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
res.commit(self, base_path=base_path, if_revision=if_revision)
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1742, in commit 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron return self._commit(
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1787, in _commit 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron
self._translate_response(response, has_body=has_body) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1253, in _translate_response
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
exceptions.raise_from_response(response, error_message=error_message)
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/exceptions.py", line 240, in raise_from_response
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron raise cls(
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
openstack.exceptions.ConflictException: ConflictException: 409: Client
Error for url:
https://172.25.4.254:9696/v2.0/ports/5c60151c-f9ee-42cb-b2a8-f94a6a2336c8,
Unable to complete operation for network 6c96c4e1-7495-41ee-
ba80-a12386819bc1. The mac address 0c:42:a1:02:42:8b is in use.
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
Then the during teardown:
(openstack-venv) [stackhpc_admin@SVN3-DR08-U34 openstack-config]$ openstack
server event show 8c6baec9-fb74-4f68-aaa7-a203d8198251
req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value
|
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | create
|
| events | details='InvalidRequest',
event='compute__do_build_and_run_instance',
finish_time='2024-10-16T17:34:25.000000', host='svn1-dr07-u2-ironic',
|
| |
host_id='c04cb592b2146dc1ff0c6f0ca708218365527c67885df43ebec22320',
result='Error', start_time='2024-10-16T17:34:21.000000', traceback=' File
|
| |
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/utils.py", line
1439, in decorated_function
|
| | return function(self, context, *args, **kwargs)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py",
line 214, in decorated_function
|
| | compute_utils.add_instance_fault_from_exc(context,
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
|
| | self.force_reraise()
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
|
| | raise self.value
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py",
line 203, in decorated_function
|
| | return function(self, context, *args, **kwargs)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py",
line 2456, in _do_build_and_run_instance
|
| | self._cleanup_allocated_networks(context, instance,
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py",
line 2996, in _cleanup_allocated_networks
|
| | self.driver.unplug_vifs(instance, network_info)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/virt/ironic/driver.py",
line 1629, in unplug_vifs
|
| | node = self._get_node(instance.node)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/virt/ironic/driver.py",
line 221, in _get_node
|
| | node = self.ironic_connection.get_node(node_id,
fields=_NODE_FIELDS)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/baremetal/v1/_proxy.py",
line 328, in get_node
|
| | return self._get_with_fields(_node.Node, node,
fields=fields)
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/baremetal/v1/_proxy.py",
line 63, in _get_with_fields
|
| | return res.fetch(
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line
1633, in fetch
|
| | request = self._prepare_request(
|
| | File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line
1229, in _prepare_request
|
| | raise exceptions.InvalidRequest(
|
| | '
|
| id | req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90
|
| message | Error
|
| project_id | 931d1604d56c45698f602ffcee907e3c
|
| request_id | req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90
|
| start_time | 2024-10-16T17:34:20.000000
|
| user_id | 8072931a57a94c0f96e45a544eb0393c
|
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Sorry for the bad formatting.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2085432/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
