Reviewed: https://review.opendev.org/668243 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d540903463aa9b0cf69cefac7cc60e5b70e40a1c Submitter: Zuul Branch: master
commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c Author: Matt Riedemann <[email protected]> Date: Fri Jun 28 16:19:49 2019 -0400 Init HostState.failed_builds If _update_from_compute_node returns early and the HostState is not filtered out we can hit an AttributeError in the BuildFailureWeigher because the failed_builds attribute is not set. This simply initializes the attribute like the other stats fields. Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c Closes-Bug: #1834691 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1834691 Title: AttributeError: 'HostState' object has no attribute 'failed_builds' Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Compute (nova) queens series: Triaged Status in OpenStack Compute (nova) rocky series: Triaged Status in OpenStack Compute (nova) stein series: Triaged Bug description: I'm not sure how this happens, but I hit this in a devstack environment with 35 nova-compute services using the fake virt driver, one scheduler worker and then trying to schedule a server: Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts = self._get_sorted_hosts(spec_obj, hosts, num) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server spec_obj) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts, spec_obj) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weights = weigher.weigh_objects(weighed_objs, weighing_properties) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weight = self._weigh_object(obj.obj, weight_properties) Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return host_state.failed_builds Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds' Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server The HostState.failed_builds attribute gets set outside of __init__ in _update_from_compute_node here: https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261 And I'd expect the _update_from_compute_node to get called from update() which is called from here: https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787 Ah bingo if the compute node hasn't had resource usage updated yet we don't set the failed_builds attribute: https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1834691/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
