** Also affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1398128
Title:
ironic tempest tests periodically failing: No valid host was found
Status in devstack - openstack dev environments:
Fix Released
Status in OpenStack Bare Metal Provisioning Service (Ironic):
New
Status in OpenStack Compute (Nova):
New
Bug description:
This was noticed on the stable/juno ironic sideways grenade jobs, but
is also confirmed to be happening on the check-tempest-dsvm-ironic-
parallel-nv job, which runs a similarly configured tempest run against
Ironic:
http://logs.openstack.org/84/137684/1/check/check-grenade-dsvm-ironic-
sideways/6d118bc/
A number of the early compute tests will fail to spawn an instance,
getting a scheduling error on the client side:
BuildErrorException: Server %(server_id)s failed to build and is in ERROR
status
Details: Server eb81ee40-ceba-484d-b665-92ec3bf4fedd failed to build and is
in ERROR status
Details: {u'message': u'No valid host was found. ', u'created':
u'2014-11-27T17:44:05Z', u'code': 500}
Looking through the nova logs, the request never even makes to the
nova-scheduler. The last error is reported in conductor:
2014-11-27 17:44:01.005 WARNING nova.scheduler.driver [req-a3c046e5
-66db-4bca-a6f8-2263763e49a6 SecurityGroupsTestJSON-2119055496
SecurityGroupsTestJSON-1381566740] [instance: 9008811a-f400-42ae-
98d5-caf828fa34dc] NoValidHost exception with message: 'No valid host
was found.'
Looking at the time stamps of the requests, the first instance is
requested at 17:44:00
2014-11-27 17:44:00.944 24730 DEBUG tempest.common.rest_client [req-
a3c046e5-66db-4bca-a6f8-2263763e49a6 None] Request
(SecurityGroupsTestJSON:test_server_security_groups): 202 POST
http://127.0.0.1:8774/v2/adf4838f0d15462da4601a5d853eafbf/servers
0.515s
However, on the nova-compute side, the resource tracker has not been
updated to include the enlisted Ironic nodes until much later. This
first time the tracker contains any of the ironic resources is at
17:44:06:
2014-11-27 17:44:06.224 21645 AUDIT nova.compute.resource_tracker [-]
Total physical ram (MB): 512, total allocated virtual ram (MB): 0
So there's a race between the resource tracker's initial inclusion of
available resources and Tempest running the first set of tests that
require an instance. This can be worked around in a couple of ways:
* Adjust the periodic task interval on nova-compute to update much more
frequently, tho this will just narrow the window.
* Have tempest run an admin 'nova hypervisor-stats' call on the client side
and wait for resources before running any instances (in the case of baremetal
only)
* Adjust devstack's nova cpu deployment to spin until hypervisor-stats
reflect the ironic node parameters
To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1398128/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
