Tim Andersson has proposed merging
~andersson123/autopkgtest-cloud:worker-nova-client-robustness into
autopkgtest-cloud:master.
Requested reviews:
Canonical's Ubuntu QA (canonical-ubuntu-qa)
For more details, see:
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/477574
Don't exit the worker script when the hypervisor is feeling poorly and not
responding to novaclient requests!!!
--
Your team Canonical's Ubuntu QA is requested to review the proposed merge of
~andersson123/autopkgtest-cloud:worker-nova-client-robustness into
autopkgtest-cloud:master.
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
index e5ab016..a5eb744 100755
--- a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
@@ -34,6 +34,7 @@ import systemd.journal
from influxdb import InfluxDBClient
from influxdb.exceptions import InfluxDBClientError
from keystoneauth1 import session
+from keystoneauth1.exceptions.http import InternalServerError
from keystoneauth1.identity import v2, v3
from osc_lib import utils as osc_utils
@@ -674,7 +675,21 @@ def kill_openstack_server(test_uuid: str):
session=sess,
region_name=os.environ["OS_REGION_NAME"],
)
- for instance in nova.servers.list():
+ nova_servers = None
+ nova_retry_seconds = 10
+ nova_retry_attempts = 10
+ while not nova_servers:
+ for attempt in range(nova_retry_attempts):
+ try:
+ nova_servers = nova.servers.list()
+ break
+ except InternalServerError:
+ logging.info(
+ f"Failed to list nova servers, retrying in {nova_retry_seconds} seconds, attempt {attempt+1}/{nova_retry_attempts+1}"
+ )
+ else:
+ return
+ for instance in nova_servers:
if test_uuid in instance.name:
try:
instance.delete()
--
Mailing list: https://launchpad.net/~canonical-ubuntu-qa
Post to : canonical-ubuntu-qa@lists.launchpad.net
Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa
More help : https://help.launchpad.net/ListHelp
