[Canonical-ubuntu-qa] [Merge] ~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into autopkgtest-cloud:master
Tim Andersson has proposed merging
~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into
autopkgtest-cloud:master.
Requested reviews:
Canonical's Ubuntu QA (canonical-ubuntu-qa)
For more details, see:
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/465238
--
Your team Canonical's Ubuntu QA is requested to review the proposed merge of
~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into
autopkgtest-cloud:master.
diff --git a/charms/focal/autopkgtest-web/webcontrol/cache-amqp b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
index adfd4df..336891e 100755
--- a/charms/focal/autopkgtest-web/webcontrol/cache-amqp
+++ b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
@@ -4,6 +4,7 @@ import argparse
import json
import logging
import os
+import re
import sqlite3
import sys
import tempfile
@@ -123,11 +124,8 @@ class AutopkgtestQueueContents:
if isinstance(r, bytes):
r = r.decode("UTF-8")
try:
-req = r.split("\n", 1)
-if len(req) > 1:
-params = json.loads(req[1])
-else:
-params = {}
+req = re.search(r"{(.*)}", r).group(1)
+params = json.loads("{%s}" % req)
if params.get("readable-by", False) or params.get(
"swiftuser", False
):
--
Mailing list: https://launchpad.net/~canonical-ubuntu-qa
Post to : canonical-ubuntu-qa@lists.launchpad.net
Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa
More help : https://help.launchpad.net/ListHelp
[Canonical-ubuntu-qa] [Bug 2062537] Re: Change the way upstream jobs percentage is done
I've made some measurements on the last 12 hours of running tests, the proportion of upstream tests seems appropriate, so I'll mark this as "Fix released" ** Changed in: auto-package-testing Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Canonical's Ubuntu QA, which is subscribed to Auto Package Testing. https://bugs.launchpad.net/bugs/2062537 Title: Change the way upstream jobs percentage is done Status in Auto Package Testing: Fix Released Bug description: This code[1] is mostly broken: it's only when connecting to the queues that we decide if a worker is going to take `upstream` jobs, but this is based on randomness. I see the following flaws: * with low number of workers (20 is low), there is a high chance that the distribution will not be uniform at all, meaning you can end up with far more or far less workers taking upstream jobs than intended. * the dice is rolled only when the worker starts, and that means that is you end up with too many/few workers taking upstream jobs, the situation is going to stay until the workers are restarted. Here is a quick Python one-liner to play with. It gives the number of workers that will pick up upstream jobs, over a total of 20, with a chance of 50%: `len([r for r in [random.randint(1, 100) for _ in range(20)] if r > 50])` For only 20 workers and 50% chances, it's common to see a run with a deviation of more than 3 points from the objective (objective: 10, we often see <=7 or >=13). This gets worse when setting a lower threshold (eg 15), because we more often hit the extreme where no worker at all would pick upstream jobs. A better solution would be to roll the dice in the `request` function callback [2] so that for every upstream test request, the chance of processing get calculated. This will make the number of rolls proportionate with the number of jobs processed, and that means far greater numbers, which means far more accurate percentage of jobs filtered by the threshold over time. [1]: https://git.launchpad.net/autopkgtest-cloud/tree/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker#n1482 [2]: around here: https://git.launchpad.net/autopkgtest-cloud/tree/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker#n640 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/2062537/+subscriptions -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into autopkgtest-cloud:master
lgtm, one online question.
Diff comments:
> diff --git a/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> index adfd4df..72e8d2d 100755
> --- a/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> +++ b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> @@ -126,12 +126,12 @@ class AutopkgtestQueueContents:
> req = r.split("\n", 1)
> if len(req) > 1:
> params = json.loads(req[1])
> +if params.get("readable-by", False) or params.get(
> +"swiftuser", False
> +):
> +r = "private job"
> else:
> -params = {}
> -if params.get("readable-by", False) or params.get(
> -"swiftuser", False
> -):
> -r = "private job"
> +r = "malformed request"
should we also log the malformed request via logging.warning()?
> res.append(r)
> except (ValueError, IndexError) as e:
> logging.warning(
--
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/465238
Your team Canonical's Ubuntu QA is requested to review the proposed merge of
~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into
autopkgtest-cloud:master.
--
Mailing list: https://launchpad.net/~canonical-ubuntu-qa
Post to : canonical-ubuntu-qa@lists.launchpad.net
Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa
More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into autopkgtest-cloud:master
woops, sorry, I forgot that, my bad, amending now -- https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/465238 Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into autopkgtest-cloud:master. -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into autopkgtest-cloud:master
LGTM with an inline suggestion.
Diff comments:
> diff --git a/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> index adfd4df..22b6623 100755
> --- a/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> +++ b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> @@ -126,12 +126,15 @@ class AutopkgtestQueueContents:
> req = r.split("\n", 1)
> if len(req) > 1:
> params = json.loads(req[1])
> +if params.get("readable-by", False) or params.get(
> +"swiftuser", False
> +):
> +r = "private job"
> else:
> -params = {}
> -if params.get("readable-by", False) or params.get(
> -"swiftuser", False
> -):
> -r = "private job"
> +logging.warning(
> +"Found malformed request: %s\nMarking it as such." %
> r
I am not a fan of newline in warning/error messages, as by parsing the logs it
may be difficult to tell where a string comes from. I'd just leave out the
"\nMarking it as such.", the message is clear enough.
> +)
> +r = "malformed request"
> res.append(r)
> except (ValueError, IndexError) as e:
> logging.warning(
--
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/465238
Your team Canonical's Ubuntu QA is requested to review the proposed merge of
~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into
autopkgtest-cloud:master.
--
Mailing list: https://launchpad.net/~canonical-ubuntu-qa
Post to : canonical-ubuntu-qa@lists.launchpad.net
Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa
More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into autopkgtest-cloud:master
Diff comments:
> diff --git a/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> index adfd4df..22b6623 100755
> --- a/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> +++ b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
> @@ -126,12 +126,15 @@ class AutopkgtestQueueContents:
> req = r.split("\n", 1)
> if len(req) > 1:
> params = json.loads(req[1])
> +if params.get("readable-by", False) or params.get(
> +"swiftuser", False
> +):
> +r = "private job"
> else:
> -params = {}
> -if params.get("readable-by", False) or params.get(
> -"swiftuser", False
> -):
> -r = "private job"
> +logging.warning(
> +"Found malformed request: %s\nMarking it as such." %
> r
sorry about the delay, I didn't get an email for this for some reason. Amending
now.
> +)
> +r = "malformed request"
> res.append(r)
> except (ValueError, IndexError) as e:
> logging.warning(
--
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/465238
Your team Canonical's Ubuntu QA is requested to review the proposed merge of
~andersson123/autopkgtest-cloud:fix-cache-amqp-creds into
autopkgtest-cloud:master.
--
Mailing list: https://launchpad.net/~canonical-ubuntu-qa
Post to : canonical-ubuntu-qa@lists.launchpad.net
Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa
More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ~andersson123/autopkgtest-cloud:backup-worker-logs into autopkgtest-cloud:master
Obsolete. Closing. -- https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/456538 Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~andersson123/autopkgtest-cloud:backup-worker-logs into autopkgtest-cloud:master. -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ~andersson123/autopkgtest-cloud:upgrade-charm-to-docs into autopkgtest-cloud:master
ping on this. The upgrade-charm script adheres to version pinning and channel pinning as in the service-bundle. Please re-review -- https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/463293 Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~andersson123/autopkgtest-cloud:upgrade-charm-to-docs into autopkgtest-cloud:master. -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ~andersson123/autopkgtest-cloud:filter-amqp-all-queues-option into autopkgtest-cloud:master
rebased off of master and amended. -- https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/461624 Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~andersson123/autopkgtest-cloud:filter-amqp-all-queues-option into autopkgtest-cloud:master. -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ubuntu-manual-tests:calamares_installer-prompt_firmware_types into ubuntu-manual-tests:main
Hey! I have already added these to the tracker, but the priorities are probably wrong, given your list above, I'll amend the calamares test suites accordingly -- https://code.launchpad.net/~ubuntu-testcase/ubuntu-manual-tests/+git/ubuntu-manual-tests/+merge/465118 Your team Ubuntu Testcase Admins is subscribed to branch ubuntu-manual-tests:main. -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
Re: [Canonical-ubuntu-qa] [Merge] ubuntu-manual-tests:calamares_installer-prompt_firmware_types into ubuntu-manual-tests:main
Please can you take a look at: https://iso.qa.ubuntu.com/qatracker/milestones/450/builds/300328/testcases and https://iso.qa.ubuntu.com/qatracker/milestones/450/builds/300330/testcases And confirm with me that this suffices for you? I can amend as you please -- https://code.launchpad.net/~ubuntu-testcase/ubuntu-manual-tests/+git/ubuntu-manual-tests/+merge/465118 Your team Ubuntu Testcase Admins is subscribed to branch ubuntu-manual-tests:main. -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
[Canonical-ubuntu-qa] [Merge] ~hyask/autopkgtest-cloud:skia/amqp_semaphores into autopkgtest-cloud:master
Skia has proposed merging ~hyask/autopkgtest-cloud:skia/amqp_semaphores into
autopkgtest-cloud:master.
Requested reviews:
Canonical's Ubuntu QA (canonical-ubuntu-qa)
For more details, see:
https://code.launchpad.net/~hyask/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/465282
Fix locking mechanism of `cache-amqp`, plus add a `--refresh-semaphores` flag
to help repair the system when it breaks.
--
Your team Canonical's Ubuntu QA is requested to review the proposed merge of
~hyask/autopkgtest-cloud:skia/amqp_semaphores into autopkgtest-cloud:master.
diff --git a/charms/focal/autopkgtest-web/webcontrol/cache-amqp b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
index aaa252d..9461ea2 100755
--- a/charms/focal/autopkgtest-web/webcontrol/cache-amqp
+++ b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
@@ -19,7 +19,7 @@ AMQP_CONTEXTS = ["ubuntu", "huge", "ppa", "upstream"]
class AutopkgtestQueueContents:
-def __init__(self, amqp_uri, database):
+def __init__(self, amqp_uri, database, refresh_semaphores=False):
assert amqp_uri is not None
assert database is not None
@@ -39,6 +39,14 @@ class AutopkgtestQueueContents:
for release, arches in self.release_arches.items():
for arch in arches:
queue_name = f"semaphore-{context}-{release}-{arch}"
+if (
+os.path.exists("/run/autopkgtest-web-is-leader")
+and refresh_semaphores
+):
+self.amqp_channel.queue_delete(queue_name)
+logger.info(
+f"Semaphore queue '{queue_name}' deleted for recreation"
+)
try:
self.amqp_channel.queue_declare(
queue_name, durable=True, passive=True
@@ -70,14 +78,6 @@ class AutopkgtestQueueContents:
"We are not the leader, and there is no semaphore queue yet, we can't do anything - exiting."
)
sys.exit(0)
-# if the queue is empty basic_get will return None
-if os.path.exists(
-"/run/autopkgtest-web-is-leader"
-) and not self.amqp_channel.basic_get(queue_name):
-self.amqp_channel.basic_publish(
-amqp.Message(f"{queue_name}", delivery_mode=2),
-routing_key=queue_name,
-)
@property
def release_arches(self):
@@ -230,6 +230,13 @@ if __name__ == "__main__":
help="Print debugging (give twice for super verbose output)",
)
parser.add_argument(
+"--refresh-semaphores",
+dest="refresh_semaphores",
+action="store_true",
+help="Force the recreation of the semaphore queues if something broke them "
+"(make sure to stop every running cache-amqp script before use)",
+)
+parser.add_argument(
"-o",
"--output",
dest="output",
@@ -282,7 +289,9 @@ if __name__ == "__main__":
# make the queue size go crazy in the KPI
if os.path.isfile("/run/autopkgtest-web-is-leader"):
# Get queue details from rabbitmq directly
-aq = AutopkgtestQueueContents(amqp_uri, database)
+aq = AutopkgtestQueueContents(
+amqp_uri, database, args.refresh_semaphores
+)
queue_contents = aq.get_queue_contents()
else:
# We get queues.json from autopkgtest.ubuntu.com, see if it's
diff --git a/docs/administration.rst b/docs/administration.rst
index 49739da..4673fd1 100644
--- a/docs/administration.rst
+++ b/docs/administration.rst
@@ -148,6 +148,22 @@ indicates something to be looked into.
``armhf`` cluster nodes in error almost always need checking out, as they
usually indicate that the LXD host has gone down and needs redeploying.
+If the queues are non empty but flat
+
+
+This may indicate that the infra is somehow unable to process jobs, but
+sometimes this is just related to ``cache-amqp`` being stuck somehow.
+This script runs on the webunits, and does its job on the leader of those
+units. It has a semaphore mechanism, so should be able to work in a fully
+distributed system. However, this hasn't been maintained much, and sometime
+this semaphores can break, either by having more than one message in the
+``semaphore---`` queue, or by having none. You can fix
+that by stopping all the ``cache-amqp`` services (on all units!), and manually
+running ``cache-amqp --refresh-semaphores --debug`` on the leader, which will
+nuke the semaphore queues and recreate them. The ``--debug`` will help you
+figure out if something goes wrong.
+
+
Opening up a new series
---
--
Mailing list: https://launchpad.net/~canonical-ubuntu-qa
Post to
Re: [Canonical-ubuntu-qa] [Merge] ubuntu-manual-tests:calamares_installer-prompt_firmware_types into ubuntu-manual-tests:main
Oh there's no priority intended by that list, so it all looks good to me. Although I think Ubuntu Unity needs the same testsuite. -- https://code.launchpad.net/~ubuntu-testcase/ubuntu-manual-tests/+git/ubuntu-manual-tests/+merge/465118 Your team Ubuntu Testcase Admins is subscribed to branch ubuntu-manual-tests:main. -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
[Canonical-ubuntu-qa] [Bug 2059228] Re: Lubuntu install replacing partition not expected
Whole new testsuite is on the tracker now! ** Changed in: ubuntu-manual-tests Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Canonical's Ubuntu QA, which is subscribed to Ubuntu Manual Tests. https://bugs.launchpad.net/bugs/2059228 Title: Lubuntu install replacing partition not expected Status in Ubuntu Manual Tests: Fix Released Bug description: For install Lubuntu QA website test cases accept only 'install on entire disk' while you can also install replacing a partition or create new partition. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-manual-tests/+bug/2059228/+subscriptions -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
[Canonical-ubuntu-qa] [Bug 2021925] Re: Many i386 tests at autopkgtest.ubuntu.com failing on unsatisfiable dependencies
The i386 tests are still passing on all packages I checked on autopkgtests.ubuntu.com ** Attachment added: "Screenshot from 2024-04-30 13-43-43.png" https://bugs.launchpad.net/auto-package-testing/+bug/2021925/+attachment/5773369/+files/Screenshot%20from%202024-04-30%2013-43-43.png -- You received this bug notification because you are a member of Canonical's Ubuntu QA, which is subscribed to Auto Package Testing. https://bugs.launchpad.net/bugs/2021925 Title: Many i386 tests at autopkgtest.ubuntu.com failing on unsatisfiable dependencies Status in Auto Package Testing: New Bug description: While investigating https://autopkgtest.ubuntu.com/ status for packages I maintain myself I noticed that the i386 autopkgtests seem to be consistently failing on unsatisfiable dependencies. The autopkgtests environment seems to have a mix of amd64 and i386 packages, which leads to apt not being able to install all i386 dependencies. The identical packages pass on ci.debian.net and elsewhere, so the issue is somehow related to the auto-package-testing setup at autopkgtest.ubuntu.com, but I don't know exactly. Hopefully maintainers here have some ideas what is going on. Examples: - https://ci.debian.net/packages/a/ahven/: all pass - https://autopkgtest.ubuntu.com/packages/ahven: all but i386 pass - https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/i386/a/ahven/20230511_222759_6dc20@/log.gz: i386 log shows that a bunch of amd64 packages are installed, and then testbed preparation stops on Depends: libahven11-dev:i386 but it is not installable Depends: libahven30:i386 but it is not installable Depends: gprbuild:i386 but it is not installable - https://ci.debian.net/packages/m/mariadb/: all pass - https://autopkgtest.ubuntu.com/packages/mariadb: all but i386 pass - https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/i386/m/mariadb/20230528_204922_fb74b@/log.gz: i386 log shows a bunch of amd64 packages being installed/upgraded, and then fails on Depends: mariadb-server:i386 but it is not installable Depends: diffutils:i386 but it is not going to be installed To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/2021925/+subscriptions -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
