Hey again Lucas,
Urg, no wonder UDD is getting blocked; the ubuntu_bugs_gatherer script
basically parse 163k launchpad pages, doing 10 parallel requesters
without any rate limiting and not slowing doing in case the service
struggles and start returning 429 or 503.
Ideally that would use launchpadlibAPI to use searchTasks() with a
modified_since= to limit the list to bugs that changed since the
previous run (if you have that information) or over a day (or a few days
to accomodate for flakyness).
Switching to use API instead of text pages parsing requires a bit of
work, but maybe a start for now would be to lower the number of parallel
workers (+ a small sleep(0.3) or something between calls + pause if the
service starts returning errors)?
The launchpad team confirmed that the number of queries/the daily
frequency isn't the problem but the "spike" is
Cheers,
Sébastien
Le 07/04/2026 à 16:24, Lucas Nussbaum a écrit :
https://salsa.debian.org/qa/udd/-/blob/master/udd/ubuntu_bugs_gatherer.py?ref_type=heads
for the second script.
The refresh frequency of ubuntu-bugs could probably be reduced if that
helps.
