Hi, On 07/04/26 at 17:08 +0200, Sebastien Bacher wrote: > Hey again Lucas, > > Urg, no wonder UDD is getting blocked; the ubuntu_bugs_gatherer script > basically parse 163k launchpad pages, doing 10 parallel requesters without > any rate limiting and not slowing doing in case the service struggles and > start returning 429 or 503. > > Ideally that would use launchpadlibAPI to use searchTasks() with a > modified_since= to limit the list to bugs that changed since the previous > run (if you have that information) or over a day (or a few days to > accomodate for flakyness). > > Switching to use API instead of text pages parsing requires a bit of work, > but maybe a start for now would be to lower the number of parallel workers > (+ a small sleep(0.3) or something between calls + pause if the service > starts returning errors)? > The launchpad team confirmed that the number of queries/the daily frequency > isn't the problem but the "spike" is
For now I reduced the parallelism (num_fetchers) from 10 to 2, which should clearly help spread the load over time. Maybe let's see if that's better? I could reduce it to 1 and add a delay, but long-running importers are a bit painful for other reasons, so I'd rather avoid that. Lucas
