Hey Lucas,
Thanks for flagging that, it had been indeed blocked again while they
were trying to cut some load issue due to IA bots/scapers.
Do you have an idea of the number of API requests/pages load UDD is
usually doing on a $period (day?), I wonder if the load is really that
high and if there is some client optimization that could be done in UDD...
Cheers,
Sébastien
Le 06/04/2026 à 21:26, Lucas Nussbaum a écrit :
On 03/04/26 at 11:16 -0400, Jeremy Bícha wrote:
On Tue, Mar 31, 2026 at 5:17 PM Lucas Nussbaum <[email protected]> wrote:
On 31/03/26 at 08:32 -0700, Jeremy Bícha wrote:
https://udd.debian.org/udd-status.cgi is reporting that these Ubuntu
importers haven't run successfully since March 23, 2026:
- archive-ubuntu
- ubuntu-bugs
- ubuntu-upload-history
The ubuntu-upload-history log excerpt includes:
urllib.error.HTTPError: HTTP Error 503: Service Unavailable
ERROR:launchpad-ubuntu-changes:Error: Something broke badly.
I know that Launchpad has been having difficulty staying fully
available and fast in the past few weeks.
I'd expect for udd to retry imports later after failure but the "next
time" column also says March 23 so maybe these importers aren't
retrying any more?
Hi,
Thanks for the report.
I think that the reason is that ubuntu-upload-history got stuck, and as
a result, blocked the other importers because they are in the same
"queue" in the UDD orchestrator.
I added a timeout. Let's see if this fixes the issue.
ubuntu_upload_history is still lagging behind (and Launchpad is
struggling today!), but the other Ubuntu jobs are up-to-date or close
enough to it. It's those other jobs that are powering the dashboards
that I use so the situation is much better for me now.
Hi,
It looks like ullmann.debian.org (the machine behind udd.debian.org) is
backlisted by launchpad again.
ullmann:~$ curl -6 -kv
https://launchpad.net/ubuntu/+archive/primary/+sourcepub/18280857/+files/changelog
* Host launchpad.net:443 was resolved.
* IPv6: 2620:2d:4000:1009::f3, 2620:2d:4000:1009::3ba
* IPv4: (none)
* Trying [2620:2d:4000:1009::f3]:443...
^C
ullmann:~$ mtr -c 5 -r -6 -T -P 443 2620:2d:4000:1009::f3
Start: 2026-04-06T19:24:21+0000
HOST: ullmann Loss% Snt Last Avg Best Wrst StDev
1.|-- 2607:f8f0:614:1::2 0.0% 5 0.4 0.4 0.3 0.8 0.2
2.|-- 2607:f8f0:610:4::302 0.0% 5 0.4 0.4 0.3 0.4 0.0
3.|-- 2607:f8f0:610:4::305 0.0% 5 1.1 1.0 0.8 1.1 0.1
2607:f8f0:610:4::304
4.|-- 381-oran-cr1-ubc.vncv1.bc 0.0% 5 2.0 1.6 1.0 2.0 0.5
2607:f8f0:610:4::1f1
5.|-- 381-oran-cr1-ubc.vncv1.bc 0.0% 5 5105. 1634. 1.7 5105. 1990.1
6.|-- 2607:f8f0:1400::16 0.0% 5 1029. 1025. 2.2 3062. 1249.4
7.|-- 2607:f8f0:1400::16 0.0% 5 2050. 616.7 2.0 2050. 916.0
8.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
9.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
10.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
11.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
12.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
13.|-- he-as6939.torix.ca 0.0% 5 3119. 1895. 52.1 4154. 1832.7
14.|-- he-as6939.torix.ca 0.0% 5 2082. 1068. 51.3 2102. 1020.5
15.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
16.|-- port-channel1.core1.lon6. 40.0% 5 127.7 809.4 127.7 1154. 590.4
17.|-- swp9.il3-core2.canonical. 0.0% 5 1157. 741.4 126.6 2168. 913.9
port-channel1.core1.lon9.he.net
port-channel1.core1.lon6.he.net
swp9.il3-core1.canonical.com
18.|-- swp9.il3-core2.canonical. 0.0% 5 126.9 127.2 126.9 128.0 0.4
swp9.il3-core1.canonical.com
port-channel1.core1.lon6.he.net
19.|-- port-channel1.core1.lon6. 0.0% 5 128.2 534.9 127.0 1157. 557.6
vrrp.bond-core.il3-fw.canonical.com
20.|-- swp9.il3-core2.canonical. 0.0% 5 3197. 1965. 127.4 4219. 1829.8
swp9.il3-core1.canonical.com
21.|-- vrrp.bond-core.il3-fw.can 0.0% 5 7249. 2167. 127.2 7249. 2965.1
22.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
ullmann.d.o is 2607:f8f0:614:1::1274:38
If some could reach out to the Canonical team in charge of this, it
would be great.
Lucas
