Paride Legovini wrote on 09/03/2023: > Hi Andreas, > > Andreas Hasenack wrote on 08/03/2023: >> Hi, >> >> taking this as a sample: >> >> https://autopkgtest.ubuntu.com/packages/f/fdroidserver/lunar/ppc64el >> <https://autopkgtest.ubuntu.com/packages/f/fdroidserver/lunar/ppc64el> >> >> It looks like something got slower since a few days ago, and tests >> started being killed due to a timeout ("kind: install" timeout, which >> has a default of 3000s in autopkgtest). >> >> By the end of February 2023, the passing tests were taking about 20min. >> Then a little bit more up to 26min in March 2nd, and then they all >> started to fail. >> >> Looking at the logs, we see apt-get install being interrupted by the >> autopkgtest timeout mid package download: >> >> Get:242 http://ftpmaster.internal/ubuntu <http://ftpmaster.internal/ubuntu> >> lunar/universe ppc64el apksigner all 31.0.2-1 [438 kB] >> Get:243 http://ftpmaster.internal/ubuntu <http://ftpmaster.internal/ubuntu> >> lunar/main ppc64el openjdk-17-jdk-headless ppc64el 17.0.6+10-1 [217 MB] >> autopkgtest-virt-ssh [04:23:09]: ------- nova console-log >> 766e17f7-5365-4cdf-8202-35b6b88a7cdc >> (adt-lunar-ppc64el-fdroidserver-20230303-012357-lrg-root5) ------ > > For some reason a few days ago the connection between te bos01 and bos02 > datacenters to ftpmaster.internal got extremely slow. Download speeds > are of the order of 100kB/s, sometimes even less, while downloading from > the Internet (archive.ubuntu.com) works just fine (so it is not storage). > > AIUI the connection from bos0[12] to ftpmaster passes through an haproxy > and then an IPsec tunnel crossing an ocean. While maybe not ideal, the link > was ~10 times faster up to a few days ago. > > I filed a ticket about it with the details I could gather. If that's not > sorted out Soon(tm) we can workaround the issue by switching to > us.archive.ubuntu.com or by bumping the timeouts.
This is half worked around now: one of the two datacenters which host the ppc64/s390x/arm64 autopkgtest VMs now has fast connectivity to ftpmaster.internal, and the other is not as slow as before (perhaps some link is less loaded now?). So we shouldn't be hitting as many timeouts as before. I'm trying to have the other datacenter get the same treatment. Paride -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
