Hi all, A typical shasta run would reserve almost all the memory on our arm64 CI runners, but it turned out it is possible to pass different parameters to the run, in such a way that the program does not reserve anywhere as much memory as the original test used to. This also happens to fix the same issue observed on Ubuntu runners, worked around by skipping the command hogging the memory.
Have a nice day, :) Étienne. ----- Forwarded message from Debian Bug Tracking System <ow...@bugs.debian.org> ----- Date: Mon, 04 Dec 2023 16:30:03 +0000 From: Debian Bug Tracking System <ow...@bugs.debian.org> To: Étienne Mollier <emoll...@debian.org> Subject: Bug#1053509: marked as done (shasta: autopkgtest regression on arm64: is memory suddenly not enough?) X-Mailer: MIME-tools 5.509 (Entity 5.509) Your message dated Mon, 04 Dec 2023 16:25:57 +0000 with message-id <e1rabld-00dubq...@fasolo.debian.org> and subject line Bug#1053509: fixed in shasta 0.11.1-3 has caused the Debian Bug report #1053509, regarding shasta: autopkgtest regression on arm64: is memory suddenly not enough? to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact ow...@bugs.debian.org immediately.) -- 1053509: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1053509 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems Date: Thu, 5 Oct 2023 13:39:14 +0200 From: Paul Gevers <elb...@debian.org> To: Debian Bug Tracking System <sub...@bugs.debian.org> Subject: shasta: autopkgtest regression on arm64: is memory suddenly not enough? User-Agent: Mozilla Thunderbird Source: shasta Version: 0.11.1-1 Severity: important User: debian...@lists.debian.org Usertags: regression Dear maintainer(s), Your package has an autopkgtest, great. However, it recently started failing on arm64 in unstable, testing *and* stable. Until mid September 2023, most runs passed (there are some historical rare failures where we don't have logs anymore), but somewhere between 2023-09-17 and 2023-10-01 the tests started failing (although, in testing we had some passes on 2023-09-28 and 2023-10-04, interestingly these runs too *much* longer than normal (factor ~20)). I first suspected that this regression aligned with my fix for bug 1050256 where I switched the kernel of our hosts to backports, but there are multiple good runs with that kernel. Today I learned that "Killed" in the log is often coming from the kernel when a program is allocating too much memory and the kernel kills the process. We have some striking logs of failures [1] where the shasta test actually bails out itself for lack of memory. So, there's a couple of things (weirdness and options): 1) why does it now suddenly start to (nearly always) fail across the board on arm64 (in Debian, Ubuntu still seems fine), without changes to the infrastructure that I know of? 2) do you also believe this is related to memory consumption? 3) If 2 == yes, what are the memory requirements for the test? The test *could* test for that before it starts and bail out (restriction: skippable with exit code 77 [2]) if the amount of memory available is too small. 4) just stop testing on arm64 (but again, in Ubuntu the test is still running fine) 5) the recent glibc security update also came to my mind, but that's not *yet* installed on our hosts, nor is it installed in the stable testbed. The release team has announced [3] that failing autopkgtest on amd64 and arm64 are considered RC in testing. However, due to nature of the symptoms, I've filed this as important for now. Paul [1] https://ci.debian.net/data/autopkgtest/testing/arm64/s/shasta/38544284/log.gz 48s 2023-Oct-04 12:01:05.603191 Memory allocation failure during mremap call for MemoryMapped::Vector. 48s This assembly requires more memory than available. 48s Rerun on a larger machine. [2] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst [3] https://lists.debian.org/debian-devel-announce/2019/07/msg00002.html Date: Mon, 04 Dec 2023 16:25:57 +0000 From: Debian FTP Masters <ftpmas...@ftp-master.debian.org> To: 1053509-cl...@bugs.debian.org Subject: Bug#1053509: fixed in shasta 0.11.1-3 Source: shasta Source-Version: 0.11.1-3 Done: Étienne Mollier <emoll...@debian.org> We believe that the bug you reported is fixed in the latest version of shasta, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to 1053...@bugs.debian.org, and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Étienne Mollier <emoll...@debian.org> (supplier of updated shasta package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing ftpmas...@ftp-master.debian.org) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 1.8 Date: Mon, 04 Dec 2023 16:48:02 +0100 Source: shasta Architecture: source Version: 0.11.1-3 Distribution: unstable Urgency: medium Maintainer: Debian Med Packaging Team <debian-med-packag...@lists.alioth.debian.org> Changed-By: Étienne Mollier <emoll...@debian.org> Closes: 1053509 Changes: shasta (0.11.1-3) unstable; urgency=medium . * d/t/run-unit-test: use older shasta config. The original test shipped with Nanopore-Sep2020 configuration file selected by shasta in the autopkgtest script. This caused shasta to reserve 8GiB of RAM on CI test hosts, which is just the limit on the arm64 runners. Dropping to the older Nanopore-Dec2019 configuration file causes shasta to reserve memory as needed instead of prefetching a huge amount of pages, causing out of memory conditions on runners. (Closes: #1053509) Checksums-Sha1: 040018ed0c9238a38ddc3230fd428cdf07daba72 2556 shasta_0.11.1-3.dsc e6c60a18e47219771ad09396f77ff758dd814ac1 10540 shasta_0.11.1-3.debian.tar.xz Checksums-Sha256: 2d3bb7c87946f64039363aa7e382b1a8073dbbe5a93698b2aaa81979f2dfaf2d 2556 shasta_0.11.1-3.dsc 33fbd5d5431870c04127884a618f7e4f43f3cd919c3267f57de383ca315a1f76 10540 shasta_0.11.1-3.debian.tar.xz Files: 6517b386aa94537798421b2f8ac6cba0 2556 science optional shasta_0.11.1-3.dsc 4b6ba7240615780b0ba071683ab16c7b 10540 science optional shasta_0.11.1-3.debian.tar.xz -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEj5GyJ8fW8rGUjII2eTz2fo8NEdoFAmVt9mQUHGVtb2xsaWVy QGRlYmlhbi5vcmcACgkQeTz2fo8NEdp5TQ/+OQMT7WV+pZBeKX/VT6NDY2MDue8c hqtsqUzqNQDsDl2QvszEgwd3c4kBYbS60dpbEM+hIo/DHqAHBR94tSBsxWJwQJ9T akZO7xAF6w9M+gNZ11sDnmC0bO/yk46S82ZqzqyV+XAea/aSNlqdC7fIHa/l8QLz fR4l8Z1AdKBxHsSx76Vv6Y4tIfbvDL4d3R1rPyNXj9L4XAGUgV/S0/YOXR2ermLq MGr4E4cGd717u8c+B8B2/x15HzwIv+53L3NbnXPSqtqimtA/itjPuqNsZm4E66Sn Abzq4tbNusFYXSHYpE+ascQMYd3Pq5ZID2VbJxKYqwBUIBiWWOj4yzzMnz7MEh8Z zT4PiLaes87oFNWOVxRMBw07bQKPHP94wQSSJ8bPim6oYUD/XrftIeLkcgr+P9r2 pINDENpOnne3+GgwxL2UYDfoDMAimN9owrTuiCTqjRKGpGfUs/3KV432rKA0DATb MkdQjRD6Wtz5Kf8/cn1H6oS8LbS3bEbzgHThEy9Dyd66g0RlL8oWd3hrPGIGxZsA quUcW8WGDypwyv3fXkut53XGgmu1aU8scAODQ4E6gLD8PVn1JzAP31Fvt0yvmrif Jwl+U4g0bglGMQFdk/M7fciLFmMjp2smdY8hQEW2HyuaeDmKdmdJfiHk2ubFNamg jytusyXTO8sbdeY= =zFjP -----END PGP SIGNATURE----- ----- End forwarded message ----- -- .''`. Étienne Mollier <emoll...@debian.org> : :' : gpg: 8f91 b227 c7d6 f2b1 948c 8236 793c f67e 8f0d 11da `. `' sent from /dev/pts/2, please excuse my verbosity `- on air: Karmamoi - Black Hole Era
signature.asc
Description: PGP signature
