Public bug reported: When upgrading runc to the latest version from focal-updates, Kubernetes, using containerd, fails to start new containers after a certain amount of container have been reached a certain amount (more than 100 ~ 150 containers). With the previous version of runc, I was able to run more than 340 containers on a single server without any issue.
I got those logs in containerd (journalctl -u containerd) ``` May 05 00:48:17 node6 containerd[1368656]: time="2021-05-05T00:48:17.260349619Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured" May 05 00:48:21 node6 containerd[1368656]: time="2021-05-05T00:48:21.807947009Z" level=error msg="get state for 71005cc34ac16f6976a7aebd72faa734792639b003c5dec67f666449cf6eb8d7" error="context deadline exceeded: unknown" May 05 00:48:23 node6 containerd[1368656]: time="2021-05-05T00:48:23.810434820Z" level=error msg="get state for 71005cc34ac16f6976a7aebd72faa734792639b003c5dec67f666449cf6eb8d7" error="context deadline exceeded: unknown" May 05 00:48:25 node6 containerd[1368656]: time="2021-05-05T00:48:25.813453574Z" level=error msg="get state for c6497bbf82f668f8b8b534204cd2efaa30a97f884da3dd5fca4f51489298f855" error="context deadline exceeded: unknown" May 05 00:48:27 node6 containerd[1368656]: time="2021-05-05T00:48:27.816285268Z" level=error msg="get state for c6497bbf82f668f8b8b534204cd2efaa30a97f884da3dd5fca4f51489298f855" error="context deadline exceeded: unknown" May 05 00:48:29 node6 containerd[1368656]: time="2021-05-05T00:48:29.823052797Z" level=error msg="get state for f90f054966ab6b78304797d420a1f948c85e5b6e425037417f8966c08cd9a0f1" error="context deadline exceeded: unknown" May 05 00:48:31 node6 containerd[1368656]: time="2021-05-05T00:48:31.826412096Z" level=error msg="get state for f90f054966ab6b78304797d420a1f948c85e5b6e425037417f8966c08cd9a0f1" error="context deadline exceeded: unknown" ``` This version of runc triggered the problem: ``` runc (1.0.0~rc93-0ubuntu1~20.04.1) focal; urgency=medium * Backport version 1.0.0~rc93-0ubuntu1 from Hirsute (LP: #1919322, LP: #1916485). -- Lucas Kanashiro <kanash...@ubuntu.com> Tue, 16 Mar 2021 15:34:35 -0300 ``` ``` # runc -v runc version spec: 1.0.2-dev go: go1.13.8 libseccomp: 2.5.1 ``` Reverting to the previous version of runc solved the problem, and I was able to run more than 340 pods / containers without any error. ``` apt-get install runc=1.0.0~rc10-0ubuntu1 # runc -v runc version spec: 1.0.1-dev ``` ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: runc 1.0.0~rc93-0ubuntu1~20.04.1 ProcVersionSignature: Ubuntu 5.4.0-72.80-generic 5.4.101 Uname: Linux 5.4.0-72-generic x86_64 ApportVersion: 2.20.11-0ubuntu27.16 Architecture: amd64 CasperMD5CheckResult: skip Date: Wed May 5 12:06:30 2021 SourcePackage: runc UpgradeStatus: No upgrade log present (probably fresh install) ** Affects: runc (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug focal uec-images -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs