On Fri, Mar 25, 2022 at 12:27 PM Greg Wooledge <g...@wooledge.org> wrote:
> On Fri, Mar 25, 2022 at 04:51:51PM +0000, Adam Weremczuk wrote: > > [Tue Mar 22 00:24:10 2022] Tasks state (memory values in pages): > > [Tue Mar 22 00:24:10 2022] [ pid ] uid tgid total_vm rss > > pgtables_bytes swapents oom_score_adj name > > [Tue Mar 22 00:24:10 2022] [ 2211] 0 2211 14228 228 > 159744 > > 127 0 systemd > > [Tue Mar 22 00:24:10 2022] [ 2622] 0 2622 93208 59485 > > 753664 73 0 systemd-journal > > Well, at this point systemd-journald (I assume the name is truncated) > was using more memory than anything else. > ..... > [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process > 11695 > > So, next it killed dhcpd. And it still wasn't done. > > > [Tue Mar 22 00:24:10 2022] [ 21057] 0 21057 1069 31 > 53248 > > 0 0 apt.systemd.dai > > [Tue Mar 22 00:24:10 2022] [ 21065] 0 21065 17753 2552 > > 180224 0 0 apt-get > > [Tue Mar 22 00:24:10 2022] [ 21068] 0 21068 9475 110 > > 110592 0 0 systemd-journal > > [Tue Mar 22 00:24:10 2022] > oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/101,task_memcg=/lxc/101/ns,task=apt-get,pid=21065,uid=0 > > [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process > 21065 > > At this point, it killed apt-get. > > Looks like this system doesn't have enough memory to perform its daily > tasks (including what I'm guessing are unattended upgrades, triggering > calls to apt-get from a systemd timer). You'll either need to stop > letting it run those daily tasks, or add more memory, or add more swap, > or get rid of some of the other programs that are using memory. > If you have re-configured your apt repositories but made a mistake, or lost contact with them for other network reasons, you will see those automated apt-get commands stack-up over time. They reach a point in execution where they acquire a lock that blocks all other updates, so just keep piling up. Each consumes swap. If the CPU workload is the local bottleneck you may notice a CPU or two pegged at 100% as the first symptom. > If you really want the unattended upgrades, adding more swap would be > the easiest solution, but be warned that this could mean the system > will run extremely slowly during those unattended upgrades. That could > be something you don't care about, or something that matters a lot. Only > you would know. > If it's a server, servers should not swap and they should not get upgraded without purpose.
