On Fri, Mar 25, 2022 at 12:27 PM Greg Wooledge <g...@wooledge.org> wrote:

> On Fri, Mar 25, 2022 at 04:51:51PM +0000, Adam Weremczuk wrote:
> > [Tue Mar 22 00:24:10 2022] Tasks state (memory values in pages):
> > [Tue Mar 22 00:24:10 2022] [  pid  ]   uid  tgid total_vm      rss
> > pgtables_bytes swapents oom_score_adj name
> > [Tue Mar 22 00:24:10 2022] [   2211]     0  2211    14228 228
> 159744
> > 127             0 systemd
> > [Tue Mar 22 00:24:10 2022] [   2622]     0  2622    93208 59485
> > 753664       73             0 systemd-journal
>
> Well, at this point systemd-journald (I assume the name is truncated)
> was using more memory than anything else.
> .....

> [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process
> 11695
>
> So, next it killed dhcpd.  And it still wasn't done.
>
> > [Tue Mar 22 00:24:10 2022] [  21057]     0 21057     1069 31
> 53248
> > 0             0 apt.systemd.dai
> > [Tue Mar 22 00:24:10 2022] [  21065]     0 21065    17753 2552
> > 180224        0             0 apt-get
> > [Tue Mar 22 00:24:10 2022] [  21068]     0 21068     9475 110
> > 110592        0             0 systemd-journal
> > [Tue Mar 22 00:24:10 2022]
> oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/101,task_memcg=/lxc/101/ns,task=apt-get,pid=21065,uid=0
> > [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process
> 21065
>
> At this point, it killed apt-get.
>
> Looks like this system doesn't have enough memory to perform its daily
> tasks (including what I'm guessing are unattended upgrades, triggering
> calls to apt-get from a systemd timer).  You'll either need to stop
> letting it run those daily tasks, or add more memory, or add more swap,
> or get rid of some of the other programs that are using memory.
>

If you have re-configured your apt repositories but made a mistake, or lost
contact with them for
other network reasons, you will see those automated apt-get commands
stack-up over time.
They reach a point in execution where they acquire a lock that blocks all
other updates, so just
keep piling up. Each consumes swap. If the CPU workload is the local
bottleneck you may notice
a CPU or two pegged at 100% as the first symptom.


> If you really want the unattended upgrades, adding more swap would be
> the easiest solution, but be warned that this could mean the system
> will run extremely slowly during those unattended upgrades.  That could
> be something you don't care about, or something that matters a lot.  Only
> you would know.
>

If it's a server, servers should not swap and they should not get upgraded
without purpose.

Reply via email to