> On Mar 13, 2023, at 10:37 AM, Joe Conway <m...@joeconway.com> wrote:
>
> On 3/13/23 13:55, Israel Brewster wrote:
>> 1) They reference a “Memory cgroup out of memory”, which refers back
>> to the opening comment on Joe Conway’s message - this would imply to
>> me that I *AM* running with a cgroup memory.limit set. Not sure how
>> that changes things?
>
> cgroup memory limit is enforced regardless of the actual host level memory
> pressure. As an example, if your host VM has 128 GB of memory, but your
> cgroup memory limit is 512MB, you will get an OOM kill when the sum memory
> usage of all of your postgres processes (and anything else sharing the same
> cgroup) exceeds 512 MB, even if the host VM has nothing else going on
> consuming memory.
>
> You can check if a memory is set by reading the corresponding virtual file,
> e.g:
>
> 8<-------------------
> # cat
> /sys/fs/cgroup/memory/system.slice/postgresql.service/memory.limit_in_bytes
> 9223372036854710272
> 8<-------------------
>
> A few notes:
> 1/ The specific path to memory.limit_in_bytes might vary, but this example is
> the default for the RHEL 8 postgresql 10 RPM.
Not finding that file specifically (this is probably too much info, but…):
root@novarupta:~# ls /sys/fs/cgroup/system.slice/
-.mount cgroup.threads dev-hugepages.mount
memory.events.local memory.swap.events proc-diskstats.mount
ssh.service system-postgresql.slice
systemd-resolved.service
accounts-daemon.service cgroup.type dev-lxc-console.mount
memory.high memory.swap.high proc-loadavg.mount
sys-devices-system-cpu-online.mount systemd-initctl.socket
systemd-sysctl.service
cgroup.controllers console-getty.service dev-lxc-tty1.mount
memory.low memory.swap.max proc-meminfo.mount
sys-devices-virtual-net.mount systemd-journal-flush.service
systemd-sysusers.service
cgroup.events console-setup.service dev-lxc-tty2.mount
memory.max networkd-dispatcher.service proc-stat.mount
sys-fs-fuse-connections.mount systemd-journald-audit.socket
systemd-tmpfiles-setup-dev.service
cgroup.freeze cpu.pressure dev-mqueue.mount
memory.min pids.current proc-swaps.mount
sys-kernel-debug.mount
systemd-journald-dev-log.socket systemd-tmpfiles-setup.service
cgroup.max.depth cpu.stat dev-ptmx.mount
memory.numa_stat pids.events
proc-sys-kernel-random-boot_id.mount syslog.socket
systemd-journald.service systemd-update-utmp.service
cgroup.max.descendants cron.service io.pressure
memory.oom.group pids.max proc-sys-net.mount
sysstat.service systemd-journald.socket
systemd-user-sessions.service
cgroup.procs data.mount keyboard-setup.service
memory.pressure pool.mount
'proc-sysrq\x2dtrigger.mount' 'system-container\x2dgetty.slice'
systemd-logind.service ufw.service
cgroup.stat dbus.service memory.current
memory.stat postfix.service proc-uptime.mount
system-modprobe.slice systemd-networkd.service
uuidd.socket
cgroup.subtree_control dbus.socket memory.events
memory.swap.current proc-cpuinfo.mount rsyslog.service
system-postfix.slice systemd-remount-fs.service
root@novarupta:~# ls /sys/fs/cgroup/system.slice/system-postgresql.slice/
cgroup.controllers cgroup.max.depth cgroup.stat cgroup.type
io.pressure memory.events.local memory.max memory.oom.group
memory.swap.current memory.swap.max pids.max
cgroup.events cgroup.max.descendants cgroup.subtree_control
cpu.pressure memory.current memory.high memory.min
memory.pressure memory.swap.events pids.current
postgresql@13-main.service
cgroup.freeze cgroup.procs cgroup.threads cpu.stat
memory.events memory.low memory.numa_stat memory.stat
memory.swap.high pids.events
root@novarupta:~# ls
/sys/fs/cgroup/system.slice/system-postgresql.slice/postgresql@13-main.service/
cgroup.controllers cgroup.max.depth cgroup.stat cgroup.type
io.pressure memory.events.local memory.max memory.oom.group
memory.swap.current memory.swap.max pids.max
cgroup.events cgroup.max.descendants cgroup.subtree_control
cpu.pressure memory.current memory.high memory.min
memory.pressure memory.swap.events pids.current
cgroup.freeze cgroup.procs cgroup.threads cpu.stat
memory.events memory.low memory.numa_stat memory.stat
memory.swap.high pids.events
>
> 2/ The value above, 9223372036854710272 basically means "no limit" has been
> set.
>
> 3/ The example assumes cgroup v1. There are very few distro's that enable
> cgroup v2 by default, and generally I have not seen much cgroup v2 usage in
> the wild (although I strongly recommend it), but if you are using cgroup v2
> the names have changed. You can check by doing:
>
> 8<--cgroupv2 enabled-----------------
> # stat -fc %T /sys/fs/cgroup/
> cgroup2fs
> 8<--cgroupv1 enabled-----------------
> # stat -fc %T /sys/fs/cgroup/
> tmpfs
> 8<-------------------
Looks like V2:
root@novarupta:~# stat -fc %T /sys/fs/cgroup/
cgroup2fs
root@novarupta:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell: 907-328-9145
>
>> 2) All the entries contain the line "oom_score_adj:0”, which would
>> seem to imply that the postmaster, with its -900 score is not being
>> directly targeted by the OOM killer.
>
> Sounds correct
>
> --
> Joe Conway
> PostgreSQL Contributors Team
> RDS Open Source Databases
> Amazon Web Services: https://aws.amazon.com
>
