Memcached was the process requesting more memory, not necessary the process using a lot of memory. Looking through the logs you attached it seems Beam.smp (which I believe is RabbitMQ), MariaDB, and Koha Plack are the top three memory processes. rebuild_zebra is also using a lot of paged memory, but its total vm usage isn't high; I'm not sure how that factors in here. There may be processes (like Kernel processes) that aren't in that list, I'm not going to do the math to see if all the processes are listed. I'm not even sure if the values here are bytes or memory pages. The OOM Killer decided that Plack was the most expendable process, that's why it killed it. Basically, all you can do is reduce memory usage or add memory. I don't know if adding more swap space is a good solution, swap memory is a lot slower than physical ram, but it's also a lot cheaper. Looks like all the swap space is allocated too, so it is being used as far as I can tell. I might see if all of the Plack processes are in use, and reduce that if it's higher than the peak active requests (Maybe Apache can provide that number). That's the plack_workers tag in the config file, and defaults to 2, so that only works if you have increased the number significantly past the default. You can also look at the plack_max_requests tag, to recycle the workers quicker. That may free up ram quicker, but spawning workers takes time. That setting defaults to 50 (requests).
On Mon, Mar 10, 2025 at 10:27 AM Michael Kuhn via Koha-devel < koha-devel@lists.koha-community.org> wrote: > Hi > > Our host "bib" owns 4 virtual CPU, 8 GB RAM and 18 GB swap space. It > runs on Debian GNU/Linux 12 with Koha 24.05.01 and around 78'000 > bibliographic records. > > Since some time Koha sporadically just stops (today already twice), in > such cases log file "/var/log/syslog" contains messages as attached at > the end of this e-mail (the "oom-killer" is not always invoked by > memcached). The messages say the host is out of memory so the Koha > process was killed... I have already increased the swap space from 2 GB > to 18 GB but to no avail. > > How can we find out what is consuming so much memory causing Koha to get > killed? How can we avoid this? > > Best wishes: Michael > -- > Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis > Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz > T 0041 (0)61 261 55 61 · E m...@adminkuhn.ch · W www.adminkuhn.ch > > > > > 2025-03-10T12:48:00.668764+01:00 bib kernel: [769596.762201] memcached > invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), > order=0, oom_score_adj=0 > 2025-03-10T12:48:00.670036+01:00 bib kernel: [769596.762885] CPU: 3 PID: > 2332685 Comm: memcached Not tainted 6.1.0-31-amd64 #1 Debian 6.1.128-1 > 2025-03-10T12:48:00.670038+01:00 bib kernel: [769596.763487] Hardware > name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014 > 2025-03-10T12:48:00.670039+01:00 bib kernel: [769596.764019] Call Trace: > 2025-03-10T12:48:00.670039+01:00 bib kernel: [769596.764528] <TASK> > 2025-03-10T12:48:00.670040+01:00 bib kernel: [769596.765027] > dump_stack_lvl+0x44/0x5c > 2025-03-10T12:48:00.670040+01:00 bib kernel: [769596.765539] > dump_header+0x4a/0x211 > 2025-03-10T12:48:00.670041+01:00 bib kernel: [769596.766045] > oom_kill_process.cold+0xb/0x10 > 2025-03-10T12:48:00.670041+01:00 bib kernel: [769596.766535] > out_of_memory+0x1fd/0x4c0 > 2025-03-10T12:48:00.670042+01:00 bib kernel: [769596.767046] > __alloc_pages_slowpath.constprop.0+0xc83/0xe40 > 2025-03-10T12:48:00.670042+01:00 bib kernel: [769596.767537] > __alloc_pages+0x305/0x330 > 2025-03-10T12:48:00.670043+01:00 bib kernel: [769596.768014] > folio_alloc+0x17/0x50 > 2025-03-10T12:48:00.670043+01:00 bib kernel: [769596.768499] > __filemap_get_folio+0x155/0x340 > 2025-03-10T12:48:00.670044+01:00 bib kernel: [769596.768967] > filemap_fault+0x139/0x910 > 2025-03-10T12:48:00.670044+01:00 bib kernel: [769596.769395] ? > filemap_map_pages+0x153/0x700 > 2025-03-10T12:48:00.670044+01:00 bib kernel: [769596.769820] > __do_fault+0x33/0x110 > 2025-03-10T12:48:00.670045+01:00 bib kernel: [769596.770303] > do_fault+0x1b9/0x410 > 2025-03-10T12:48:00.670157+01:00 bib kernel: [769596.770734] > __handle_mm_fault+0x660/0xfa0 > 2025-03-10T12:48:00.670298+01:00 bib kernel: [769596.771189] > handle_mm_fault+0xdb/0x2d0 > 2025-03-10T12:48:00.670299+01:00 bib kernel: [769596.771594] > do_user_addr_fault+0x191/0x550 > 2025-03-10T12:48:00.670300+01:00 bib kernel: [769596.771982] > exc_page_fault+0x70/0x170 > 2025-03-10T12:48:00.670300+01:00 bib kernel: [769596.772411] > asm_exc_page_fault+0x22/0x30 > 2025-03-10T12:48:00.670301+01:00 bib kernel: [769596.772829] RIP: > 0033:0x557b12827140 > 2025-03-10T12:48:00.670302+01:00 bib kernel: [769596.773207] Code: > Unable to access opcode bytes at 0x557b12827116. > 2025-03-10T12:48:00.670302+01:00 bib kernel: [769596.773589] RSP: > 002b:00007ffc7f74a9c8 EFLAGS: 00010246 > 2025-03-10T12:48:00.670303+01:00 bib kernel: [769596.773991] RAX: > 00007fc1802bd440 RBX: 0000557b44749ac0 RCX: 0000000000000669 > 2025-03-10T12:48:00.670303+01:00 bib kernel: [769596.774387] RDX: > 0000000000000000 RSI: 0000000000000001 RDI: 00000000ffffffff > 2025-03-10T12:48:00.670303+01:00 bib kernel: [769596.774723] RBP: > 0000000000000001 R08: 0000557b12827140 R09: 00007fc1802dd000 > 2025-03-10T12:48:00.670304+01:00 bib kernel: [769596.775111] R10: > 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 > 2025-03-10T12:48:00.670305+01:00 bib kernel: [769596.775506] R13: > 0000000000000000 R14: 000000007fffffff R15: 0000557b1285cb80 > 2025-03-10T12:48:00.670305+01:00 bib kernel: [769596.775831] </TASK> > 2025-03-10T12:48:00.670305+01:00 bib kernel: [769596.776226] Mem-Info: > 2025-03-10T12:48:00.670365+01:00 bib kernel: [769596.776592] > active_anon:550829 inactive_anon:1391840 isolated_anon:0 > 2025-03-10T12:48:00.670366+01:00 bib kernel: [769596.776592] > active_file:0 inactive_file:94 isolated_file:0 > 2025-03-10T12:48:00.670366+01:00 bib kernel: [769596.776592] > unevictable:0 dirty:0 writeback:0 > 2025-03-10T12:48:00.670367+01:00 bib kernel: [769596.776592] > slab_reclaimable:9935 slab_unreclaimable:13188 > 2025-03-10T12:48:00.670367+01:00 bib kernel: [769596.776592] > mapped:1134 shmem:5697 pagetables:5740 > 2025-03-10T12:48:00.670367+01:00 bib kernel: [769596.776592] > sec_pagetables:0 bounce:0 > 2025-03-10T12:48:00.670368+01:00 bib kernel: [769596.776592] > kernel_misc_reclaimable:0 > 2025-03-10T12:48:00.670368+01:00 bib kernel: [769596.776592] free:25298 > free_pcp:0 free_cma:0 > 2025-03-10T12:48:00.670368+01:00 bib kernel: [769596.789186] Node 0 > active_anon:2203316kB inactive_anon:5567360kB active_file:80kB > inactive_file:156kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB mapped:4536kB dirty:0kB writeback:0kB shmem:22788kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 876544kB writeback_tmp:0kB > kernel_stack:9056kB pagetables:22960kB sec_pagetables:0kB > all_unreclaimable? no > 2025-03-10T12:48:00.670369+01:00 bib kernel: [769596.790069] Node 0 DMA > free:14336kB boost:0kB min:128kB low:160kB high:192kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB > present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB > local_pcp:0kB free_cma:0kB > 2025-03-10T12:48:00.670370+01:00 bib kernel: [769596.790924] > lowmem_reserve[]: 0 2899 7843 7843 7843 > 2025-03-10T12:48:00.670370+01:00 bib kernel: [769596.791252] Node 0 > DMA32 free:44376kB boost:0kB min:24936kB low:31168kB high:37400kB > reserved_highatomic:0KB active_anon:842432kB inactive_anon:2119316kB > active_file:0kB inactive_file:84kB unevictable:0kB writepending:0kB > present:3129196kB managed:3018408kB mlocked:0kB bounce:0kB free_pcp:0kB > local_pcp:0kB free_cma:0kB > 2025-03-10T12:48:00.670370+01:00 bib kernel: [769596.792175] > lowmem_reserve[]: 0 0 4943 4943 4943 > 2025-03-10T12:48:00.670371+01:00 bib kernel: [769596.792455] Node 0 > Normal free:42480kB boost:0kB min:42516kB low:53144kB high:63772kB > reserved_highatomic:0KB active_anon:1360884kB inactive_anon:3447664kB > active_file:120kB inactive_file:0kB unevictable:0kB writepending:0kB > present:5242880kB managed:5062604kB mlocked:0kB bounce:0kB free_pcp:0kB > local_pcp:0kB free_cma:0kB > 2025-03-10T12:48:00.670371+01:00 bib kernel: [769596.793398] > lowmem_reserve[]: 0 0 0 0 0 > 2025-03-10T12:48:00.670372+01:00 bib kernel: [769596.793716] Node 0 DMA: > 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB > 1*2048kB (M) 3*4096kB (M) = 14336kB > 2025-03-10T12:48:00.670372+01:00 bib kernel: [769596.794404] Node 0 > DMA32: 115*4kB (UME) 68*8kB (UME) 110*16kB (UME) 94*32kB (UME) 78*64kB > (UME) 56*128kB (UE) 40*256kB (UME) 15*512kB (E) 3*1024kB (UE) 3*2048kB > (E) 0*4096kB = 45068kB > 2025-03-10T12:48:00.670373+01:00 bib kernel: [769596.795118] Node 0 > Normal: 412*4kB (UME) 307*8kB (UME) 166*16kB (UME) 167*32kB (UME) > 83*64kB (UME) 50*128kB (UME) 23*256kB (E) 13*512kB (ME) 7*1024kB (ME) > 0*2048kB 0*4096kB = 43528kB > 2025-03-10T12:48:00.670373+01:00 bib kernel: [769596.795916] Node 0 > hugepages_total=0 hugepages_free=0 hugepages_surp=0 > hugepages_size=1048576kB > 2025-03-10T12:48:00.670373+01:00 bib kernel: [769596.796271] Node 0 > hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > 2025-03-10T12:48:00.670374+01:00 bib kernel: [769596.796713] 155111 > total pagecache pages > 2025-03-10T12:48:00.670374+01:00 bib kernel: [769596.797122] 149255 > pages in swap cache > 2025-03-10T12:48:00.670374+01:00 bib kernel: [769596.797463] Free swap > = 0kB > 2025-03-10T12:48:00.670375+01:00 bib kernel: [769596.797848] Total swap > = 1998844kB > 2025-03-10T12:48:00.670922+01:00 bib kernel: [769596.798238] 2097017 > pages RAM > 2025-03-10T12:48:00.670926+01:00 bib kernel: [769596.798577] 0 pages > HighMem/MovableOnly > 2025-03-10T12:48:00.670926+01:00 bib kernel: [769596.798954] 72924 pages > reserved > 2025-03-10T12:48:00.671386+01:00 bib kernel: [769596.799356] 0 pages > hwpoisoned > 2025-03-10T12:48:00.671388+01:00 bib kernel: [769596.799693] Tasks state > (memory values in pages): > 2025-03-10T12:48:00.671388+01:00 bib kernel: [769596.800070] [ pid ] > uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name > 2025-03-10T12:48:00.671389+01:00 bib kernel: [769596.800463] [ 274] > 0 274 30075 22 241664 274 -250 > systemd-journal > 2025-03-10T12:48:00.671389+01:00 bib kernel: [769596.800839] [ 295] > 0 295 6770 278 86016 221 -1000 systemd-udevd > 2025-03-10T12:48:00.671390+01:00 bib kernel: [769596.801224] [ 524] > 997 524 22521 154 77824 82 0 systemd-timesyn > 2025-03-10T12:48:00.671390+01:00 bib kernel: [769596.801603] [ 609] > 0 609 2372 35 53248 28 0 cron > 2025-03-10T12:48:00.671390+01:00 bib kernel: [769596.801997] [ 610] > 100 610 2011 143 61440 8 -900 dbus-daemon > 2025-03-10T12:48:00.671391+01:00 bib kernel: [769596.802387] [ 612] > 0 612 55447 184 81920 116 0 rsyslogd > 2025-03-10T12:48:00.671391+01:00 bib kernel: [769596.802749] [ 613] > 0 613 4185 203 73728 73 0 systemd-logind > 2025-03-10T12:48:00.671391+01:00 bib kernel: [769596.803147] [ 629] > 0 629 895 40 49152 3 0 atd > 2025-03-10T12:48:00.671392+01:00 bib kernel: [769596.803558] [ 666] > 0 666 1499 202 53248 45 0 dhclient > 2025-03-10T12:48:00.671392+01:00 bib kernel: [769596.803913] [ 715] > 0 715 1435 100 49152 105 0 dhclient > 2025-03-10T12:48:00.671393+01:00 bib kernel: [769596.804313] [ 747] > 104 747 2208 550 61440 34 0 epmd > 2025-03-10T12:48:00.671393+01:00 bib kernel: [769596.804716] [ 753] > 0 753 173229 3280 430080 880 0 > fail2ban-server > 2025-03-10T12:48:00.671393+01:00 bib kernel: [769596.805078] [ 774] > 106 774 920414 28673 1294336 23431 -500 beam.smp > 2025-03-10T12:48:00.671393+01:00 bib kernel: [769596.805494] [ 797] > 0 797 2188 20 53248 2 0 agetty > 2025-03-10T12:48:00.671394+01:00 bib kernel: [769596.806006] [ 812] > 0 812 3859 213 65536 120 -1000 sshd > 2025-03-10T12:48:00.671394+01:00 bib kernel: [769596.806567] [ 841] > 106 841 616 19 53248 6 -500 erl_child_setup > 2025-03-10T12:48:00.671395+01:00 bib kernel: [769596.807045] [ 1774] > 0 1774 10665 108 69632 53 0 master > 2025-03-10T12:48:00.671395+01:00 bib kernel: [769596.807442] [ 1779] > 102 1779 10700 127 81920 48 0 qmgr > 2025-03-10T12:48:00.671396+01:00 bib kernel: [769596.807848] [ 2023] > 106 2023 848 11 45056 14 -500 inet_gethost > 2025-03-10T12:48:00.671396+01:00 bib kernel: [769596.808192] [ 2026] > 106 2026 848 24 45056 8 -500 inet_gethost > 2025-03-10T12:48:00.671397+01:00 bib kernel: [769596.808560] [ 2034] > 106 2034 644 14 40960 14 -500 sh > 2025-03-10T12:48:00.671397+01:00 bib kernel: [769596.808953] [ 3262] > 103 3262 2058793 518769 7716864 392357 0 mariadbd > 2025-03-10T12:48:00.671397+01:00 bib kernel: [769596.809287] [ 392666] > 102 392666 12109 128 90112 225 0 tlsmgr > 2025-03-10T12:48:00.671398+01:00 bib kernel: [769596.809647] [2332677] > 0 2332677 3456 434 61440 303 0 apache2 > 2025-03-10T12:48:00.671398+01:00 bib kernel: [769596.810031] [2332685] > 105 2332685 107470 2677 159744 2787 0 memcached > 2025-03-10T12:48:00.671399+01:00 bib kernel: [769596.810358] [2332735] > 1003 2332735 2128 21 49152 15 0 daemon > 2025-03-10T12:48:00.671399+01:00 bib kernel: [769596.810703] [2332737] > 1003 2332737 14826 161 110592 256 0 zebrasrv > 2025-03-10T12:48:00.671399+01:00 bib kernel: [769596.811075] [2332802] > 1003 2332802 6211 2612 94208 287 0 starman > master > 2025-03-10T12:48:00.671400+01:00 bib kernel: [769596.811400] [2332846] > 1003 2332846 2128 37 49152 0 0 daemon > 2025-03-10T12:48:00.671400+01:00 bib kernel: [769596.811734] [2332847] > 1003 2332847 29751 13047 221184 414 0 > background_jobs > 2025-03-10T12:48:00.671401+01:00 bib kernel: [769596.812076] [2332875] > 1003 2332875 2128 21 45056 14 0 daemon > 2025-03-10T12:48:00.671401+01:00 bib kernel: [769596.812396] [2332877] > 1003 2332877 41320 19112 323584 4526 0 > background_jobs > 2025-03-10T12:48:00.671401+01:00 bib kernel: [769596.812718] [2332902] > 1003 2332902 2128 13 49152 22 0 daemon > 2025-03-10T12:48:00.671402+01:00 bib kernel: [769596.813044] [2332904] > 1003 2332904 73759 36200 540672 6622 0 > rebuild_zebra.p > 2025-03-10T12:48:00.671402+01:00 bib kernel: [769596.813369] [1011996] > 102 1011996 10671 154 73728 0 0 pickup > 2025-03-10T12:48:00.671402+01:00 bib kernel: [769596.813651] [1014020] > 33 1014020 3517 422 61440 316 0 apache2 > 2025-03-10T12:48:00.671403+01:00 bib kernel: [769596.813976] [1015250] > 33 1015250 3517 426 61440 311 0 apache2 > 2025-03-10T12:48:00.671403+01:00 bib kernel: [769596.814307] [1018044] > 33 1018044 3517 472 61440 265 0 apache2 > 2025-03-10T12:48:00.671403+01:00 bib kernel: [769596.814583] [1018109] > 33 1018109 3517 465 61440 272 0 apache2 > 2025-03-10T12:48:00.671404+01:00 bib kernel: [769596.814882] [1018113] > 33 1018113 3517 473 61440 265 0 apache2 > 2025-03-10T12:48:00.671404+01:00 bib kernel: [769596.815171] [1020113] > 33 1020113 3517 473 61440 264 0 apache2 > 2025-03-10T12:48:00.671404+01:00 bib kernel: [769596.815506] [1024635] > 33 1024635 3517 474 61440 264 0 apache2 > 2025-03-10T12:48:00.671405+01:00 bib kernel: [769596.815796] [1026139] > 33 1026139 3517 474 61440 264 0 apache2 > 2025-03-10T12:48:00.671405+01:00 bib kernel: [769596.816089] [1026170] > 33 1026170 3517 473 61440 264 0 apache2 > 2025-03-10T12:48:00.671405+01:00 bib kernel: [769596.816378] [1026641] > 1003 1026641 1185289 1122584 9445376 29621 0 > /usr/share/koha > 2025-03-10T12:48:00.671406+01:00 bib kernel: [769596.816688] [1026652] > 33 1026652 3517 473 61440 264 0 apache2 > 2025-03-10T12:48:00.671406+01:00 bib kernel: [769596.816972] [1027075] > 1003 1027075 99496 34733 749568 33424 0 > /usr/share/koha > 2025-03-10T12:48:00.671407+01:00 bib kernel: [769596.817289] [1027426] > 1003 1027426 3594 624 65536 241 0 apache2 > 2025-03-10T12:48:00.671416+01:00 bib kernel: [769596.817615] [1027453] > 1003 1027453 3592 669 61440 240 0 apache2 > 2025-03-10T12:48:00.671416+01:00 bib kernel: [769596.817890] [1027749] > 1003 1027749 16877 179 114688 2268 0 zebrasrv > 2025-03-10T12:48:00.671416+01:00 bib kernel: [769596.818218] [1027761] > 1003 1027761 3538 563 61440 244 0 apache2 > 2025-03-10T12:48:00.671417+01:00 bib kernel: [769596.818501] [1027769] > 1003 1027769 16966 2354 114688 229 0 zebrasrv > 2025-03-10T12:48:00.671417+01:00 bib kernel: [769596.818812] [1027770] > 1003 1027770 3523 559 61440 241 0 apache2 > 2025-03-10T12:48:00.671417+01:00 bib kernel: [769596.819085] > > oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/koha-common.service,task=/usr/share/koha,pid=1026641,uid=1003 > 2025-03-10T12:48:00.671418+01:00 bib kernel: [769596.819717] Out of > memory: Killed process 1026641 (/usr/share/koha) total-vm:4741156kB, > anon-rss:4490332kB, file-rss:4kB, shmem-rss:0kB, UID:1003 > pgtables:9224kB oom_score_adj:0 > 2025-03-10T12:48:00.721396+01:00 bib systemd[1]: koha-common.service: A > process of this unit has been killed by the OOM killer. > 2025-03-10T12:48:03.702481+01:00 bib koha-common[1027819]: Stopping > Plack daemon for bib:. > 2025-03-10T12:48:03.755927+01:00 bib koha-common[1027861]: Stopping Koha > worker daemon for bib (default):. > 2025-03-10T12:48:03.789372+01:00 bib koha-common[1027886]: Stopping Koha > worker daemon for bib (long_tasks):. > 2025-03-10T12:48:03.825612+01:00 bib koha-common[1027911]: Stopping Koha > indexing daemon for bib:. > 2025-03-10T12:48:05.057975+01:00 bib systemd[1]: koha-common.service: > Failed with result 'oom-kill'. > 2025-03-10T12:48:05.058217+01:00 bib systemd[1]: koha-common.service: > Consumed 2d 22h 25min 610ms CPU time. > _______________________________________________ > Koha-devel mailing list > Koha-devel@lists.koha-community.org > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : https://www.koha-community.org/ > git : https://git.koha-community.org/ > bugs : https://bugs.koha-community.org/ > -- Michael Hafen Washington County School District Technology Department Systems & Security Analyst
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
