On 17.5.2014 19:55, Tom Lane wrote: > Tomas Vondra <t...@fuzzy.cz> writes: >> ... then of course the usual 'terminating connection because of crash of >> another server process' warning. Apparently, it's getting killed by the >> OOM killer, because it exhausts all the memory assigned to that VM (2GB). > > Can you fix things so it runs into its process ulimit before the OOM killer > triggers? Then we'd get a memory map dumped to stderr, which would be > helpful in localizing the problem.
I did this in /etc/security/limits.d/80-pgbuild.conf: pgbuild hard as 1835008 so the user the buildfarm runs under will have up to ~1.75GB of RAM (of the 2GB available to the container). > >> ... So this seems like a >> memory leak somewhere in the cache invalidation code. > > Smells that way to me too, but let's get some more evidence. The tests are already running, and there are a few postgres processes: PID VIRT RES %CPU TIME+ COMMAND 11478 449m 240m 100.0 112:53.57 postgres: pgbuild regression [local] CREATE VIEW 11423 219m 19m 0.0 0:00.17 postgres: checkpointer process 11424 219m 2880 0.0 0:00.05 postgres: writer process 11425 219m 5920 0.0 0:00.12 postgres: wal writer process 11426 219m 2708 0.0 0:00.05 postgres: autovacuum launcher process 11427 79544 1836 0.0 0:00.17 postgres: stats collector process 11479 1198m 1.0g 0.0 91:09.99 postgres: pgbuild regression [local] CREATE INDEX waiting Attached is 'pmap -x' output for the two interesting processes (11478, 11479). Tomas
11478: postgres: pgbuild regression [local] SELECT Address Kbytes RSS Dirty Mode Mapping 0000000000400000 5840 3348 0 r-x-- postgres 0000000000bb4000 48 48 48 rw--- postgres 0000000000bc0000 308 188 188 rw--- [ anon ] 00000000015b3000 244 244 244 rw--- [ anon ] 00000000015f0000 956 956 956 rw--- [ anon ] 00007f642d696000 48 0 0 r-x-- libnss_files-2.12.so 00007f642d6a2000 2048 0 0 ----- libnss_files-2.12.so 00007f642d8a2000 4 4 4 r---- libnss_files-2.12.so 00007f642d8a3000 4 4 4 rw--- libnss_files-2.12.so 00007f642d8a4000 144920 5552 5552 rw-s- zero (deleted) 00007f643662a000 452 0 0 r-x-- libfreebl3.so 00007f643669b000 2044 0 0 ----- libfreebl3.so 00007f643689a000 8 8 8 r---- libfreebl3.so 00007f643689c000 4 4 4 rw--- libfreebl3.so 00007f643689d000 16 0 0 rw--- [ anon ] 00007f64368a1000 28 0 0 r-x-- libcrypt-2.12.so 00007f64368a8000 2048 0 0 ----- libcrypt-2.12.so 00007f6436aa8000 4 4 4 r---- libcrypt-2.12.so 00007f6436aa9000 4 4 4 rw--- libcrypt-2.12.so 00007f6436aaa000 184 0 0 rw--- [ anon ] 00007f6436ad8000 116 0 0 r-x-- libselinux.so.1 00007f6436af5000 2044 0 0 ----- libselinux.so.1 00007f6436cf4000 4 4 4 r---- libselinux.so.1 00007f6436cf5000 4 4 4 rw--- libselinux.so.1 00007f6436cf6000 4 4 4 rw--- [ anon ] 00007f6436cf7000 100 0 0 r-x-- libsasl2.so.2.0.23 00007f6436d10000 2044 0 0 ----- libsasl2.so.2.0.23 00007f6436f0f000 4 4 4 r---- libsasl2.so.2.0.23 00007f6436f10000 4 4 4 rw--- libsasl2.so.2.0.23 00007f6436f11000 228 0 0 r-x-- libnspr4.so 00007f6436f4a000 2044 0 0 ----- libnspr4.so 00007f6437149000 4 4 4 r---- libnspr4.so 00007f643714a000 8 8 8 rw--- libnspr4.so 00007f643714c000 8 0 0 rw--- [ anon ] 00007f643714e000 16 0 0 r-x-- libplc4.so 00007f6437152000 2044 0 0 ----- libplc4.so 00007f6437351000 4 4 4 r---- libplc4.so 00007f6437352000 4 4 4 rw--- libplc4.so 00007f6437353000 12 0 0 r-x-- libplds4.so 00007f6437356000 2044 0 0 ----- libplds4.so 00007f6437555000 4 4 4 r---- libplds4.so 00007f6437556000 4 4 4 rw--- libplds4.so 00007f6437557000 148 0 0 r-x-- libnssutil3.so 00007f643757c000 2048 0 0 ----- libnssutil3.so 00007f643777c000 24 24 24 r---- libnssutil3.so 00007f6437782000 4 4 4 rw--- libnssutil3.so 00007f6437783000 1236 0 0 r-x-- libnss3.so 00007f64378b8000 2048 0 0 ----- libnss3.so 00007f6437ab8000 20 20 20 r---- libnss3.so 00007f6437abd000 8 8 8 rw--- libnss3.so 00007f6437abf000 8 0 0 rw--- [ anon ] 00007f6437ac1000 160 0 0 r-x-- libsmime3.so 00007f6437ae9000 2044 0 0 ----- libsmime3.so 00007f6437ce8000 16 16 12 r---- libsmime3.so 00007f6437cec000 4 4 4 rw--- libsmime3.so 00007f6437ced000 228 0 0 r-x-- libssl3.so 00007f6437d26000 2048 0 0 ----- libssl3.so 00007f6437f26000 12 12 12 r---- libssl3.so 00007f6437f29000 4 4 4 rw--- libssl3.so 00007f6437f2a000 4 0 0 rw--- [ anon ] 00007f6437f2b000 56 0 0 r-x-- liblber-2.4.so.2.5.6 00007f6437f39000 2044 0 0 ----- liblber-2.4.so.2.5.6 00007f6438138000 4 4 4 r---- liblber-2.4.so.2.5.6 00007f6438139000 4 4 4 rw--- liblber-2.4.so.2.5.6 00007f643813a000 92 24 0 r-x-- libpthread-2.12.so 00007f6438151000 2048 0 0 ----- libpthread-2.12.so 00007f6438351000 4 4 4 r---- libpthread-2.12.so 00007f6438352000 4 4 4 rw--- libpthread-2.12.so 00007f6438353000 16 4 4 rw--- [ anon ] 00007f6438357000 88 0 0 r-x-- libresolv-2.12.so 00007f643836d000 2048 0 0 ----- libresolv-2.12.so 00007f643856d000 4 4 4 r---- libresolv-2.12.so 00007f643856e000 4 4 4 rw--- libresolv-2.12.so 00007f643856f000 8 0 0 rw--- [ anon ] 00007f6438571000 8 0 0 r-x-- libkeyutils.so.1.3 00007f6438573000 2044 0 0 ----- libkeyutils.so.1.3 00007f6438772000 4 4 4 r---- libkeyutils.so.1.3 00007f6438773000 4 4 4 rw--- libkeyutils.so.1.3 00007f6438774000 40 0 0 r-x-- libkrb5support.so.0.1 00007f643877e000 2044 0 0 ----- libkrb5support.so.0.1 00007f643897d000 4 4 4 r---- libkrb5support.so.0.1 00007f643897e000 4 4 4 rw--- libkrb5support.so.0.1 00007f643897f000 164 0 0 r-x-- libk5crypto.so.3.1 00007f64389a8000 2048 0 0 ----- libk5crypto.so.3.1 00007f6438ba8000 4 4 4 r---- libk5crypto.so.3.1 00007f6438ba9000 4 4 4 rw--- libk5crypto.so.3.1 00007f6438baa000 4 0 0 rw--- [ anon ] 00007f6438bab000 12 0 0 r-x-- libcom_err.so.2.1 00007f6438bae000 2044 0 0 ----- libcom_err.so.2.1 00007f6438dad000 4 4 4 r---- libcom_err.so.2.1 00007f6438dae000 4 4 4 rw--- libcom_err.so.2.1 00007f6438daf000 876 0 0 r-x-- libkrb5.so.3.3 00007f6438e8a000 2044 0 0 ----- libkrb5.so.3.3 00007f6439089000 40 40 40 r---- libkrb5.so.3.3 00007f6439093000 8 8 8 rw--- libkrb5.so.3.3 00007f6439095000 84 0 0 r-x-- libz.so.1.2.3 00007f64390aa000 2044 0 0 ----- libz.so.1.2.3 00007f64392a9000 4 4 4 r---- libz.so.1.2.3 00007f64392aa000 4 4 4 rw--- libz.so.1.2.3 00007f64392ab000 1580 432 0 r-x-- libc-2.12.so 00007f6439436000 2044 0 0 ----- libc-2.12.so 00007f6439635000 16 16 16 r---- libc-2.12.so 00007f6439639000 4 4 4 rw--- libc-2.12.so 00007f643963a000 20 20 20 rw--- [ anon ] 00007f643963f000 292 4 0 r-x-- libldap-2.4.so.2.5.6 00007f6439688000 2048 0 0 ----- libldap-2.4.so.2.5.6 00007f6439888000 8 8 8 r---- libldap-2.4.so.2.5.6 00007f643988a000 8 8 8 rw--- libldap-2.4.so.2.5.6 00007f643988c000 524 80 0 r-x-- libm-2.12.so 00007f643990f000 2044 0 0 ----- libm-2.12.so 00007f6439b0e000 4 4 4 r---- libm-2.12.so 00007f6439b0f000 4 4 4 rw--- libm-2.12.so 00007f6439b10000 8 4 0 r-x-- libdl-2.12.so 00007f6439b12000 2048 0 0 ----- libdl-2.12.so 00007f6439d12000 4 4 4 r---- libdl-2.12.so 00007f6439d13000 4 4 4 rw--- libdl-2.12.so 00007f6439d14000 28 4 0 r-x-- librt-2.12.so 00007f6439d1b000 2044 0 0 ----- librt-2.12.so 00007f6439f1a000 4 4 4 r---- librt-2.12.so 00007f6439f1b000 4 4 4 rw--- librt-2.12.so 00007f6439f1c000 260 4 0 r-x-- libgssapi_krb5.so.2.2 00007f6439f5d000 2048 0 0 ----- libgssapi_krb5.so.2.2 00007f643a15d000 4 4 4 r---- libgssapi_krb5.so.2.2 00007f643a15e000 8 8 8 rw--- libgssapi_krb5.so.2.2 00007f643a160000 1748 52 0 r-x-- libcrypto.so.1.0.1e 00007f643a315000 2048 0 0 ----- libcrypto.so.1.0.1e 00007f643a515000 108 108 108 r---- libcrypto.so.1.0.1e 00007f643a530000 48 48 48 rw--- libcrypto.so.1.0.1e 00007f643a53c000 16 8 8 rw--- [ anon ] 00007f643a540000 388 4 0 r-x-- libssl.so.1.0.1e 00007f643a5a1000 2048 0 0 ----- libssl.so.1.0.1e 00007f643a7a1000 16 16 16 r---- libssl.so.1.0.1e 00007f643a7a5000 28 28 28 rw--- libssl.so.1.0.1e 00007f643a7ac000 1312 12 0 r-x-- libxml2.so.2.7.6 00007f643a8f4000 2044 0 0 ----- libxml2.so.2.7.6 00007f643aaf3000 40 40 40 rw--- libxml2.so.2.7.6 00007f643aafd000 4 0 0 rw--- [ anon ] 00007f643aafe000 128 24 0 r-x-- ld-2.12.so 00007f643ac85000 524 332 332 rw--- [ anon ] 00007f643ad08000 60 60 60 rw--- [ anon ] 00007f643ad1a000 4 0 0 rw-s- [ shmid=0x2318001 ] 00007f643ad1b000 4 0 0 rw-s- [ shmid=0x2310000 ] 00007f643ad1c000 4 4 4 rw--- [ anon ] 00007f643ad1d000 4 4 4 r---- ld-2.12.so 00007f643ad1e000 4 4 4 rw--- ld-2.12.so 00007f643ad1f000 4 4 4 rw--- [ anon ] 00007fffaedfe000 148 88 88 rw--- [ stack ] 00007fffaef1c000 4 4 0 r-x-- [ anon ] ffffffffff600000 4 0 0 r-x-- [ anon ] ---------------- ------ ------ ------ total kB 225792 12104 8104
11479: postgres: pgbuild regression [local] CREATE INDEX Address Kbytes RSS Dirty Mode Mapping 0000000000400000 5840 3588 0 r-x-- postgres 0000000000bb4000 48 48 48 rw--- postgres 0000000000bc0000 308 52 52 rw--- [ anon ] 00000000015b3000 244 244 244 rw--- [ anon ] 00000000015f0000 1228 1132 1132 rw--- [ anon ] 00007f63f0699000 999412 999412 999412 rw--- [ anon ] 00007f642d696000 48 0 0 r-x-- libnss_files-2.12.so 00007f642d6a2000 2048 0 0 ----- libnss_files-2.12.so 00007f642d8a2000 4 4 4 r---- libnss_files-2.12.so 00007f642d8a3000 4 4 4 rw--- libnss_files-2.12.so 00007f642d8a4000 144920 25184 25184 rw-s- zero (deleted) 00007f643662a000 452 0 0 r-x-- libfreebl3.so 00007f643669b000 2044 0 0 ----- libfreebl3.so 00007f643689a000 8 8 8 r---- libfreebl3.so 00007f643689c000 4 4 4 rw--- libfreebl3.so 00007f643689d000 16 0 0 rw--- [ anon ] 00007f64368a1000 28 0 0 r-x-- libcrypt-2.12.so 00007f64368a8000 2048 0 0 ----- libcrypt-2.12.so 00007f6436aa8000 4 4 4 r---- libcrypt-2.12.so 00007f6436aa9000 4 4 4 rw--- libcrypt-2.12.so 00007f6436aaa000 184 0 0 rw--- [ anon ] 00007f6436ad8000 116 0 0 r-x-- libselinux.so.1 00007f6436af5000 2044 0 0 ----- libselinux.so.1 00007f6436cf4000 4 4 4 r---- libselinux.so.1 00007f6436cf5000 4 4 4 rw--- libselinux.so.1 00007f6436cf6000 4 4 4 rw--- [ anon ] 00007f6436cf7000 100 0 0 r-x-- libsasl2.so.2.0.23 00007f6436d10000 2044 0 0 ----- libsasl2.so.2.0.23 00007f6436f0f000 4 4 4 r---- libsasl2.so.2.0.23 00007f6436f10000 4 4 4 rw--- libsasl2.so.2.0.23 00007f6436f11000 228 0 0 r-x-- libnspr4.so 00007f6436f4a000 2044 0 0 ----- libnspr4.so 00007f6437149000 4 4 4 r---- libnspr4.so 00007f643714a000 8 8 8 rw--- libnspr4.so 00007f643714c000 8 0 0 rw--- [ anon ] 00007f643714e000 16 0 0 r-x-- libplc4.so 00007f6437152000 2044 0 0 ----- libplc4.so 00007f6437351000 4 4 4 r---- libplc4.so 00007f6437352000 4 4 4 rw--- libplc4.so 00007f6437353000 12 0 0 r-x-- libplds4.so 00007f6437356000 2044 0 0 ----- libplds4.so 00007f6437555000 4 4 4 r---- libplds4.so 00007f6437556000 4 4 4 rw--- libplds4.so 00007f6437557000 148 0 0 r-x-- libnssutil3.so 00007f643757c000 2048 0 0 ----- libnssutil3.so 00007f643777c000 24 24 24 r---- libnssutil3.so 00007f6437782000 4 4 4 rw--- libnssutil3.so 00007f6437783000 1236 0 0 r-x-- libnss3.so 00007f64378b8000 2048 0 0 ----- libnss3.so 00007f6437ab8000 20 20 20 r---- libnss3.so 00007f6437abd000 8 8 8 rw--- libnss3.so 00007f6437abf000 8 0 0 rw--- [ anon ] 00007f6437ac1000 160 0 0 r-x-- libsmime3.so 00007f6437ae9000 2044 0 0 ----- libsmime3.so 00007f6437ce8000 16 16 12 r---- libsmime3.so 00007f6437cec000 4 4 4 rw--- libsmime3.so 00007f6437ced000 228 0 0 r-x-- libssl3.so 00007f6437d26000 2048 0 0 ----- libssl3.so 00007f6437f26000 12 12 12 r---- libssl3.so 00007f6437f29000 4 4 4 rw--- libssl3.so 00007f6437f2a000 4 0 0 rw--- [ anon ] 00007f6437f2b000 56 0 0 r-x-- liblber-2.4.so.2.5.6 00007f6437f39000 2044 0 0 ----- liblber-2.4.so.2.5.6 00007f6438138000 4 4 4 r---- liblber-2.4.so.2.5.6 00007f6438139000 4 4 4 rw--- liblber-2.4.so.2.5.6 00007f643813a000 92 24 0 r-x-- libpthread-2.12.so 00007f6438151000 2048 0 0 ----- libpthread-2.12.so 00007f6438351000 4 4 4 r---- libpthread-2.12.so 00007f6438352000 4 4 4 rw--- libpthread-2.12.so 00007f6438353000 16 4 4 rw--- [ anon ] 00007f6438357000 88 0 0 r-x-- libresolv-2.12.so 00007f643836d000 2048 0 0 ----- libresolv-2.12.so 00007f643856d000 4 4 4 r---- libresolv-2.12.so 00007f643856e000 4 4 4 rw--- libresolv-2.12.so 00007f643856f000 8 0 0 rw--- [ anon ] 00007f6438571000 8 0 0 r-x-- libkeyutils.so.1.3 00007f6438573000 2044 0 0 ----- libkeyutils.so.1.3 00007f6438772000 4 4 4 r---- libkeyutils.so.1.3 00007f6438773000 4 4 4 rw--- libkeyutils.so.1.3 00007f6438774000 40 0 0 r-x-- libkrb5support.so.0.1 00007f643877e000 2044 0 0 ----- libkrb5support.so.0.1 00007f643897d000 4 4 4 r---- libkrb5support.so.0.1 00007f643897e000 4 4 4 rw--- libkrb5support.so.0.1 00007f643897f000 164 0 0 r-x-- libk5crypto.so.3.1 00007f64389a8000 2048 0 0 ----- libk5crypto.so.3.1 00007f6438ba8000 4 4 4 r---- libk5crypto.so.3.1 00007f6438ba9000 4 4 4 rw--- libk5crypto.so.3.1 00007f6438baa000 4 0 0 rw--- [ anon ] 00007f6438bab000 12 0 0 r-x-- libcom_err.so.2.1 00007f6438bae000 2044 0 0 ----- libcom_err.so.2.1 00007f6438dad000 4 4 4 r---- libcom_err.so.2.1 00007f6438dae000 4 4 4 rw--- libcom_err.so.2.1 00007f6438daf000 876 0 0 r-x-- libkrb5.so.3.3 00007f6438e8a000 2044 0 0 ----- libkrb5.so.3.3 00007f6439089000 40 40 40 r---- libkrb5.so.3.3 00007f6439093000 8 8 8 rw--- libkrb5.so.3.3 00007f6439095000 84 0 0 r-x-- libz.so.1.2.3 00007f64390aa000 2044 0 0 ----- libz.so.1.2.3 00007f64392a9000 4 4 4 r---- libz.so.1.2.3 00007f64392aa000 4 4 4 rw--- libz.so.1.2.3 00007f64392ab000 1580 464 0 r-x-- libc-2.12.so 00007f6439436000 2044 0 0 ----- libc-2.12.so 00007f6439635000 16 16 16 r---- libc-2.12.so 00007f6439639000 4 4 4 rw--- libc-2.12.so 00007f643963a000 20 20 20 rw--- [ anon ] 00007f643963f000 292 4 0 r-x-- libldap-2.4.so.2.5.6 00007f6439688000 2048 0 0 ----- libldap-2.4.so.2.5.6 00007f6439888000 8 8 8 r---- libldap-2.4.so.2.5.6 00007f643988a000 8 8 8 rw--- libldap-2.4.so.2.5.6 00007f643988c000 524 120 0 r-x-- libm-2.12.so 00007f643990f000 2044 0 0 ----- libm-2.12.so 00007f6439b0e000 4 4 4 r---- libm-2.12.so 00007f6439b0f000 4 4 4 rw--- libm-2.12.so 00007f6439b10000 8 4 0 r-x-- libdl-2.12.so 00007f6439b12000 2048 0 0 ----- libdl-2.12.so 00007f6439d12000 4 4 4 r---- libdl-2.12.so 00007f6439d13000 4 4 4 rw--- libdl-2.12.so 00007f6439d14000 28 4 0 r-x-- librt-2.12.so 00007f6439d1b000 2044 0 0 ----- librt-2.12.so 00007f6439f1a000 4 4 4 r---- librt-2.12.so 00007f6439f1b000 4 4 4 rw--- librt-2.12.so 00007f6439f1c000 260 4 0 r-x-- libgssapi_krb5.so.2.2 00007f6439f5d000 2048 0 0 ----- libgssapi_krb5.so.2.2 00007f643a15d000 4 4 4 r---- libgssapi_krb5.so.2.2 00007f643a15e000 8 8 8 rw--- libgssapi_krb5.so.2.2 00007f643a160000 1748 52 0 r-x-- libcrypto.so.1.0.1e 00007f643a315000 2048 0 0 ----- libcrypto.so.1.0.1e 00007f643a515000 108 108 108 r---- libcrypto.so.1.0.1e 00007f643a530000 48 48 48 rw--- libcrypto.so.1.0.1e 00007f643a53c000 16 8 8 rw--- [ anon ] 00007f643a540000 388 4 0 r-x-- libssl.so.1.0.1e 00007f643a5a1000 2048 0 0 ----- libssl.so.1.0.1e 00007f643a7a1000 16 16 16 r---- libssl.so.1.0.1e 00007f643a7a5000 28 28 28 rw--- libssl.so.1.0.1e 00007f643a7ac000 1312 12 0 r-x-- libxml2.so.2.7.6 00007f643a8f4000 2044 0 0 ----- libxml2.so.2.7.6 00007f643aaf3000 40 40 40 rw--- libxml2.so.2.7.6 00007f643aafd000 4 0 0 rw--- [ anon ] 00007f643aafe000 128 24 0 r-x-- ld-2.12.so 00007f643ab24000 1936 1932 1932 rw--- [ anon ] 00007f643ad08000 60 60 60 rw--- [ anon ] 00007f643ad1a000 4 0 0 rw-s- [ shmid=0x2318001 ] 00007f643ad1b000 4 0 0 rw-s- [ shmid=0x2310000 ] 00007f643ad1c000 4 4 4 rw--- [ anon ] 00007f643ad1d000 4 4 4 r---- ld-2.12.so 00007f643ad1e000 4 4 4 rw--- ld-2.12.so 00007f643ad1f000 4 4 4 rw--- [ anon ] 00007fffaedfe000 148 88 88 rw--- [ stack ] 00007fffaef1c000 4 4 0 r-x-- [ anon ] ffffffffff600000 4 0 0 r-x-- [ anon ] ---------------- ------ ------ ------ total kB 1226888 1033100 1028788
-- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
