Hello,
čt 14. 7. 2022 v 23:31 odesílatel Aleš Zelený <zeleny.a...@gmail.com> napsal: > > > Dne čt 14. 7. 2022 23:11 uživatel Tomas Vondra < > tomas.von...@enterprisedb.com> napsal: > >> >> >> On 7/14/22 21:25, Aleš Zelený wrote: >> > >> > st 13. 7. 2022 v 2:20 odesílatel Michael Paquier <mich...@paquier.xyz >> > <mailto:mich...@paquier.xyz>> napsal: >> > >> > On Mon, Jul 11, 2022 at 10:50:23AM +0200, Aleš Zelený wrote: >> > > So far, it has happened three times (during a single week) from >> > the 14.3 -> >> > > 14.4 upgrade, before 14.4 we haven't suffered from such an issue. >> > > >> > > Questions: >> > > 1) Can we safely downgrade from 14.4 to 14.3 by shutting down the >> > instance >> > > and reinstalling 14.3 PG packages (to prove, that the issue >> > disappear)? >> > > 2) What is the best way to diagnose what is the root cause? >> > >> > Hmm. 14.4 has nothing in its release notes that would point to a >> > change in the vacuum or autovacuum's code paths: >> > https://www.postgresql.org/docs/14/release-14-4.html#id-1.11.6.5.4 >> > <https://www.postgresql.org/docs/14/release-14-4.html#id-1.11.6.5.4 >> > >> > >> > There is nothing specific after a look at the changes as of, and I >> am >> > not grabbing anything that would imply a change in memory context >> > handling either: >> > `git log --stat REL_14_3..REL_14_4` >> > `git diff REL_14_3..REL_14_4 -- *.c` >> > >> > Saying that, you should be able to downgrade safely as there are no >> > changes in WAL format or such that would break things. Saying that, >> > the corruption issue caused by CONCURRENTLY is something you'd still >> > have to face. >> > >> > >> > Thanks, good to know that, we can use it for a test case, since we >> > already hit the CONCURRENTLY bug on 14.3. >> > >> > > 2022-07-02 14:48:07 CEST [3930]: [3-1] user=,db=,host=,app= >> > ERROR: out of >> > > memory >> > > 2022-07-02 14:48:07 CEST [3930]: [4-1] user=,db=,host=,app= >> > DETAIL: Failed >> > > on request of size 152094068 in memory context >> > "TopTransactionContext". >> > > 2022-07-02 14:48:07 CEST [3930]: [5-1] user=,db=,host=,app= >> CONTEXT: >> > > automatic vacuum of table "prematch.replication.tab_queue_tmp" >> > >> > This is the interesting part. Do you happen to use logical >> > replication in a custom C++ plugin? >> > >> > >> > We are using logical replication to other instances (pg_output) and >> > decoderbufs >> > https://github.com/debezium/postgres-decoderbufs >> > <https://github.com/debezium/postgres-decoderbufs> for other >> applications. >> > >> >> This is probably just a red herring - std:bad_alloc is what the process >> that runs into the overcommit limit gets. But the real issue (e.g. >> memory leak) is likely somewhere else - different part of the code, >> different process ... >> >> > ... >> > >> > Checking the RssAnon from proc/pid/status I've found some points where >> > RssAnon memory usage grew very steep for a minute, but no "suspicious" >> > queries/arguments were found in the instance logfile. >> > >> > Any hint, on how to get the root cause would be appreciated since so far >> > I've failed to isolate the issue reproducible testcase. >> > At least I hope that looking for the RssAnon process memory is an >> > appropriate metric, if not, let me know and I'll try to update the >> > monitoring to get the root cause. >> > >> > I can imagine a workaround with client application regular reconnect..., >> > but u to 14.3 it works, so I'd like to fix the issue either on our >> > application side or at PG side if it is a PG problem. >> > >> >> I think it's be interesting to get memory context stats from the >> processes consuming a lot of memory. If you know which processes are >> suspect (and it seems you know, bacause if a reconnect helps it's the >> backend handling the connection), you can attach a debugger and do >> >> $ gdb -p $PID >> call MemoryContextStats(TopMemoryContext) >> >> which will log info about memory contexts, just like autovacuum. >> Hopefully that tells us memory context is bloated, and that might point >> us to particular part of the code. >> > after some time, I've found a process consuming over 1GB of memory" -bash-4.2$ grep RssAnon /proc/*/status | sort -nk2 | tail /proc/12156/status:RssAnon: 902984 kB /proc/8303/status:RssAnon: 912848 kB /proc/31187/status:RssAnon: 920424 kB /proc/16469/status:RssAnon: 922864 kB /proc/13769/status:RssAnon: 925824 kB /proc/15449/status:RssAnon: 932452 kB /proc/26144/status:RssAnon: 936892 kB /proc/22570/status:RssAnon: 937920 kB /proc/2910/status:RssAnon: 1039124 kB /proc/17048/status:RssAnon: 1053952 kB Here are memory contexts for PID 17048: TopMemoryContext: 422592 total in 14 blocks; 42536 free (169 chunks); 380056 used pgstat TabStatusArray lookup hash table: 16384 total in 2 blocks; 7520 free (3 chunks); 8864 used PL/pgSQL function: 16384 total in 2 blocks; 5832 free (1 chunks); 10552 used: feeds_api.update_match(bigint,date,time without time zone) PL/pgSQL function: 16384 total in 2 blocks; 2080 free (1 chunks); 14304 used: opportunities.tr_taboddstopchangeshistory_oddstopchangestime() PL/pgSQL function: 16384 total in 2 blocks; 8576 free (1 chunks); 7808 used: opportunities.tr_tabodds_risklimits() PL/pgSQL function: 16384 total in 2 blocks; 2072 free (1 chunks); 14312 used: opportunities.tr_taboddshistory_oddshistorytime() PL/pgSQL function: 32768 total in 3 blocks; 6488 free (1 chunks); 26280 used: opportunities.fix_last_minute_odds(bigint,bigint) PL/pgSQL function: 65536 total in 4 blocks; 26624 free (1 chunks); 38912 used: opportunities.create_odds_tip_longname_lang(bigint,integer,integer,integer) PL/pgSQL function: 16384 total in 2 blocks; 3472 free (1 chunks); 12912 used: feeds_api.create_odds(bigint,integer,numeric,numeric,numeric,numeric,numeric,integer,jsonb) PL/pgSQL function: 65536 total in 4 blocks; 29648 free (2 chunks); 35888 used: feeds_api.create_opportunity(integer,bigint,bigint,integer,integer,integer,integer,integer,integer,i... PL/pgSQL function: 16384 total in 2 blocks; 7576 free (1 chunks); 8808 used: feeds.tr_tabopptysparentdata_ismain() PL/pgSQL function: 32768 total in 3 blocks; 11720 free (1 chunks); 21048 used: feeds_api.set_game_data(integer,bigint,bigint,character varying) PL/pgSQL function: 32768 total in 3 blocks; 7712 free (1 chunks); 25056 used: opportunities_api.set_oppty_info(bigint,boolean) PL/pgSQL function: 8192 total in 1 blocks; 3488 free (1 chunks); 4704 used: feeds_api.set_oppty_info_number(bigint) PL/pgSQL function: 16384 total in 2 blocks; 6616 free (1 chunks); 9768 used: b5_offer.tr_tabexternalopportunities_checknotnull() PL/pgSQL function: 32768 total in 3 blocks; 17928 free (5 chunks); 14840 used: opportunities.generate_league2oppty_correlations(bigint) PL/pgSQL function: 131072 total in 5 blocks; 61648 free (6 chunks); 69424 used: opportunities.generate_league2league_correlations(bigint) PL/pgSQL function: 16384 total in 2 blocks; 5616 free (1 chunks); 10768 used: opportunities.tr_tabopptytree_addcorrelations() PL/pgSQL function: 65536 total in 4 blocks; 19864 free (1 chunks); 45672 used: opportunities.replication_151_tab_oppty_tree() PL/pgSQL function: 16384 total in 2 blocks; 7568 free (1 chunks); 8816 used: greep.tr_shared_changedat() PL/pgSQL function: 32768 total in 3 blocks; 2112 free (1 chunks); 30656 used: feeds_api.create_match(integer,integer,integer,jsonb,bigint,integer,integer,integer,integer,jsonb,in... HandleParallelMessages: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used pgqs_localhash: 21448 total in 2 blocks; 2616 free (0 chunks); 18832 used PL/pgSQL function: 16384 total in 2 blocks; 2072 free (1 chunks); 14312 used: opportunities.tr_tabopptyhistory_opptyhistorytime() PL/pgSQL function: 16384 total in 2 blocks; 2360 free (1 chunks); 14024 used: opportunities.tr_tabopportunitiesevaluationqueue_validtime() PL/pgSQL function: 16384 total in 2 blocks; 2512 free (1 chunks); 13872 used: opportunities.tr_tabopportunitiesevaluationqueue_evaluatedtime() PL/pgSQL function: 16384 total in 2 blocks; 2360 free (1 chunks); 14024 used: opportunities.tr_tabopportunitiesevaluationqueue_addedtime() PL/pgSQL function: 16384 total in 2 blocks; 7512 free (1 chunks); 8872 used: feeds_api.evaluate_opportunities(bigint[],integer,integer,integer,integer,integer) Btree proof lookup cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used PL/pgSQL function: 8192 total in 1 blocks; 1384 free (1 chunks); 6808 used: feeds_api.change_odds_is_winning(bigint[],integer) PL/pgSQL function: 32768 total in 3 blocks; 3160 free (1 chunks); 29608 used: opportunities.tr_tabopportunity_resultchange() PL/pgSQL function: 16384 total in 2 blocks; 4424 free (1 chunks); 11960 used: opportunities.tr_tabopportunity_opptynamechange() PL/pgSQL function: 16384 total in 2 blocks; 7448 free (1 chunks); 8936 used: opportunities.tr_tabopportunity_marketingstatechange() PL/pgSQL function: 65536 total in 4 blocks; 27224 free (2 chunks); 38312 used: opportunities.tr_tabopportunity_history() PL/pgSQL function: 32768 total in 3 blocks; 16224 free (3 chunks); 16544 used: opportunities.tr_tabopportunity_datetimechange() PL/pgSQL function: 32768 total in 3 blocks; 15440 free (1 chunks); 17328 used: opportunities.tr_tabopportunity_addcorrelation() PL/pgSQL function: 32768 total in 3 blocks; 15672 free (1 chunks); 17096 used: opportunities.tr_tabopportunities_notifycanchanges() Sequence values: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used PL/pgSQL function: 262144 total in 6 blocks; 80744 free (0 chunks); 181400 used: opportunities.replication_151_tab_opportunities() PL/pgSQL function: 16384 total in 2 blocks; 8904 free (5 chunks); 7480 used: betsys.lastuser_change() PL/pgSQL function: 16384 total in 2 blocks; 2448 free (1 chunks); 13936 used: opportunities.tr_tabopportunities_opptyevaltime() PL/pgSQL function: 65536 total in 4 blocks; 29064 free (1 chunks); 36472 used: opportunities.tr_tabopportunities_checkstatechange() PL/pgSQL function: 16384 total in 2 blocks; 7568 free (1 chunks); 8816 used: greep.tr_shared_changedat() PL/pgSQL function: 32768 total in 3 blocks; 15512 free (1 chunks); 17256 used: opportunities.tr_tabopportunities_opptyenddatetime() PL/pgSQL function: 32768 total in 3 blocks; 16008 free (4 chunks); 16760 used: opportunities.tr_tabopportunity_checkstatechange_au() PL/pgSQL function: 8192 total in 1 blocks; 1752 free (1 chunks); 6440 used: feeds_api.change_opportunities_state(bigint[],integer) PL/pgSQL function: 65536 total in 4 blocks; 27640 free (1 chunks); 37896 used: opportunities.tr_tabodds_aggregs() PL/pgSQL function: 16384 total in 2 blocks; 7128 free (1 chunks); 9256 used: opportunities.tr_tabodds_notifyoddsblocked() PL/pgSQL function: 65536 total in 4 blocks; 31200 free (1 chunks); 34336 used: opportunities.tr_tabodds_history() Record information cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used PL/pgSQL function: 131072 total in 5 blocks; 65672 free (2 chunks); 65400 used: opportunities.replication_151_tab_odds() RI compare cache: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used RI query cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used RI constraint cache: 40648 total in 2 blocks; 2616 free (0 chunks); 38032 used PL/pgSQL function: 16384 total in 2 blocks; 7920 free (1 chunks); 8464 used: opportunities.tr_tabodds_fixoddsvalue() PLpgSQL cast info: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used PLpgSQL cast cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used PL/pgSQL function: 16384 total in 2 blocks; 7568 free (5 chunks); 8816 used: greep.tr_shared_changedat() CFuncHash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used TableSpace cache: 8192 total in 1 blocks; 2088 free (0 chunks); 6104 used Operator lookup cache: 40960 total in 3 blocks; 18888 free (10 chunks); 22072 used Type information cache: 24376 total in 2 blocks; 2616 free (0 chunks); 21760 used Function stat entries: 16384 total in 2 blocks; 4576 free (2 chunks); 11808 used Prepared Queries: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used RowDescriptionContext: 8192 total in 1 blocks; 6888 free (0 chunks); 1304 used MessageContext: 8192 total in 1 blocks; 6888 free (1 chunks); 1304 used Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used smgr relation table: 65536 total in 4 blocks; 20768 free (14 chunks); 44768 used TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used TopPortalContext: 8192 total in 1 blocks; 7928 free (2 chunks); 264 used Relcache by OID: 32768 total in 3 blocks; 13608 free (6 chunks); 19160 used CacheMemoryContext: 2919800 total in 48 blocks; 218712 free (4 chunks); 2701088 used CachedPlanSource: 4096 total in 3 blocks; 1496 free (0 chunks); 2600 used: SELECT * FROM feeds_api.change_odds_is_winning(_id_odds := $1, _odds_is_winning := $2) CachedPlanQuery: 4096 total in 3 blocks; 1568 free (3 chunks); 2528 used CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES (currval('replication.tab_queue_d... CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES (currval('replication.tab_queue_d... index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: tab_queue_tmp_pkey CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES (currval('repli... CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES (currval('replication.tab_queue_dat... CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES (currval('repli... CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES (currval('replication.tab_queue_dat... CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data) VALUES (currval('replication.tab_queue_dat... index info: 4096 total in 3 blocks; 2264 free (2 chunks); 1832 used: ix_tabopportunitiesevaluationqueue_idopportunity_pk_active_2 index info: 4096 total in 3 blocks; 2024 free (2 chunks); 2072 used: ix_tabopportunitiesevaluationqueue_idopportunity_pk_active_1 index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunitiesevaluationqueue_idodds index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: pk_tabopportunitiesevaluationqueue_idopportunitiesevaluationque CachedPlan: 16384 total in 5 blocks; 3656 free (0 chunks); 12728 used: INSERT INTO opportunities.tab_opportunities_evaluation_queue (id_odds, evaluation_type, id_opportuni... CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ... CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ... CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ... CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ... CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ... CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ... index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: i_replication_tabqueuedata_idreplicationgroup index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: tab_queue_data_pkey CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ... CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlan: 134336 total in 10 blocks; 42112 free (0 chunks); 92224 used: WITH opp AS MATERIALIZED ( SELECT DISTINCT o2.id_opportunity FROM o... CachedPlan: 65536 total in 7 blocks; 31968 free (0 chunks); 33568 used: SELECT lc.contestant_name_lang FROM opportunities.tab_op... CachedPlan: 65536 total in 7 blocks; 31968 free (0 chunks); 33568 used: SELECT lc.contestant_name_lang FROM opportunities.tab_op... CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlan: 8192 total in 4 blocks; 3672 free (0 chunks); 4520 used: EXISTS ( SELECT FROM opportunities.tab_oppty_tree WHE... CachedPlan: 65536 total in 7 blocks; 32224 free (0 chunks); 33312 used: SELECT CASE WHEN SUBSTRING(op.marketing_states FROM 1 FOR 1) = '1' AND op.oppty_end_date + op... CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used CachedPlan: 4096 total in 3 blocks; 392 free (0 chunks); 3704 used: INSERT INTO opportunities.tab_oppty_tree ( id_opportunity, id_opportunity_parent... CachedPlan: 35840 total in 6 blocks; 15344 free (1 chunks); 20496 used: INSERT INTO opportunities.tab_opportunities ( id_event, id_oppty_type, i... CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used CachedPlan: 8192 total in 4 blocks; 3664 free (1 chunks); 4528 used: NOT EXISTS ( SELECT FROM opportunities.tab_oppty_tree WH... CachedPlan: 16384 total in 5 blocks; 7960 free (2 chunks); 8424 used: UPDATE opportunities.tab_opportunities SET oppty_info_number = _info_number, id_op... CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlanQuery: 8192 total in 4 blocks; 1504 free (3 chunks); 6688 used SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used CachedPlan: 8192 total in 4 blocks; 616 free (2 chunks); 7576 used: SELECT oppty_info_number, id_event, id_oppty_type ... CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used CachedPlan: 134408 total in 10 blocks; 51528 free (0 chunks); 82880 used: INSERT INTO opportunities.tab_correlations (id_opportunity, id_opportunity_2) SELECT crr... CachedPlan: 277504 total in 9 blocks; 15680 free (0 chunks); 261824 used: INSERT INTO opportunities.tab_correlations (id_opportunity, id_opportunity_2) -- id_league from ... CachedPlan: 134144 total in 8 blocks; 30024 free (0 chunks); 104120 used: SELECT nm1.contestant_name, COALESCE(nm2.contestant_name, nm1.contestant_name), ... CachedPlan: 32768 total in 6 blocks; 2064 free (0 chunks); 30704 used: SELECT evn.id_league FROM opportunities.tab_opportunities AS opp JOIN opportunities.tab_e... CachedPlan: 16384 total in 5 blocks; 6880 free (0 chunks); 9504 used: NEW.id_opportunity <> NEW.id_opportunity_parent OR EXISTS ( SELECT ... CachedPlan: 16384 total in 5 blocks; 4952 free (1 chunks); 11432 used: (TG_OP='INSERT' AND CASE WHEN get_session_var('id_data_provider') = '12' THEN FALSE ELSE (SELECT CAS... CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used CachedPlan: 4096 total in 3 blocks; 392 free (0 chunks); 3704 used: INSERT INTO opportunities.tab_oppty_tree ( id_opportunity, id_opportunity_parent... CachedPlan: 33792 total in 6 blocks; 14192 free (0 chunks); 19600 used: INSERT INTO opportunities.tab_opportunities ( id_event, id_oppty_type, o... CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog.... CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used CachedPlan: 32768 total in 6 blocks; 9352 free (0 chunks); 23416 used: EXISTS ( SELECT FROM new_items AS ni JOIN old_items AS oi ON oi.id_opportu... CachedPlan: 8192 total in 4 blocks; 3120 free (1 chunks); 5072 used: OLD.oppty_name IS DISTINCT FROM NEW.oppty_name AND EXISTS ( SELECT FROM oppo... CachedPlan: 9216 total in 4 blocks; 1408 free (1 chunks); 7808 used: NEW.oppty_end_date + NEW.oppty_end_time > NOW() AND (COALESCE(OLD.oppty_end_date + OLD.oppty... index info: 2048 total in 2 blocks; 392 free (0 chunks); 1656 used: ix_tabopptytree_idopportunityparent_temporary index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pk_tabopptytree_idopportunity index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopptytree_idopportunityparent index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: pg_statistic_relid_att_inh_index index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: ix_tabopportunities_opptybetradarid_iddataprovider index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptyevaltimetz index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptyendtimestamptz index info: 2048 total in 2 blocks; 184 free (0 chunks); 1864 used: ix_tabopportunities_opptyenddatetime index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idconstestant4 index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idconstestant3 index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptyexternalcompleteid index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptybetradarid index info: 4096 total in 3 blocks; 2024 free (2 chunks); 2072 used: ix_tabopportunities_opptyenddatetime_idopptystate index info: 2048 total in 2 blocks; 392 free (0 chunks); 1656 used: ix_tabopportunities_opptyenddatetime_obsolete index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptyenddate index info: 2048 total in 2 blocks; 264 free (0 chunks); 1784 used: ix_tabopportunities_lastminute index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: ix_tabopportunities_info index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: ix_tabopportunities_idopptytype_vsportsid_idevent index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idopportunitytype index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: ix_tabopportunities_idopportunitystate_opptyenddate_idevent index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: ix_tabopportunities_idopportunitystate_opptyenddate index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idopportunitystate index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: ix_tabopportunities_idopportunity_opptyenddate_opptyinfonumber index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: ix_tabopportunities_idevent_opptyenddate index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idconstestant2 index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idconstestant1 index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: ix_tabopportunities_enddateinfonumber index info: 2048 total in 2 blocks; 360 free (0 chunks); 1688 used: idx_tabopportunities_opptyspecialvalues_season_id index info: 2048 total in 2 blocks; 360 free (0 chunks); 1688 used: idx_tabopportunities_opptyspecialvalues_match_id index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: pk_tabopportunities_idopportunity CachedPlan: 16384 total in 5 blocks; 6712 free (0 chunks); 9672 used: UPDATE opportunities.lst_oppty_numbers SET id_opportunity = _id_opportunity WHERE id_oppty_n... CachedPlan: 8192 total in 4 blocks; 3704 free (1 chunks); 4488 used: UPDATE opportunities.lst_oppty_numbers SET id_opportunity = NULL ... index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: pg_toast_2619_index index info: 2048 total in 2 blocks; 584 free (0 chunks); 1464 used: ix_lstopptynumbers_freeopptynumbers index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_lstopptynumbers_idopportunity 1455 more child contexts containing 6947040 total in 4141 blocks; 2312976 free (668 chunks); 4634064 used WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used MdSmgr: 16384 total in 2 blocks; 5624 free (20 chunks); 10760 used LOCALLOCK hash: 32768 total in 3 blocks; 12712 free (8 chunks); 20056 used plpgsql_check - profiler context: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used plpgsql_check function execution statistics: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used plpgsql_check function profiler local chunks: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used plpgsql_check function profiler local cache: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used plpgsql_check function cache: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used Rendezvous variable hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used PLpgSQL function hash: 40864 total in 3 blocks; 2616 free (0 chunks); 38248 used Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used ErrorContext: 8192 total in 1 blocks; 7928 free (8 chunks); 264 used Grand total: 14312808 bytes in 4752 blocks; 3920880 free (1043 chunks); 10391928 used I hope, this will help. Some additional information: -bash-4.2$ cat /proc/17048/status Name: postmaster Umask: 0077 State: S (sleeping) Tgid: 17048 Ngid: 0 Pid: 17048 PPid: 4476 TracerPid: 0 Uid: 26 26 26 26 Gid: 26 26 26 26 FDSize: 1024 Groups: 26 VmPeak: 12212748 kB VmSize: 12212284 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 9421940 kB VmRSS: 9421940 kB RssAnon: 1054764 kB RssFile: 15796 kB RssShmem: 8351380 kB VmData: 1055404 kB VmStk: 132 kB VmExe: 7816 kB VmLib: 85312 kB VmPTE: 23264 kB VmSwap: 0 kB Threads: 1 SigQ: 0/160197 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000400000 SigIgn: 0000000001701800 SigCgt: 00000001a0006287 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000001fffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp: 0 Speculation_Store_Bypass: thread vulnerable Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 4748002 nonvoluntary_ctxt_switches: 12237 -bash-4.2$ pmap 17048 17048: postgres: db-prematch: app_feed prematch 172.25.0.231(47890) idle 0000000000400000 7816K r-x-- postgres 0000000000da1000 4K r---- postgres 0000000000da2000 68K rw--- postgres 0000000000db3000 200K rw--- [ anon ] 0000000002d15000 504K rw--- [ anon ] 0000000002d93000 838092K rw--- [ anon ] 00007fd999777000 180232K rw--- [ anon ] 00007fd9a8d75000 32772K rw--- [ anon ] 00007fd9ac5d2000 1024K rw-s- PostgreSQL.380355184 00007fd9ac6d2000 196K rw-s- PostgreSQL.38763112 00007fd9ac703000 148K r-x-- libtinfo.so.5.9 00007fd9ac728000 2048K ----- libtinfo.so.5.9 00007fd9ac928000 16K r---- libtinfo.so.5.9 00007fd9ac92c000 4K rw--- libtinfo.so.5.9 00007fd9ac92d000 212K r-x-- libedit.so.0.0.42 00007fd9ac962000 2044K ----- libedit.so.0.0.42 00007fd9acb61000 12K r---- libedit.so.0.0.42 00007fd9acb64000 4K rw--- libedit.so.0.0.42 00007fd9acb65000 20K rw--- [ anon ] 00007fd9acb6a000 28K r-x-- libffi.so.6.0.1 00007fd9acb71000 2044K ----- libffi.so.6.0.1 00007fd9acd70000 4K r---- libffi.so.6.0.1 00007fd9acd71000 4K rw--- libffi.so.6.0.1 00007fd9acd72000 45952K r-x-- libLLVM-5.0.so 00007fd9afa52000 2048K ----- libLLVM-5.0.so 00007fd9afc52000 3072K r---- libLLVM-5.0.so 00007fd9aff52000 444K rw--- libLLVM-5.0.so 00007fd9affc1000 372K rw--- [ anon ] 00007fd9b001e000 116K r-x-- llvmjit.so 00007fd9b003b000 2048K ----- llvmjit.so 00007fd9b023b000 4K r---- llvmjit.so 00007fd9b023c000 4K rw--- llvmjit.so 00007fd9b023d000 1028K rw--- [ anon ] 00007fd9b033e000 48K r-x-- libnss_files-2.17.so 00007fd9b034a000 2044K ----- libnss_files-2.17.so 00007fd9b0549000 4K r---- libnss_files-2.17.so 00007fd9b054a000 4K rw--- libnss_files-2.17.so 00007fd9b054b000 24K rw--- [ anon ] 00007fd9b0551000 10827040K rw-s- zero (deleted) 00007fdc45299000 32K r-x-- libprotobuf-c.so.1 00007fdc452a1000 2044K ----- libprotobuf-c.so.1 00007fdc454a0000 4K r---- libprotobuf-c.so.1 00007fdc454a1000 4K rw--- libprotobuf-c.so.1 00007fdc454a2000 24K r-x-- decoderbufs.so 00007fdc454a8000 2048K ----- decoderbufs.so 00007fdc456a8000 4K r---- decoderbufs.so 00007fdc456a9000 4K rw--- decoderbufs.so 00007fdc456aa000 184K r-x-- plpgsql.so 00007fdc456d8000 2044K ----- plpgsql.so 00007fdc458d7000 8K r---- plpgsql.so 00007fdc458d9000 4K rw--- plpgsql.so 00007fdc458da000 192K r-x-- plpgsql_check.so 00007fdc4590a000 2044K ----- plpgsql_check.so 00007fdc45b09000 4K r---- plpgsql_check.so 00007fdc45b0a000 4K rw--- plpgsql_check.so 00007fdc45b0b000 24K r-x-- pg_wait_sampling.so 00007fdc45b11000 2044K ----- pg_wait_sampling.so 00007fdc45d10000 4K r---- pg_wait_sampling.so 00007fdc45d11000 4K rw--- pg_wait_sampling.so 00007fdc45d12000 20K r-x-- pg_stat_kcache.so 00007fdc45d17000 2044K ----- pg_stat_kcache.so 00007fdc45f16000 4K r---- pg_stat_kcache.so 00007fdc45f17000 4K rw--- pg_stat_kcache.so 00007fdc45f18000 16K rw--- [ anon ] 00007fdc45f1c000 32K r-x-- pg_qualstats.so 00007fdc45f24000 2044K ----- pg_qualstats.so 00007fdc46123000 4K r---- pg_qualstats.so 00007fdc46124000 4K rw--- pg_qualstats.so 00007fdc46125000 32K r-x-- pg_stat_statements.so 00007fdc4612d000 2044K ----- pg_stat_statements.so 00007fdc4632c000 4K r---- pg_stat_statements.so 00007fdc4632d000 4K rw--- pg_stat_statements.so 00007fdc4632e000 103692K r---- locale-archive 00007fdc4c871000 8K r-x-- libfreebl3.so 00007fdc4c873000 2044K ----- libfreebl3.so 00007fdc4ca72000 4K r---- libfreebl3.so 00007fdc4ca73000 4K rw--- libfreebl3.so 00007fdc4ca74000 60K r-x-- libbz2.so.1.0.6 00007fdc4ca83000 2044K ----- libbz2.so.1.0.6 00007fdc4cc82000 4K r---- libbz2.so.1.0.6 00007fdc4cc83000 4K rw--- libbz2.so.1.0.6 00007fdc4cc84000 92K r-x-- libelf-0.176.so 00007fdc4cc9b000 2044K ----- libelf-0.176.so 00007fdc4ce9a000 4K r---- libelf-0.176.so 00007fdc4ce9b000 4K rw--- libelf-0.176.so 00007fdc4ce9c000 384K r-x-- libpcre.so.1.2.0 00007fdc4cefc000 2048K ----- libpcre.so.1.2.0 00007fdc4d0fc000 4K r---- libpcre.so.1.2.0 00007fdc4d0fd000 4K rw--- libpcre.so.1.2.0 00007fdc4d0fe000 16K r-x-- libattr.so.1.1.0 00007fdc4d102000 2044K ----- libattr.so.1.1.0 00007fdc4d301000 4K r---- libattr.so.1.1.0 00007fdc4d302000 4K rw--- libattr.so.1.1.0 00007fdc4d303000 32K r-x-- libcrypt-2.17.so 00007fdc4d30b000 2044K ----- libcrypt-2.17.so 00007fdc4d50a000 4K r---- libcrypt-2.17.so 00007fdc4d50b000 4K rw--- libcrypt-2.17.so 00007fdc4d50c000 184K rw--- [ anon ] 00007fdc4d53a000 16K r-x-- libcap-ng.so.0.0.0 00007fdc4d53e000 2048K ----- libcap-ng.so.0.0.0 00007fdc4d73e000 4K r---- libcap-ng.so.0.0.0 00007fdc4d73f000 4K rw--- libcap-ng.so.0.0.0 00007fdc4d740000 312K r-x-- libdw-0.176.so 00007fdc4d78e000 2048K ----- libdw-0.176.so 00007fdc4d98e000 8K r---- libdw-0.176.so 00007fdc4d990000 4K rw--- libdw-0.176.so 00007fdc4d991000 16K r-x-- libgpg-error.so.0.10.0 00007fdc4d995000 2044K ----- libgpg-error.so.0.10.0 00007fdc4db94000 4K r---- libgpg-error.so.0.10.0 00007fdc4db95000 4K rw--- libgpg-error.so.0.10.0 00007fdc4db96000 500K r-x-- libgcrypt.so.11.8.2 00007fdc4dc13000 2044K ----- libgcrypt.so.11.8.2 00007fdc4de12000 4K r---- libgcrypt.so.11.8.2 00007fdc4de13000 12K rw--- libgcrypt.so.11.8.2 00007fdc4de16000 4K rw--- [ anon ] 00007fdc4de17000 144K r-x-- libselinux.so.1 00007fdc4de3b000 2044K ----- libselinux.so.1 00007fdc4e03a000 4K r---- libselinux.so.1 00007fdc4e03b000 4K rw--- libselinux.so.1 00007fdc4e03c000 8K rw--- [ anon ] 00007fdc4e03e000 16K r-x-- libcap.so.2.22 00007fdc4e042000 2044K ----- libcap.so.2.22 00007fdc4e241000 4K r---- libcap.so.2.22 00007fdc4e242000 4K rw--- libcap.so.2.22 00007fdc4e243000 20296K r-x-- libicudata.so.50.2 00007fdc4f615000 2044K ----- libicudata.so.50.2 00007fdc4f814000 4K r---- libicudata.so.50.2 00007fdc4f815000 4K rw--- libicudata.so.50.2 00007fdc4f816000 84K r-x-- libgcc_s-4.8.5-20150702.so.1 00007fdc4f82b000 2044K ----- libgcc_s-4.8.5-20150702.so.1 00007fdc4fa2a000 4K r---- libgcc_s-4.8.5-20150702.so.1 00007fdc4fa2b000 4K rw--- libgcc_s-4.8.5-20150702.so.1 00007fdc4fa2c000 932K r-x-- libstdc++.so.6.0.19 00007fdc4fb15000 2044K ----- libstdc++.so.6.0.19 00007fdc4fd14000 32K r---- libstdc++.so.6.0.19 00007fdc4fd1c000 8K rw--- libstdc++.so.6.0.19 00007fdc4fd1e000 84K rw--- [ anon ] 00007fdc4fd33000 232K r-x-- libnspr4.so 00007fdc4fd6d000 2044K ----- libnspr4.so 00007fdc4ff6c000 4K r---- libnspr4.so 00007fdc4ff6d000 8K rw--- libnspr4.so 00007fdc4ff6f000 8K rw--- [ anon ] 00007fdc4ff71000 16K r-x-- libplc4.so 00007fdc4ff75000 2044K ----- libplc4.so 00007fdc50174000 4K r---- libplc4.so 00007fdc50175000 4K rw--- libplc4.so 00007fdc50176000 12K r-x-- libplds4.so 00007fdc50179000 2044K ----- libplds4.so 00007fdc50378000 4K r---- libplds4.so 00007fdc50379000 4K rw--- libplds4.so 00007fdc5037a000 164K r-x-- libnssutil3.so 00007fdc503a3000 2044K ----- libnssutil3.so 00007fdc505a2000 28K r---- libnssutil3.so 00007fdc505a9000 4K rw--- libnssutil3.so 00007fdc505aa000 1176K r-x-- libnss3.so 00007fdc506d0000 2048K ----- libnss3.so 00007fdc508d0000 20K r---- libnss3.so 00007fdc508d5000 8K rw--- libnss3.so 00007fdc508d7000 8K rw--- [ anon ] 00007fdc508d9000 148K r-x-- libsmime3.so 00007fdc508fe000 2044K ----- libsmime3.so 00007fdc50afd000 12K r---- libsmime3.so 00007fdc50b00000 4K rw--- libsmime3.so 00007fdc50b01000 332K r-x-- libssl3.so 00007fdc50b54000 2048K ----- libssl3.so 00007fdc50d54000 16K r---- libssl3.so 00007fdc50d58000 4K rw--- libssl3.so 00007fdc50d59000 4K rw--- [ anon ] 00007fdc50d5a000 112K r-x-- libsasl2.so.3.0.0 00007fdc50d76000 2044K ----- libsasl2.so.3.0.0 00007fdc50f75000 4K r---- libsasl2.so.3.0.0 00007fdc50f76000 4K rw--- libsasl2.so.3.0.0 00007fdc50f77000 56K r-x-- liblber-2.4.so.2.10.7 00007fdc50f85000 2044K ----- liblber-2.4.so.2.10.7 00007fdc51184000 4K r---- liblber-2.4.so.2.10.7 00007fdc51185000 4K rw--- liblber-2.4.so.2.10.7 00007fdc51186000 88K r-x-- libresolv-2.17.so 00007fdc5119c000 2048K ----- libresolv-2.17.so 00007fdc5139c000 4K r---- libresolv-2.17.so 00007fdc5139d000 4K rw--- libresolv-2.17.so 00007fdc5139e000 8K rw--- [ anon ] 00007fdc513a0000 12K r-x-- libkeyutils.so.1.5 00007fdc513a3000 2044K ----- libkeyutils.so.1.5 00007fdc515a2000 4K r---- libkeyutils.so.1.5 00007fdc515a3000 4K rw--- libkeyutils.so.1.5 00007fdc515a4000 56K r-x-- libkrb5support.so.0.1 00007fdc515b2000 2048K ----- libkrb5support.so.0.1 00007fdc517b2000 4K r---- libkrb5support.so.0.1 00007fdc517b3000 4K rw--- libkrb5support.so.0.1 00007fdc517b4000 196K r-x-- libk5crypto.so.3.1 00007fdc517e5000 2044K ----- libk5crypto.so.3.1 00007fdc519e4000 8K r---- libk5crypto.so.3.1 00007fdc519e6000 4K rw--- libk5crypto.so.3.1 00007fdc519e7000 12K r-x-- libcom_err.so.2.1 00007fdc519ea000 2044K ----- libcom_err.so.2.1 00007fdc51be9000 4K r---- libcom_err.so.2.1 00007fdc51bea000 4K rw--- libcom_err.so.2.1 00007fdc51beb000 868K r-x-- libkrb5.so.3.3 00007fdc51cc4000 2044K ----- libkrb5.so.3.3 00007fdc51ec3000 56K r---- libkrb5.so.3.3 00007fdc51ed1000 12K rw--- libkrb5.so.3.3 00007fdc51ed4000 120K r-x-- libaudit.so.1.0.0 00007fdc51ef2000 2044K ----- libaudit.so.1.0.0 00007fdc520f1000 4K r---- libaudit.so.1.0.0 00007fdc520f2000 4K rw--- libaudit.so.1.0.0 00007fdc520f3000 40K rw--- [ anon ] 00007fdc520fd000 148K r-x-- liblzma.so.5.2.2 00007fdc52122000 2044K ----- liblzma.so.5.2.2 00007fdc52321000 4K r---- liblzma.so.5.2.2 00007fdc52322000 4K rw--- liblzma.so.5.2.2 00007fdc52323000 84K r-x-- libz.so.1.2.7 00007fdc52338000 2044K ----- libz.so.1.2.7 00007fdc52537000 4K r---- libz.so.1.2.7 00007fdc52538000 4K rw--- libz.so.1.2.7 00007fdc52539000 1804K r-x-- libc-2.17.so 00007fdc526fc000 2048K ----- libc-2.17.so 00007fdc528fc000 16K r---- libc-2.17.so 00007fdc52900000 8K rw--- libc-2.17.so 00007fdc52902000 20K rw--- [ anon ] 00007fdc52907000 188K r-x-- libsystemd.so.0.6.0 00007fdc52936000 2048K ----- libsystemd.so.0.6.0 00007fdc52b36000 4K r---- libsystemd.so.0.6.0 00007fdc52b37000 4K rw--- libsystemd.so.0.6.0 00007fdc52b38000 1424K r-x-- libicuuc.so.50.2 00007fdc52c9c000 2048K ----- libicuuc.so.50.2 00007fdc52e9c000 64K r---- libicuuc.so.50.2 00007fdc52eac000 4K rw--- libicuuc.so.50.2 00007fdc52ead000 16K rw--- [ anon ] 00007fdc52eb1000 1984K r-x-- libicui18n.so.50.2 00007fdc530a1000 2048K ----- libicui18n.so.50.2 00007fdc532a1000 48K r---- libicui18n.so.50.2 00007fdc532ad000 8K rw--- libicui18n.so.50.2 00007fdc532af000 4K rw--- [ anon ] 00007fdc532b0000 328K r-x-- libldap-2.4.so.2.10.7 00007fdc53302000 2048K ----- libldap-2.4.so.2.10.7 00007fdc53502000 8K r---- libldap-2.4.so.2.10.7 00007fdc53504000 4K rw--- libldap-2.4.so.2.10.7 00007fdc53505000 1028K r-x-- libm-2.17.so 00007fdc53606000 2044K ----- libm-2.17.so 00007fdc53805000 4K r---- libm-2.17.so 00007fdc53806000 4K rw--- libm-2.17.so 00007fdc53807000 8K r-x-- libdl-2.17.so 00007fdc53809000 2048K ----- libdl-2.17.so 00007fdc53a09000 4K r---- libdl-2.17.so 00007fdc53a0a000 4K rw--- libdl-2.17.so 00007fdc53a0b000 28K r-x-- librt-2.17.so 00007fdc53a12000 2044K ----- librt-2.17.so 00007fdc53c11000 4K r---- librt-2.17.so 00007fdc53c12000 4K rw--- librt-2.17.so 00007fdc53c13000 92K r-x-- libpthread-2.17.so 00007fdc53c2a000 2044K ----- libpthread-2.17.so 00007fdc53e29000 4K r---- libpthread-2.17.so 00007fdc53e2a000 4K rw--- libpthread-2.17.so 00007fdc53e2b000 16K rw--- [ anon ] 00007fdc53e2f000 296K r-x-- libgssapi_krb5.so.2.2 00007fdc53e79000 2048K ----- libgssapi_krb5.so.2.2 00007fdc54079000 4K r---- libgssapi_krb5.so.2.2 00007fdc5407a000 8K rw--- libgssapi_krb5.so.2.2 00007fdc5407c000 2264K r-x-- libcrypto.so.1.0.2k 00007fdc542b2000 2048K ----- libcrypto.so.1.0.2k 00007fdc544b2000 112K r---- libcrypto.so.1.0.2k 00007fdc544ce000 52K rw--- libcrypto.so.1.0.2k 00007fdc544db000 16K rw--- [ anon ] 00007fdc544df000 412K r-x-- libssl.so.1.0.2k 00007fdc54546000 2048K ----- libssl.so.1.0.2k 00007fdc54746000 16K r---- libssl.so.1.0.2k 00007fdc5474a000 28K rw--- libssl.so.1.0.2k 00007fdc54751000 52K r-x-- libpam.so.0.83.1 00007fdc5475e000 2048K ----- libpam.so.0.83.1 00007fdc5495e000 4K r---- libpam.so.0.83.1 00007fdc5495f000 4K rw--- libpam.so.0.83.1 00007fdc54960000 1404K r-x-- libxml2.so.2.9.1 00007fdc54abf000 2044K ----- libxml2.so.2.9.1 00007fdc54cbe000 32K r---- libxml2.so.2.9.1 00007fdc54cc6000 8K rw--- libxml2.so.2.9.1 00007fdc54cc8000 8K rw--- [ anon ] 00007fdc54cca000 80K r-x-- liblz4.so.1.7.5 00007fdc54cde000 2044K ----- liblz4.so.1.7.5 00007fdc54edd000 4K r---- liblz4.so.1.7.5 00007fdc54ede000 4K rw--- liblz4.so.1.7.5 00007fdc54edf000 136K r-x-- ld-2.17.so 00007fdc54f1a000 1588K rw--- [ anon ] 00007fdc550a7000 212K rw-s- PostgreSQL.4009629762 00007fdc550dc000 104K rw--- [ anon ] 00007fdc550fc000 8K r-x-- [ anon ] 00007fdc550fe000 4K rw-s- [ shmid=0x67418000 ] 00007fdc550ff000 4K rw--- [ anon ] 00007fdc55100000 4K r---- ld-2.17.so 00007fdc55101000 4K rw--- ld-2.17.so 00007fdc55102000 4K rw--- [ anon ] 00007fffd382b000 132K rw--- [ stack ] 00007fffd38f0000 8K r-x-- [ anon ] ffffffffff600000 4K r-x-- [ anon ] total 12212288K -bash-4.2$ smem -u User Count Swap USS PSS RSS postgres 326 0 10503276 20969859 175615624 -bash-4.2$ smem ^C PID User Command Swap USS PSS RSS 27250 postgres postgres: db-prematch: arch 0 224 246 2592 27251 postgres postgres: db-prematch: stat 0 512 538 2716 27253 postgres postgres: db-prematch: logi 0 548 567 3480 27249 postgres postgres: db-prematch: auto 0 664 696 3956 27248 postgres postgres: db-prematch: walw 0 244 878 19296 ...27246 postgres postgres: db-prematch: chec 0 88308 851490 10271612 12156 postgres postgres: db-prematch: app_ 0 902776 1400234 9066116 8303 postgres postgres: db-prematch: app_ 0 913768 1411286 9067356 16469 postgres postgres: db-prematch: app_ 0 923668 1417847 9071168 31187 postgres postgres: db-prematch: app_ 0 923024 1425936 9196732 13769 postgres postgres: db-prematch: app_ 0 925820 1437147 9283344 26144 postgres postgres: db-prematch: app_ 0 937440 1437802 9180720 15449 postgres postgres: db-prematch: app_ 0 932776 1440671 9262056 22570 postgres postgres: db-prematch: app_ 0 938680 1447851 9292176 2910 postgres postgres: db-prematch: app_ 0 1039452 1536980 9183668 17048 postgres postgres: db-prematch: app_ 0 1054244 1564436 9425556 -bash-4.2$ smem -w Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 18242380 16741684 1500696 userspace memory 21358228 10746068 10612160 free memory 1433148 1433148 0 -bash-4.2$ smem -m Map PIDs AVGPSS PSS /dev/shm/PostgreSQL.4009629762 316 0 0 /usr/lib64/gconv/gconv-modules.cache 18 0 0 /usr/lib64/libattr.so.1.1.0 318 0 0 ... /usr/pgsql-14/lib/libpq.so.5.14 15 27 415 /usr/lib64/libkrb5.so.3.3 319 1 445 /usr/pgsql-14/lib/pg_qualstats.so 318 1 572 /usr/pgsql-14/lib/pg_stat_kcache.so 318 1 572 /usr/lib64/libpython2.7.so.1.0 1 945 945 /usr/pgsql-14/lib/pg_stat_statements.so 318 3 1212 /usr/lib64/libpthread-2.17.so 320 4 1304 /usr/lib64/ld-2.17.so 322 4 1373 /usr/pgsql-14/lib/plpgsql.so 318 4 1376 /usr/lib64/libcrypto.so.1.0.2k 319 4 1499 /usr/lib64/libc-2.17.so 322 9 3179 [stack] 322 26 8498 /usr/pgsql-14/bin/postgres 318 57 18281 /usr/lib64/llvm5.0/lib/libLLVM-5.0.so 10 4098 40984 <anonymous> 322 6557 2111438 [heap] 322 25093 8080218 /dev/zero 316 33834 10691660 Kind Regards Ales Zeleny > > If the RssAnon memory is a good indicator, i can then determine the > backends and dump memory context. > It'll take me some time since I'm out of office for vacation, but I'll > manage that somewhat way. > > Thanks for all to the hints! > > Aleš > >> >> regards >> >> -- >> Tomas Vondra >> EnterpriseDB: http://www.enterprisedb.com >> The Enterprise PostgreSQL Company >> >
