Hello,
we are experiencing out-of-memory issues after Postygres upgrade from 14.3
to 14.4 on CentOS7.
OS: CenotOS7
No OOM killer messages in the syslog. SWAP is disabled.
-bash-4.2$ sysctl -q vm.overcommit_memory
vm.overcommit_memory = 2
[root@sts-uat-pgsql100 ~]# swapon -vs
[root@sts-uat-pgsql100 ~]#
[root@sts-uat-pgsql100 ~]# sysctl -q vm.overcommit_ratio
vm.overcommit_ratio = 95
-bash-4.2$ uname -r
3.10.0-1160.42.2.el7.x86_64
-bash-4.2$ free -h
total used free shared buff/cache
available
Mem: 39G 7,8G 220M 10G 31G
20G
Swap: 0B 0B 0B
postgres=# show shared_buffers ;
shared_buffers
----------------
10GB
(1 row)
postgres=# show work_mem ;
work_mem
----------
4MB
(1 row)
postgres=# show maintenance_work_mem ;
maintenance_work_mem
----------------------
512MB
(1 row)
postgres=# show max_parallel_workers;
max_parallel_workers
----------------------
8
(1 row)
postgres=# show max_parallel_maintenance_workers ;
max_parallel_maintenance_workers
----------------------------------
2
(1 row)
postgres=# select count(*) from pg_stat_activity ;
count
-------
93
(1 row)
postgres=# show max_connections ;
max_connections
-----------------
1000
(1 row)
No changes on the application side were made before the out-of-memory
happened.
So far, it has happened three times (during a single week) from the 14.3 ->
14.4 upgrade, before 14.4 we haven't suffered from such an issue.
Questions:
1) Can we safely downgrade from 14.4 to 14.3 by shutting down the instance
and reinstalling 14.3 PG packages (to prove, that the issue disappear)?
2) What is the best way to diagnose what is the root cause?
What we have started to collect:
- vmstat -nwt (per second)
- vmstat -nawt (per second)
~onece a minute
df for /dev/shm
/proc/meminfo
proc/<pid>/stat for all PG processes
proc/<pid>/cmdline for all PG processes
proc/<pid>/status for all PG processes
proc/<pid>/smaps for all PG processes
list from pg_stat_activcity
pmap -x for all PG processes
ps aux listing
smem -u --abbreviate
smem -m --abbreviate
smem -w -k
Except the "pmpap" all of the metrics were already collected before the
last out of meory issue.
Error messages:
----------------------
postgresql-Fri.log:2022-07-01 06:49:53 CEST [4476]: [20-1]
user=,db=,host=,app= LOG: could not fork worker process: Cannot allocate
memory
postgresql-Fri.log:2022-07-01 06:49:53 CEST [4476]: [21-1]
user=,db=,host=,app= LOG: could not fork worker process: Cannot allocate
memory
postgresql-Fri.log:2022-07-01 06:50:02 CEST [4476]: [22-1]
user=,db=,host=,app= LOG: could not fork new process for connection:
Cannot allocate memory
postgresql-Fri.log:2022-07-01 06:50:02 CEST [4476]: [23-1]
user=,db=,host=,app= LOG: could not fork new process for connection:
Cannot allocate memory
postgresql-Fri.log:2022-07-01 06:50:02 CEST [4476]: [24-1]
user=,db=,host=,app= LOG: could not fork new process for connection:
Cannot allocate memory
...
2022-07-02 14:48:07 CEST [3930]: [2-1] user=,db=,host=,app= LOG: automatic
vacuum of table "prematch.monitoring.tab_replication_clock": index scans: 0
pages: 0 removed, 36 remain, 0 skipped due to pins, 34 skipped
frozen
tuples: 0 removed, 203 remain, 201 are dead but not yet removable,
oldest xmin: 269822444
index scan not needed: 0 pages from table (0.00% of total) had 0
dead item identifiers removed
I/O timings: read: 0.000 ms, write: 0.000 ms
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s
buffer usage: 67 hits, 0 misses, 0 dirtied
WAL usage: 1 records, 0 full page images, 245 bytes
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
TopMemoryContext: 279360 total in 10 blocks; 15384 free (9 chunks); 263976
used
TopTransactionContext: 8192 total in 1 blocks; 7280 free (12 chunks); 912
used
TOAST to main relid map: 65536 total in 4 blocks; 35168 free (15 chunks);
30368 used
AV worker: 16384 total in 2 blocks; 10080 free (10 chunks); 6304 used
Autovacuum Portal: 8192 total in 1 blocks; 7720 free (0 chunks); 472
used
Vacuum: 8192 total in 1 blocks; 7768 free (0 chunks); 424 used
Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640
used
smgr relation table: 16384 total in 2 blocks; 4592 free (2 chunks); 11792
used
pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1448
free (0 chunks); 6744 used
TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks);
264 used
Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
TopPortalContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
Relcache by OID: 16384 total in 2 blocks; 3504 free (2 chunks); 12880 used
CacheMemoryContext: 524288 total in 7 blocks; 123552 free (4 chunks);
400736 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
tab_queue_tmp_pkey
index info: 2048 total in 2 blocks; 952 free (0 chunks); 1096 used:
pk_replicationclock_databasename
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used:
pg_attrdef_adrelid_adnum_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_index_indrelid_index
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used:
pg_db_role_setting_databaseid_rol_index
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_opclass_am_name_nsp_index
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_foreign_data_wrapper_name_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_enum_oid_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_class_relname_nsp_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_foreign_server_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_publication_pubname_index
index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used:
pg_statistic_relid_att_inh_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_cast_source_target_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_language_name_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_transform_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_collation_oid_index
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used:
pg_amop_fam_strat_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_index_indexrelid_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_ts_template_tmplname_index
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_ts_config_map_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_opclass_oid_index
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_foreign_data_wrapper_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_event_trigger_evtname_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_statistic_ext_name_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_publication_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_ts_dict_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_event_trigger_oid_index
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used:
pg_conversion_default_index
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used:
pg_operator_oprname_l_r_n_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_trigger_tgrelid_tgname_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_enum_typid_label_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_ts_config_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_user_mapping_oid_index
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_opfamily_am_name_nsp_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_foreign_table_relid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_type_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_aggregate_fnoid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_constraint_oid_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_rewrite_rel_rulename_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_ts_parser_prsname_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_ts_config_cfgname_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_ts_parser_oid_index
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_publication_rel_prrelid_prpubid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_operator_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_namespace_nspname_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_ts_template_oid_index
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_amop_opr_fam_index
index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used:
pg_default_acl_role_nsp_obj_index
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_collation_name_enc_nsp_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_publication_rel_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_range_rngtypid_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_ts_dict_dictname_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_type_typname_nsp_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_opfamily_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_statistic_ext_oid_index
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_statistic_ext_data_stxoid_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_class_oid_index
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_proc_proname_args_nsp_index
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_partitioned_table_partrelid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_range_rngmultitypid_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_transform_type_lang_index
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used:
pg_attribute_relid_attnum_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_proc_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_language_oid_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_namespace_oid_index
index info: 3072 total in 2 blocks; 776 free (1 chunks); 2296 used:
pg_amproc_fam_proc_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_foreign_server_name_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_attribute_relid_attnam_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_conversion_oid_index
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_user_mapping_user_server_index
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_subscription_rel_srrelid_srsubid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_sequence_seqrelid_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_conversion_name_nsp_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_authid_oid_index
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_auth_members_member_role_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_subscription_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_tablespace_oid_index
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_shseclabel_object_index
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_replication_origin_roname_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_database_datname_index
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_subscription_subname_index
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_replication_origin_roiident_index
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_auth_members_role_member_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_database_oid_index
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_authid_rolname_index
WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks);
43408 used
PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
MdSmgr: 8192 total in 1 blocks; 7808 free (2 chunks); 384 used
LOCALLOCK hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
plpgsql_check - profiler context: 8192 total in 1 blocks; 7928 free (0
chunks); 264 used
plpgsql_check function execution statistics: 8192 total in 1 blocks;
2488 free (0 chunks); 5704 used
plpgsql_check function profiler local chunks: 8192 total in 1 blocks;
2488 free (0 chunks); 5704 used
plpgsql_check function profiler local cache: 8192 total in 1 blocks;
2488 free (0 chunks); 5704 used
plpgsql_check function cache: 8192 total in 1 blocks; 2616 free (0
chunks); 5576 used
Rendezvous variable hash: 8192 total in 1 blocks; 552 free (0 chunks);
7640 used
PLpgSQL function hash: 8192 total in 1 blocks; 2616 free (0 chunks); 5576
used
Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
Postmaster: 24576 total in 2 blocks; 23824 free (18 chunks); 752 used
ident parser context: 1024 total in 1 blocks; 760 free (0 chunks); 264
used
hba parser context: 25600 total in 6 blocks; 4464 free (8 chunks);
21136 used
ErrorContext: 8192 total in 1 blocks; 7928 free (5 chunks); 264 used
Grand total: 1456224 bytes in 195 blocks; 378824 free (165 chunks); 1077400
used
2022-07-02 14:48:07 CEST [3930]: [3-1] user=,db=,host=,app= ERROR: out of
memory
2022-07-02 14:48:07 CEST [3930]: [4-1] user=,db=,host=,app= DETAIL: Failed
on request of size 152094068 in memory context "TopTransactionContext".
2022-07-02 14:48:07 CEST [3930]: [5-1] user=,db=,host=,app= CONTEXT:
automatic vacuum of table "prematch.replication.tab_queue_tmp"
2022-07-02 14:48:47 CEST [4476]: [42-1] user=,db=,host=,app= LOG: could
not fork worker process: Cannot allocate memory
2022-07-02 14:48:47 CEST [4476]: [43-1] user=,db=,host=,app= LOG: could
not fork worker process: Cannot allocate memory
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
2022-07-02 14:48:47 CEST [4476]: [44-1] user=,db=,host=,app= LOG:
background worker "parallel worker" (PID 4303) was terminated by signal 6:
Aborted
A day later, the same issue happens again:
2022-07-02 14:48:07 CEST [3930]: [2-1] user=,db=,host=,app= LOG: automatic
vacuum of table "prematch.monitoring.tab_replication_clock": index scans: 0
11 pages: 0 removed, 36 remain, 0 skipped due to pins, 34 skipped
frozen
12 tuples: 0 removed, 203 remain, 201 are dead but not yet
removable, oldest xmin: 269822444
13 index scan not needed: 0 pages from table (0.00% of total) had 0
dead item identifiers removed
14 I/O timings: read: 0.000 ms, write: 0.000 ms
15 avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s
16 buffer usage: 67 hits, 0 misses, 0 dirtied
17 WAL usage: 1 records, 0 full page images, 245 bytes
18 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
19TopMemoryContext: 279360 total in 10 blocks; 15384 free (9 chunks);
263976 used
20 TopTransactionContext: 8192 total in 1 blocks; 7280 free (12 chunks);
912 used
21 TOAST to main relid map: 65536 total in 4 blocks; 35168 free (15
chunks); 30368 used
22 AV worker: 16384 total in 2 blocks; 10080 free (10 chunks); 6304 used
23 Autovacuum Portal: 8192 total in 1 blocks; 7720 free (0 chunks); 472
used
24 Vacuum: 8192 total in 1 blocks; 7768 free (0 chunks); 424 used
25 Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640
used
26 smgr relation table: 16384 total in 2 blocks; 4592 free (2 chunks);
11792 used
27 pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1448
free (0 chunks); 6744 used
28 TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0
chunks); 264 used
29 Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
30 TopPortalContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
31 Relcache by OID: 16384 total in 2 blocks; 3504 free (2 chunks); 12880
used
32 CacheMemoryContext: 524288 total in 7 blocks; 123552 free (4 chunks);
400736 used
33 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
tab_queue_tmp_pkey
34 index info: 2048 total in 2 blocks; 952 free (0 chunks); 1096 used:
pk_replicationclock_databasename
35 index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used:
pg_attrdef_adrelid_adnum_index
36 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_index_indrelid_index
37 index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used:
pg_db_role_setting_databaseid_rol_index
38 index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_opclass_am_name_nsp_index
39 index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_foreign_data_wrapper_name_index
40 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_enum_oid_index
41 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_class_relname_nsp_index
42 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_foreign_server_oid_index
43 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_publication_pubname_index
44 index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used:
pg_statistic_relid_att_inh_index
45 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_cast_source_target_index
46 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_language_name_index
47 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_transform_oid_index
48 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_collation_oid_index
49 index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used:
pg_amop_fam_strat_index
50 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_index_indexrelid_index
51 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_ts_template_tmplname_index
52 index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_ts_config_map_index
53 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_opclass_oid_index
54 index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_foreign_data_wrapper_oid_index
55 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_event_trigger_evtname_index
56 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_statistic_ext_name_index
57 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_publication_oid_index
58 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_ts_dict_oid_index
59 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_event_trigger_oid_index
60 index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used:
pg_conversion_default_index
61 index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used:
pg_operator_oprname_l_r_n_index
62 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_trigger_tgrelid_tgname_index
63 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_enum_typid_label_index
64 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_ts_config_oid_index
65 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_user_mapping_oid_index
66 index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_opfamily_am_name_nsp_index
67 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_foreign_table_relid_index
68 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_type_oid_index
69 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_aggregate_fnoid_index
70 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_constraint_oid_index
71 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_rewrite_rel_rulename_index
72 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_ts_parser_prsname_index
73 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_ts_config_cfgname_index
74 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_ts_parser_oid_index
75 index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_publication_rel_prrelid_prpubid_index
76 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_operator_oid_index
77 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_namespace_nspname_index
78 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_ts_template_oid_index
79 index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_amop_opr_fam_index
80 index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used:
pg_default_acl_role_nsp_obj_index
81 index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_collation_name_enc_nsp_index
82 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_publication_rel_oid_index
83 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_range_rngtypid_index
84 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_ts_dict_dictname_index
85 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_type_typname_nsp_index
86 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_opfamily_oid_index
87 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_statistic_ext_oid_index
88 index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_statistic_ext_data_stxoid_index
89 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_class_oid_index
90 index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_proc_proname_args_nsp_index
91 index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_partitioned_table_partrelid_index
92 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_range_rngmultitypid_index
93 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_transform_type_lang_index
94 index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used:
pg_attribute_relid_attnum_index
95 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_proc_oid_index
96 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_language_oid_index
97 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_namespace_oid_index
98 index info: 3072 total in 2 blocks; 776 free (1 chunks); 2296 used:
pg_amproc_fam_proc_index
99 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_foreign_server_name_index
100 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_attribute_relid_attnam_index
101 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_conversion_oid_index
102 index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_user_mapping_user_server_index
103 index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_subscription_rel_srrelid_srsubid_index
104 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_sequence_seqrelid_index
105 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_conversion_name_nsp_index
106 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_authid_oid_index
107 index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_auth_members_member_role_index
108 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_subscription_oid_index
109 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_tablespace_oid_index
110 index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used:
pg_shseclabel_object_index
111 index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_replication_origin_roname_index
112 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_database_datname_index
113 index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used:
pg_subscription_subname_index
114 index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used:
pg_replication_origin_roiident_index
115 index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used:
pg_auth_members_role_member_index
116 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used:
pg_database_oid_index
117 index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used:
pg_authid_rolname_index
118 WAL record construction: 49768 total in 2 blocks; 6360 free (0
chunks); 43408 used
119 PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576
used
120 MdSmgr: 8192 total in 1 blocks; 7808 free (2 chunks); 384 used
121 LOCALLOCK hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
122 plpgsql_check - profiler context: 8192 total in 1 blocks; 7928 free (0
chunks); 264 used
123 plpgsql_check function execution statistics: 8192 total in 1 blocks;
2488 free (0 chunks); 5704 used
124 plpgsql_check function profiler local chunks: 8192 total in 1
blocks; 2488 free (0 chunks); 5704 used
125 plpgsql_check function profiler local cache: 8192 total in 1 blocks;
2488 free (0 chunks); 5704 used
126 plpgsql_check function cache: 8192 total in 1 blocks; 2616 free (0
chunks); 5576 used
127 Rendezvous variable hash: 8192 total in 1 blocks; 552 free (0 chunks);
7640 used
128 PLpgSQL function hash: 8192 total in 1 blocks; 2616 free (0 chunks);
5576 used
129 Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
130 Postmaster: 24576 total in 2 blocks; 23824 free (18 chunks); 752 used
131 ident parser context: 1024 total in 1 blocks; 760 free (0 chunks);
264 used
132 hba parser context: 25600 total in 6 blocks; 4464 free (8 chunks);
21136 used
133 ErrorContext: 8192 total in 1 blocks; 7928 free (5 chunks); 264 used
134Grand total: 1456224 bytes in 195 blocks; 378824 free (165 chunks);
1077400 used
1352022-07-02 14:48:07 CEST [3930]: [3-1] user=,db=,host=,app= ERROR: out
of memory
1362022-07-02 14:48:07 CEST [3930]: [4-1] user=,db=,host=,app= DETAIL:
Failed on request of size 152094068 in memory context
"TopTransactionContext".
1372022-07-02 14:48:07 CEST [3930]: [5-1] user=,db=,host=,app= CONTEXT:
automatic vacuum of table "prematch.replication.tab_queue_tmp"
And a few days later again:
2022-07-08 14:54:22 CEST [4791]: [81186-1]
user=app_evaluation,db=prematch,host=172.25.0.80,app=Evaluation_STS_UAT
DETAIL: parameters: $1 = '1', $2 = '1748010445', $3 = '0', $4 = '1000'
terminate
called after throwing an instance of 'std::bad_alloc' terminate called
after throwing an instance of 'std::bad_alloc' what(): what():
std::bad_allocstd::bad_alloc 2022-07-08 14:54:23 CEST [4476]: [49-1]
user=,db=,host=,app= LOG: background worker "parallel worker" (PID 25251)
was terminated by signal 6: Aborted
2022-07-08 14:54:23 CEST [4476]: [51-1] user=,db=,host=,app= LOG:
terminating any other active server processes
Thanks Ales Zeleny
