On 2020-12-04 19:16, torikoshia wrote:
On 2020-12-03 10:36, Tom Lane wrote:
Fujii Masao <masao.fu...@oss.nttdata.com> writes:
I'm starting to study how this feature behaves. At first, when I
executed
the following query, the function never returned. ISTM that since
the autovacuum launcher cannot respond to the request of memory
contexts dump, the function keeps waiting infinity. Is this a bug?
Probably we should exclude non-backend proceses from the target
processes to dump? Sorry if this was already discussed.
SELECT pg_get_backend_memory_contexts(pid) FROM
pg_stat_activity;
Thanks for trying it!
It was not discussed explicitly, and I was going to do it later
as commented.
+ /* TODO: Check also whether backend or not. */
FWIW, I think this patch is fundamentally unsafe. It's got a
lot of the same problems that I complained about w.r.t. the
nearby proposal to allow cross-backend stack trace dumping.
It does avoid the trap of thinking that it can do work in
a signal handler, but instead it supposes that it can do
work involving very high-level objects such as shared hash tables
in anyplace that might execute CHECK_FOR_INTERRUPTS. That's
never going to be safe: the only real expectation the system
has is that CHECK_FOR_INTERRUPTS is called at places where our
state is sane enough that a transaction abort can clean up.
Trying to do things like taking LWLocks is going to lead to
deadlocks or worse. We need not even get into the hard questions,
such as what happens when one process or the other exits
unexpectedly.
Thanks for reviewing!
I may misunderstand something, but the dumper works not at
CHECK_FOR_INTERRUPTS but during the client read, i.e.,
ProcessClientReadInterrupt().
Is it also unsafe?
BTW, since there was a comment that the shared hash table
used too much memory, I'm now rewriting this patch not to use
the shared hash table but a simpler static shared memory struct.
Attached a rewritten patch.
Accordingly, I also slightly modified the basic design as below.
---
# Communication flow between the dumper and the requester
- (1) When requesting memory context dumping, the dumper changes
the struct on the shared memory state from 'ACCEPTABLE' to
'REQUESTING'.
- (2) The dumper sends the signal to the dumper process and wait on
the latch.
- (3) When the dumper noticed the signal, it changes the state to
'DUMPING'.
- (4) When the dumper completes dumping, it changes the state to
'DONE' and set the latch.
- (5) The requestor reads the dump file and shows it to the user.
Finally, the requestor removes the dump file and reset the shared
memory state to 'ACCEPTABLE'.
# Query cancellation
- When the requestor cancels dumping, e.g. signaling using ctrl-C,
the requestor changes the state of the shared memory to 'CANCELING'.
- The dumper checks the state when it tries to change the state to
'DONE' at (4), and if the state is 'CANCELING', it initializes the
dump file and reset the shared memory state to 'ACCEPTABLE'.
# Cleanup dump file and the shared memory
- In the normal case, the dumper removes the dump file and resets
the shared memory entry as described in (5).
- When something like query cancellation or process termination
happens on the dumper after (1) and before (3), in other words,
the state is 'REQUESTING', the requestor does the cleanup.
- When something happens on the dumper or the requestor after (3)
and before (4), in other words, the state is 'DUMPING', the dumper
does the cleanup. Specifically, if the requestor cancels the query,
it just changes the state to 'CANCELING' and the dumper notices it
and cleans up things later.
OTOH, when the dumper fails to dump, it cleans up the dump file and
reset the shared memory state.
- When something happens on the requestor after (4), i.e., the state
is 'DONE', the requestor does the cleanup.
- In the case of receiving SIGKILL or power failure, all dump files
are removed in the crash recovery process.
---
I also find the idea that this should be the same SQL function
as pg_get_backend_memory_contexts to be a seriously bad decision.
That means that it's not possible to GRANT the right to examine
only your own process's memory --- with this proposal, that means
granting the right to inspect every other process as well.
Beyond that, the fact that there's no way to restrict the capability
to just, say, other processes owned by the same user means that
it's not really safe to GRANT to non-superusers anyway. Even with
such a restriction added, things are problematic, since for example
it would be possible to inquire into the workings of a
security-definer function executing in another process that
nominally is owned by your user.
I'm going to change the function name and restrict the executor to
superusers. Is it enough?
In the attached patch, I changed the function name to
pg_get_target_backend_memory_contexts() for now.
Regards,
From 120b2c33e2b295305dedc63921183c103e65ddc6 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikos...@oss.nttdata.com>
Date: Thu, 9 Dec 2020 09:52:15 +0900
Subject: [PATCH v5] Add pg_get_target_backend_memory_contexts(() to collect
arbitrary backend process's memory contexts.
After commit 3e98c0bafb28de, we can display the usage of the
memory contexts using pg_backend_memory_contexts system
view. However, its target is limited to the process attached
to the current session.
This patch introduces pg_get_target_backend_memory_contexts()
and makes it possible to collect memory contexts of the
specified process.
---
src/backend/access/transam/xlog.c | 7 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 691 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 12 +-
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 44 ++
18 files changed, 770 insertions(+), 21 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 7e81ce4f17..2474a36ac1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -73,6 +73,7 @@
#include "storage/sync.h"
#include "utils/builtins.h"
#include "utils/guc.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/relmapper.h"
@@ -6982,6 +6983,12 @@ StartupXLOG(void)
*/
pgstat_reset_all();
+ /*
+ * Reset dump files in pg_memusage, because target processes do
+ * not exist any more.
+ */
+ RemoveMemcxtFile(0);
+
/*
* If there was a backup label file, it's done its job and the info
* has now been propagated into pg_control. We must get rid of the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b140c210bc..77e97f4805 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -558,7 +558,8 @@ CREATE VIEW pg_backend_memory_contexts AS
SELECT * FROM pg_get_backend_memory_contexts();
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_target_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 7c75a25d21..6bd471af2f 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4042,6 +4042,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_XACT_GROUP_UPDATE:
event_name = "XactGroupUpdate";
break;
+ case WAIT_EVENT_DUMP_MEMORY_CONTEXT:
+ event_name = "DumpMemoryContext";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 1d8d1742a7..c65a1b4f66 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dump files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 96c2aaabbd..92f21ad2bf 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -45,6 +45,7 @@
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "storage/spin.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
/* GUCs */
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtDumpShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index ffe67acea1..2b1ddd3a0d 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMCXT))
+ HandleProcSignalDumpMemoryContext();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 774292fd94..6b4ff6f08b 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+McxtDumpLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3679799e50..9d7f2458c0 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -75,6 +75,7 @@
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
#include "utils/lsyscache.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -539,6 +540,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryContextPending)
+ ProcessDumpMemoryContextInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 50e1b07ff0..d2ffc88c37 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,30 +15,101 @@
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/logging.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "pgstat.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
+
+/* The max bytes for showing names and identifiers of MemoryContext. */
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+/* Number of columns in pg_backend_memory_contexts view */
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-/* ----------
- * The max bytes for showing identifiers of MemoryContext.
- * ----------
+/* Shared memory struct for managing the status of memory context dump. */
+static mcxtdumpShmemStruct *mcxtdumpShmem = NULL;
+
+
+/*
+ * McxtReqKill
+ * Error cleanup callback for memory context requestor.
*/
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+static void
+McxtReqKill(int code, Datum arg)
+{
+ int dst_pid = DatumGetInt32(arg);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * If the requestor is not me, simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING)
+ {
+ /*
+ * Since the dumper has not received the dump order yet, clean up
+ * things by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING)
+ {
+ /*
+ * Since the dumper has received the request already,
+ * requestor just change the status and the dumper cleans up
+ * things later.
+ */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_CANCELING;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /*
+ * Since the dumper has already finished dumping, clean up things
+ * by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ RemoveMemcxtFile(dst_pid);
+ }
+ LWLockRelease(McxtDumpLock);
+}
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
@@ -74,14 +145,12 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
@@ -101,18 +170,198 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Since pg_get_backend_memory_contexts() is called from local process,
+ * simply put tuples.
+ */
+ if(fpout == NULL)
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Write out the current memory context information in the form of
+ * "key: value" pairs to the file specified by the requestor.
+ */
+ else
+ {
+ /*
+ * Make each memory context information starts with 'D'.
+ * This is checked by the requestor when reading the file.
+ */
+ fputc('D', fpout);
+
+ fprintf(fpout,
+ "name: %s, ident: %s, parent: %s, level: %d, total_bytes: %lu, \
+ total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name,
+ ident ? clipped_ident : "none",
+ parent ? parent : "none", level,
+ stat.totalspace,
+ stat.nblocks,
+ stat.freespace,
+ stat.freechunks,
+ stat.totalspace - stat.freespace);
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
}
}
+/*
+ * SetupMcxtdumpShem
+ * Setup shared memory struct for dumping specified PID.
+ */
+static void
+SetupMcxtdumpShmem(int pid)
+{
+ /*
+ * We only allow one session per target process to request memory
+ * contexts dump at a time.
+ * If someone uses mcxtdumpShmem, wait until it was changed to acceptable.
+ */
+ while (true)
+ {
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING;
+ mcxtdumpShmem->src_pid = MyProcPid;
+ mcxtdumpShmem->dst_pid = pid;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+ else
+ {
+ ereport(INFO,
+ (errmsg("PID %d is looked up by another process", pid)));
+
+ LWLockRelease(McxtDumpLock);
+
+ pg_usleep(5000000L);
+ }
+ }
+}
+
+/*
+ * PutDumpedValuesOnTuplestore
+ * Read specified memory context dump file and put its values
+ * on the tuplestore.
+ */
+static void
+PutDumpedValuesOnTuplestore(char *dumpfile, Tuplestorestate *tupstore,
+ TupleDesc tupdesc, int pid)
+{
+ FILE *fpin;
+ int format_id;
+
+ if ((fpin = AllocateFile(dumpfile, "r")) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open memory context dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ /* Verify it's of the expected format. */
+ if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ format_id != PG_MCXT_FILE_FORMAT_ID)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"", dumpfile)));
+ goto done;
+ }
+
+ /* Read dump file and put values on tuple store. */
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ Size total_bytes;
+ Size total_nblocks;
+ Size free_bytes;
+ Size free_chunks;
+ Size used_bytes;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows */
+ case 'D':
+ if (fscanf(fpin, "name: %1023[^,], ident: %1023[^,], parent: %1023[^,], \
+ level: %d, total_bytes: %lu, total_nblocks: %lu, \
+ free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name, clipped_ident, parent, &level, &total_bytes, &total_nblocks,
+ &free_bytes, &free_chunks, &used_bytes)
+ != PG_GET_BACKEND_MEMORY_CONTEXTS_COLS)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (strcmp(clipped_ident, "none"))
+ values[1] = CStringGetTextDatum(clipped_ident);
+ else
+ nulls[1] = true;
+
+ if (strcmp(parent, "none"))
+ values[2] = CStringGetTextDatum(parent);
+ else
+ nulls[2] = true;
+
+ values[3] = Int32GetDatum(level);
+ values[4] = Int64GetDatum(total_bytes);
+ values[5] = Int64GetDatum(total_nblocks);
+ values[6] = Int64GetDatum(free_bytes);
+ values[7] = Int64GetDatum(free_chunks);
+ values[8] = Int64GetDatum(used_bytes);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ }
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ Assert(mcxtdumpShmem->src_pid == MyProcPid);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+}
+
/*
* pg_get_backend_memory_contexts
- * SQL SRF showing backend memory context.
+ * SQL SRF showing local backend memory context.
*/
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
@@ -148,10 +397,422 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ TopMemoryContext, "", 0, NULL);
+
+ /* clean up and return the tuplestore */
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
+
+/*
+ * pg_get_target_backend_memory_contexts
+ * SQL SRF showing specified process's backend memory context.
+ */
+Datum
+pg_get_target_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int dst_pid = PG_GETARG_INT32(0);
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ char dumpfile[MAXPGPATH];
+ PGPROC *proc;
+ PgBackendStatus *beentry;
+
+ if (dst_pid == MyProcPid)
+ {
+ pg_get_backend_memory_contexts(fcinfo);
+ return (Datum) 0;
+ }
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, dst_pid);
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ * By the time we reach SendProcSignal(), a process for which we get
+ * a valid proc here might have terminated or got stuck.
+ * In these cases, users are expected to cancel the dump request.
+ */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", dst_pid)));
+
+ return (Datum) 1;
+ }
+
+ beentry = pgstat_fetch_stat_beentry(proc->backendId);
+
+ if (beentry->st_backendType != B_BACKEND)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL backend process", dst_pid)));
+
+ return (Datum) 1;
+ }
+
+ /*
+ * The ENSURE stuff ensures we clean up the shared memory struct and files
+ * on failure.
+ */
+ PG_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+ {
+ SetupMcxtdumpShmem(dst_pid);
+
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMCXT, InvalidBackendId);
+
+ /* Wait until target process finishes dumping file. */
+ for (;;)
+ {
+ /* Check for dump cancel request. */
+ CHECK_FOR_INTERRUPTS();
+
+ /* Must reset the latch before testing state. */
+ ResetLatch(MyLatch);
+
+ LWLockAcquire(McxtDumpLock, LW_SHARED);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * It seems the dumper fails and subsequently another
+ * process is requesting dumping.
+ */
+ ereport(INFO,
+ (errmsg("The request has failed and now PID %d is requsting dumping.",
+ mcxtdumpShmem->src_pid)));
+
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * Dumper seems to clean up the enry because of failures or
+ * cancellation.
+ * Since the dumper has already removed the dump file,
+ * simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * Request has been canceled. Exit without dumping.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /* Dumping has completed. */
+ LWLockRelease(McxtDumpLock);
+ break;
+ }
+ /*
+ * The dumper must be in the middle of a dumping or the request
+ * hasn't been reached yet.
+ */
+ Assert(mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING ||
+ mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING);
+
+ LWLockRelease(McxtDumpLock);
+
+ /*
+ * Wait. We expect to get a latch signal back from the dumper.
+ * Use a timeout to enable cancellation.
+ */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 1000L, WAIT_EVENT_DUMP_MEMORY_CONTEXT);
+ }
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+
+ /* Read values from the dump file and put them on tuplestore. */
+ PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dump local memory contexts to a file.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char dumpfile[MAXPGPATH];
+ int format_id;
+ pid_t src_pid;
+ PGPROC *src_proc;
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, MyProcPid);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * The requestor canceled the request and initialized
+ * the shared memory. Simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * The requestor canceled the request.
+ * Initialize the shared memory and exit.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ Assert(mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING);
+
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DUMPING;
+ src_pid = mcxtdumpShmem->src_pid;
+
+ LWLockRelease(McxtDumpLock);
+
+ fpout = AllocateFile(dumpfile, "w");
+
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write memory context file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+ format_id = PG_MCXT_FILE_FORMAT_ID;
+ fwrite(&format_id, sizeof(format_id), 1, fpout);
+
+ /* Look into each memory context from TopMemoryContext recursively. */
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, NULL, 0, fpout);
+
+ /*
+ * Make dump file ends with 'E'.
+ * This is checked by the requestor later.
+ */
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* No more output to be done. Close file. */
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /* During dumping, the requestor canceled the request. */
+ unlink(dumpfile);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* Dumping has succeeded, notify it to the requestor. */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DONE;
+ LWLockRelease(McxtDumpLock);
+ src_proc = BackendPidGetProc(src_pid);
+ SetLatch(&(src_proc->procLatch));
+
+ return;
+}
+
+/*
+ * ProcessDumpMemoryContextInterrupt
+ * The portion of memory context dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryContextInterrupt(void)
+{
+ ProcSignalDumpMemoryContextPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * HandleProcSignalDumpMemoryContext
+ * Handle receipt of an interrupt indicating a memory context dump.
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemoryContext(void)
+{
+ ProcSignalDumpMemoryContextPending = true;
+}
+
+/*
+ * McxtDumpShmemInit
+ * Initialize mcxtdump shared memory struct.
+ */
+void
+McxtDumpShmemInit(void)
+{
+ bool found;
+
+ mcxtdumpShmem = (mcxtdumpShmemStruct *)
+ ShmemInitStruct("Memory Context Dump Data",
+ sizeof(mcxtdumpShmemStruct),
+ &found);
+ if (!found)
+ {
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+}
+
+/*
+ * RemoveMemcxtFile
+ * Remove dump files.
+ */
+void
+RemoveMemcxtFile(int pid)
+{
+ DIR *dir;
+ struct dirent *dumpfile;
+
+ if (pid == 0)
+ {
+ /* delete all dump files */
+ dir = AllocateDir(PG_MEMUSAGE_DIR);
+ while ((dumpfile = ReadDir(dir, PG_MEMUSAGE_DIR)) != NULL)
+ {
+ char dumpfilepath[32];
+
+ if (strcmp(dumpfile->d_name, ".") == 0 || strcmp(dumpfile->d_name, "..") == 0)
+ continue;
+
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, dumpfile->d_name);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ FreeDir(dir);
+ }
+ else
+ {
+ /* delete specified dump file */
+ char str_pid[12];
+ char dumpfilepath[32];
+ struct stat stat_tmp;
+
+ pg_ltoa(pid, str_pid);
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, str_pid);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (stat(dumpfilepath, &stat_tmp) == 0)
+ {
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ab8216839..288eb8bc0c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -33,6 +33,7 @@ volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index ee3bfa82f4..52cdb26272 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -221,7 +221,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..340a80fc11 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 110;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index ba34dbac14..c4a79528ac 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -119,6 +119,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fc2202b843..70604faefc 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7844,15 +7844,25 @@
# memory context of local backend
{ oid => '2282',
- descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 'r',
prorettype => 'record', proargtypes => '',
proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
proargmodes => '{o,o,o,o,o,o,o,o,o}',
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ descr => 'information about all memory contexts of local backend',
prosrc => 'pg_get_backend_memory_contexts' },
+# memory context of specified backend
+{ oid => '2860',
+ descr => 'information about all memory contexts of specified backend',
+ proname => 'pg_get_target_backend_memory_contexts', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'r', prorettype => 'record',
+ proargtypes => 'int4', proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ prosrc => 'pg_get_target_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..1ff03b86a0 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -83,6 +83,7 @@ extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5954068dec..262adbb782 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -964,7 +964,8 @@ typedef enum
WAIT_EVENT_REPLICATION_SLOT_DROP,
WAIT_EVENT_SAFE_SNAPSHOT,
WAIT_EVENT_SYNC_REP,
- WAIT_EVENT_XACT_GROUP_UPDATE
+ WAIT_EVENT_XACT_GROUP_UPDATE,
+ WAIT_EVENT_DUMP_MEMORY_CONTEXT
} WaitEventIPC;
/* ----------
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 5cb39697f3..d4a7ae0761 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMCXT, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..eff2c62cbb
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+/* Directory to store dumped memory files */
+#define PG_MEMUSAGE_DIR "pg_memusage"
+
+#define PG_MCXT_FILE_FORMAT_ID 0x01B5BC9E
+
+typedef enum McxtDumpStatus
+{
+ MCXTDUMPSTATUS_ACCEPTABLE, /* no one is requesting dumping */
+ MCXTDUMPSTATUS_REQUESTING, /* request has been issued, but dumper has not received it yet */
+ MCXTDUMPSTATUS_DUMPING, /* dumper is dumping files */
+ MCXTDUMPSTATUS_DONE, /* dumper has finished dumping and the requestor is working */
+ MCXTDUMPSTATUS_CANCELING /* requestor canceled the dumping */
+} McxtDumpStatus;
+
+typedef struct mcxtdumpShmemStruct
+{
+ pid_t dst_pid; /* pid of the signal receiver */
+ pid_t src_pid; /* pid of the signal sender */
+ McxtDumpStatus dump_status; /* dump status */
+} mcxtdumpShmemStruct;
+
+extern void ProcessDumpMemoryContextInterrupt(void);
+extern void HandleProcSignalDumpMemoryContext(void);
+extern void McxtDumpShmemInit(void);
+extern void RemoveMcxtDumpFile(int);
+extern void RemoveMemcxtFile(int);
+#endif /* MCXT_H */
--
2.18.1