On 2020-07-09 02:03, Andres Freund wrote:
Hi,
I think this is an incredibly useful feature.
Thanks for your kind comments and suggestion!
On 2020-07-07 22:02:10 +0900, torikoshia wrote:
> There can be multiple memory contexts with the same name. So I'm afraid
> that it's difficult to identify the actual parent memory context from
> this
> "parent" column. This is ok when logging memory contexts by calling
> MemoryContextStats() via gdb. Because child memory contexts are printed
> just under their parent, with indents. But this doesn't work in the
> view.
> To identify the actual parent memory or calculate the memory contexts
> tree
> from the view, we might need to assign unique ID to each memory context
> and display it. But IMO this is overkill. So I'm fine with current
> "parent"
> column. Thought? Do you have any better idea?
Indeed.
I also feel it's not usual to assign a unique ID, which
can vary every time the view displayed.
Hm. I wonder if we just could include the address of the context
itself. There might be reasons not to do so (e.g. security concerns
about leaked pointers making attacks easier), but I think it's worth
considering.
I tried exposing addresses of each context and their parent.
Attached a poc patch.
=# SELECT name, address, parent_address, total_bytes FROM
pg_backend_memory_contexts ;
name | address | parent_address | total_bytes
--------------------------+-----------+----------------+-------------
TopMemoryContext | 0x1280da0 | | 80800
TopTransactionContext | 0x1309040 | 0x1280da0 | 8192
Prepared Queries | 0x138a480 | 0x1280da0 | 16384
Type information cache | 0x134b8c0 | 0x1280da0 | 24624
...
CacheMemoryContext | 0x12cb390 | 0x1280da0 | 1048576
CachedPlanSource | 0x13c47f0 | 0x12cb390 | 4096
CachedPlanQuery | 0x13c9ae0 | 0x13c47f0 | 4096
CachedPlanSource | 0x13c7310 | 0x12cb390 | 4096
CachedPlanQuery | 0x13c1230 | 0x13c7310 | 4096
...
Now it's possible to identify the actual parent memory context even when
there are multiple memory contexts with the same name.
I'm not sure, but I'm also worrying about this might incur some security
related problems..
I'd like to hear more opinions about:
- whether information for identifying parent-child relation is necessary
or it's an overkill
- if this information is necessary, memory address is suitable or other
means like assigning unique numbers are required
+/*
+ * PutMemoryContextsStatsTupleStore
+ * One recursion level for pg_get_backend_memory_contexts.
+ */
+static void
+PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
+ TupleDesc
tupdesc, MemoryContext context,
+ MemoryContext
parent, int level)
+{
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ MemoryContextCounters stat;
+ MemoryContext child;
+ const char *name = context->name;
+ const char *ident = context->ident;
+
+ if (context == NULL)
+ return;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts
+ * with just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (ident && strcmp(name, "dynahash") == 0)
+ {
+ name = ident;
+ ident = NULL;
+ }
+
+ /* Examine the context itself */
+ memset(&stat, 0, sizeof(stat));
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (ident)
+ {
+ int idlen = strlen(ident);
+ char
clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+
Why?
As described below[1], too long messages caused problems in the past and
now
MemoryContextStatsPrint() truncates ident, so I decided to truncate it
also
here.
Do you think it's not necessary here?
[1] https://www.postgresql.org/message-id/12319.1521999...@sss.pgh.pa.us
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
From 055af903a3dbf146d97dd3fb01a6a7d3d3bd2ae0 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikos...@oss.nttdata.com>
Date: Fri, 10 Jul 2020 17:01:01 +0900
Subject: [PATCH] Add a function exposing memory usage of local backend.
This patch implements a new SQL-callable function
pg_get_backend_memory_contexts which exposes memory usage of the
local backend.
It also adds a new view pg_backend_memory_contexts for exposing
local backend memory contexts.
---
doc/src/sgml/catalogs.sgml | 131 +++++++++++++++++++++++++
src/backend/catalog/system_views.sql | 3 +
src/backend/utils/mmgr/mcxt.c | 141 ++++++++++++++++++++++++++-
src/include/catalog/pg_proc.dat | 9 ++
src/test/regress/expected/rules.out | 10 ++
5 files changed, 293 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index e9cdff4864..b84a06ee7d 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9249,6 +9249,11 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>materialized views</entry>
</row>
+ <row>
+ <entry><link linkend="view-pg-backend-memory-contexts"><structname>pg_backend_memory_contexts</structname></link></entry>
+ <entry>backend memory contexts</entry>
+ </row>
+
<row>
<entry><link linkend="view-pg-policies"><structname>pg_policies</structname></link></entry>
<entry>policies</entry>
@@ -10527,6 +10532,132 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-backend-memory-contexts">
+ <title><structname>pg_backend_memory_contexts</structname></title>
+
+ <indexterm zone="view-pg-backend-memory-contexts">
+ <primary>pg_backend_memory_contexts</primary>
+ </indexterm>
+
+ <para>
+ The view <structname>pg_backend_memory_contexts</structname> displays all
+ the local backend memory contexts.
+ </para>
+ <para>
+ <structname>pg_backend_memory_contexts</structname> contains one row
+ for each memory context.
+ </para>
+
+ <table>
+ <title><structname>pg_backend_memory_contexts</structname> Columns</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the memory context
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>ident</structfield> <type>text</type>
+ </para>
+ <para>
+ Identification information of the memory context. This field is truncated at 1024 bytes
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>parent</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the parent of this memory context
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>address</structfield> <type>text</type>
+ </para>
+ <para>
+ Memory address of this memory context
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>parent_address</structfield> <type>text</type>
+ </para>
+ <para>
+ Memory address of the parent of this memory context
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>total_bytes</structfield> <type>int8</type>
+ </para>
+ <para>
+ Total bytes allocated for this memory context
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>total_nblocks</structfield> <type>int8</type>
+ </para>
+ <para>
+ Total number of blocks allocated for this memory context
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>free_bytes</structfield> <type>int8</type>
+ </para>
+ <para>
+ Free space in bytes
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>free_chunks</structfield> <type>int8</type>
+ </para>
+ <para>
+ Total number of free chunks
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>used_bytes</structfield> <type>int8</type>
+ </para>
+ <para>
+ Used space in bytes
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect1>
+
<sect1 id="view-pg-matviews">
<title><structname>pg_matviews</structname></title>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 73676d04cf..f39189e19d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -554,6 +554,9 @@ CREATE VIEW pg_shmem_allocations AS
REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
+CREATE VIEW pg_backend_memory_contexts AS
+ SELECT * FROM pg_get_backend_memory_contexts();
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index abda22fa57..850ad188a7 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -21,12 +21,13 @@
#include "postgres.h"
+#include "funcapi.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/builtins.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
-
/*****************************************************************************
* GLOBAL MEMORY *
*****************************************************************************/
@@ -67,6 +68,12 @@ static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
#define AssertNotInCriticalSection(context) \
Assert(CritSectionCount == 0 || (context)->allowInCritSection)
+/* ----------
+ * The max bytes for showing identifiers of MemoryContext.
+ * ----------
+ */
+#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+
/*****************************************************************************
* EXPORTED ROUTINES *
*****************************************************************************/
@@ -1220,3 +1227,135 @@ pchomp(const char *in)
n--;
return pnstrdup(in, n);
}
+
+/*
+ * PutMemoryContextsStatsTupleStore
+ * One recursion level for pg_get_backend_memory_contexts.
+ */
+static void
+PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
+ TupleDesc tupdesc, MemoryContext context,
+ MemoryContext parent, int level)
+{
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ MemoryContextCounters stat;
+ MemoryContext child;
+ const char *name = context->name;
+ const char *ident = context->ident;
+ char context_addr[64];
+
+ if (context == NULL)
+ return;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts
+ * with just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (ident && strcmp(name, "dynahash") == 0)
+ {
+ name = ident;
+ ident = NULL;
+ }
+
+ /* Examine the context itself */
+ memset(&stat, 0, sizeof(stat));
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (ident)
+ {
+ int idlen = strlen(ident);
+ char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+
+ memcpy(clipped_ident, ident, idlen);
+ clipped_ident[idlen] = '\0';
+ values[1] = CStringGetTextDatum(clipped_ident);
+ }
+ else
+ nulls[1] = true;
+
+ sprintf(context_addr, "%p", context);
+ values[2] = CStringGetTextDatum(context_addr);
+
+ if (parent == NULL)
+ nulls[3] = true;
+ else
+ {
+ char parent_context_addr[64];
+
+ sprintf(parent_context_addr, "%p", parent);
+ values[3] = CStringGetTextDatum(parent_context_addr);
+ }
+
+ values[4] = Int64GetDatum(stat.totalspace);
+ values[5] = Int64GetDatum(stat.nblocks);
+ values[6] = Int64GetDatum(stat.freespace);
+ values[7] = Int64GetDatum(stat.freechunks);
+ values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ for (child = context->firstchild; child != NULL; child = child->nextchild)
+ {
+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
+ child, context, level + 1);
+ }
+}
+
+/*
+ * pg_get_backend_memory_contexts
+ * SQL SRF showing backend memory context.
+ */
+Datum
+pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
+ TopMemoryContext, NULL, 0);
+
+ /* clean up and return the tuplestore */
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d951b4a36f..edc8d9ec05 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7797,6 +7797,15 @@
proargnames => '{name,off,size,allocated_size}',
prosrc => 'pg_get_shmem_allocations' },
+# memory context of local backend
+{ oid => '2282', descr => 'information about all memory contexts of local backend',
+ proname => 'pg_get_backend_memory_contexts', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,text,text,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o}',
+ proargnames => '{name, ident, address, parent_address, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ prosrc => 'pg_get_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 93bb2159ca..3f96fedae4 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1324,6 +1324,16 @@ pg_available_extensions| SELECT e.name,
e.comment
FROM (pg_available_extensions() e(name, default_version, comment)
LEFT JOIN pg_extension x ON ((e.name = x.extname)));
+pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,
+ pg_get_backend_memory_contexts.ident,
+ pg_get_backend_memory_contexts.address,
+ pg_get_backend_memory_contexts.parent_address,
+ pg_get_backend_memory_contexts.total_bytes,
+ pg_get_backend_memory_contexts.total_nblocks,
+ pg_get_backend_memory_contexts.free_bytes,
+ pg_get_backend_memory_contexts.free_chunks,
+ pg_get_backend_memory_contexts.used_bytes
+ FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, address, parent_address, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
--
2.18.1
