On Wed, Dec 19, 2018 at 4:17 AM REIX, Tony <tony.r...@atos.net> wrote: > Here is the patch we are using now on AIX for enabling SysV shm for AIX, > which improves greatly the performance on AIX. > > It is compile time. > > It seems to me that you'd like this to become a shared_memory_type GUC. > Correct? However, I do not know how to do. > > > Even as-is, this patch would greatly improve the performance of PostgreSQL > v11.1 in the field on AIX machines. So, we'd like this change to be available > for AIX asap. > > > What are the next steps to get this patch accepted? or What are your > suggestions for improving it?
Hi Tony, Since it's not fixing a bug, we wouldn't back-patch that into existing releases. But I agree that we should do something like this for PostgreSQL 12, and I think we should make it user configurable. Here is a quick rebase of Andres's shared_memory_type patch for master, so that you can put shared_memory_type=sysv in postgresql.conf to get the old pre-9.3 behaviour (this may also be useful for other operating systems). Here also is a "blind" patch that makes it respect huge_pages=try/on on AIX (or at least, I think it does; I don't have an AIX to try it, it almost certainly needs some adjustments). Thoughts? -- Thomas Munro http://www.enterprisedb.com
From 1d3d78b7bcf1e2be939e8a5b9cc09e5b9f679e2b Mon Sep 17 00:00:00 2001 From: Thomas Munro <thomas.munro@enterprisedb.com> Date: Wed, 26 Dec 2018 12:03:51 +1300 Subject: [PATCH 1/2] Add shared_memory_type GUC. Since 9.3 we have used anonymous shared mmap for our main shmem segment, except in EXEC_BACKEND builds. Provide a GUC so that users can opt into System V shared memory once again. This is reported to be useful at least on AIX systems. Future commits will add huge/large page support for System V memory. Author: Andres Freund (revived and documented by Thomas Munro) Discussion: https://postgr.es/m/HE1PR0202MB28126DB4E0B6621CC6A1A91286D90%40HE1PR0202MB2812.eurprd02.prod.outlook.com Discussion: https://postgr.es/m/2AE143D2-87D3-4AD1-AC78-CE2258230C05%40FreeBSD.org --- doc/src/sgml/config.sgml | 24 +++++++++++ src/backend/port/sysv_shmem.c | 42 +++++++------------ src/backend/storage/ipc/ipci.c | 2 + src/backend/utils/misc/guc.c | 23 ++++++++++ src/backend/utils/misc/postgresql.conf.sample | 6 +++ src/include/storage/pg_shmem.h | 20 +++++++++ 6 files changed, 91 insertions(+), 26 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index e94b305add..78d1c654d7 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -1694,6 +1694,30 @@ include_dir 'conf.d' </listitem> </varlistentry> + <varlistentry id="guc-shared-memory-type" xreflabel="shared_memory_type"> + <term><varname>shared_memory_type</varname> (<type>enum</type>) + <indexterm> + <primary><varname>shared_memory_type</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Specifies the shared memory implementation that the server + should use for the main shared memory region used for + <productname>PostgreSQL</productname>'s buffer pool and other core facilities. + Possible values are <literal>mmap</literal> (for anonymous shared + memory allocated using <literal>mmap</literal>), <literal>sysv</literal> + (for System V shared memory allocated via <literal>shmget</literal>) and + <literal>windows</literal> (for Windows shared memory). + Not all values are supported on all platforms; the first supported + option is the default for that platform. The use of the + <literal>sysv</literal> option, which is not the default on any platform, + is generally discouraged because it typically requires non-default + kernel settings to allow for large allocations. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-dynamic-shared-memory-type" xreflabel="dynamic_shared_memory_type"> <term><varname>dynamic_shared_memory_type</varname> (<type>enum</type>) <indexterm> diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c index 741c455ccb..c4aaa08c6a 100644 --- a/src/backend/port/sysv_shmem.c +++ b/src/backend/port/sysv_shmem.c @@ -62,10 +62,11 @@ * to a process after exec(). Since EXEC_BACKEND is intended only for * developer use, this shouldn't be a big problem. Because of this, we do * not worry about supporting anonymous shmem in the EXEC_BACKEND cases below. + * + * As of PostgreSQL 12, we regained the ability to use a large System V shared + * memory region even in non-EXEC_BACKEND builds, if shared_memory_type is set + * to sysv (though this is not the default). */ -#ifndef EXEC_BACKEND -#define USE_ANONYMOUS_SHMEM -#endif typedef key_t IpcMemoryKey; /* shared memory key passed to shmget(2) */ @@ -75,10 +76,8 @@ typedef int IpcMemoryId; /* shared memory ID returned by shmget(2) */ unsigned long UsedShmemSegID = 0; void *UsedShmemSegAddr = NULL; -#ifdef USE_ANONYMOUS_SHMEM static Size AnonymousShmemSize; static void *AnonymousShmem = NULL; -#endif static void *InternalIpcMemoryCreate(IpcMemoryKey memKey, Size size); static void IpcMemoryDetach(int status, Datum shmaddr); @@ -370,8 +369,6 @@ PGSharedMemoryIsInUse(unsigned long id1, unsigned long id2) return true; } -#ifdef USE_ANONYMOUS_SHMEM - #ifdef MAP_HUGETLB /* @@ -534,8 +531,6 @@ AnonymousShmemDetach(int status, Datum arg) } } -#endif /* USE_ANONYMOUS_SHMEM */ - /* * PGSharedMemoryCreate * @@ -566,7 +561,7 @@ PGSharedMemoryCreate(Size size, bool makePrivate, int port, Size sysvsize; /* Complain if hugepages demanded but we can't possibly support them */ -#if !defined(USE_ANONYMOUS_SHMEM) || !defined(MAP_HUGETLB) +#if !defined(MAP_HUGETLB) if (huge_pages == HUGE_PAGES_ON) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -576,18 +571,19 @@ PGSharedMemoryCreate(Size size, bool makePrivate, int port, /* Room for a header? */ Assert(size > MAXALIGN(sizeof(PGShmemHeader))); -#ifdef USE_ANONYMOUS_SHMEM - AnonymousShmem = CreateAnonymousSegment(&size); - AnonymousShmemSize = size; + if (shared_memory_type == SHMEM_TYPE_MMAP) + { + AnonymousShmem = CreateAnonymousSegment(&size); + AnonymousShmemSize = size; - /* Register on-exit routine to unmap the anonymous segment */ - on_shmem_exit(AnonymousShmemDetach, (Datum) 0); + /* Register on-exit routine to unmap the anonymous segment */ + on_shmem_exit(AnonymousShmemDetach, (Datum) 0); - /* Now we need only allocate a minimal-sized SysV shmem block. */ - sysvsize = sizeof(PGShmemHeader); -#else - sysvsize = size; -#endif + /* Now we need only allocate a minimal-sized SysV shmem block. */ + sysvsize = sizeof(PGShmemHeader); + } + else + sysvsize = size; /* Make sure PGSharedMemoryAttach doesn't fail without need */ UsedShmemSegAddr = NULL; @@ -687,14 +683,10 @@ PGSharedMemoryCreate(Size size, bool makePrivate, int port, * block. Otherwise, the System V shared memory block is only a shim, and * we must return a pointer to the real block. */ -#ifdef USE_ANONYMOUS_SHMEM if (AnonymousShmem == NULL) return hdr; memcpy(AnonymousShmem, hdr, sizeof(PGShmemHeader)); return (PGShmemHeader *) AnonymousShmem; -#else - return hdr; -#endif } #ifdef EXEC_BACKEND @@ -801,7 +793,6 @@ PGSharedMemoryDetach(void) UsedShmemSegAddr = NULL; } -#ifdef USE_ANONYMOUS_SHMEM if (AnonymousShmem != NULL) { if (munmap(AnonymousShmem, AnonymousShmemSize) < 0) @@ -809,7 +800,6 @@ PGSharedMemoryDetach(void) AnonymousShmem, AnonymousShmemSize); AnonymousShmem = NULL; } -#endif } diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 0c86a581c0..2e97ba6b2d 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -47,6 +47,8 @@ #include "utils/backend_random.h" #include "utils/snapmgr.h" +/* GUCs */ +int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE; shmem_startup_hook_type shmem_startup_hook = NULL; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 6fe1939881..6d9319dfa1 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -451,6 +451,19 @@ const struct config_enum_entry ssl_protocol_versions_info[] = { {NULL, 0, false} }; +static struct config_enum_entry shared_memory_options[] = { +#ifndef WIN32 + { "sysv", SHMEM_TYPE_SYSV, false}, +#endif +#ifndef EXEC_BACKEND + { "mmap", SHMEM_TYPE_MMAP, false}, +#endif +#ifdef WIN32 + { "windows", SHMEM_TYPE_WINDOWS, false}, +#endif + {NULL, 0, false} +}; + /* * Options for enum values stored in other modules */ @@ -4304,6 +4317,16 @@ static struct config_enum ConfigureNamesEnum[] = NULL, NULL, NULL }, + { + {"shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM, + gettext_noop("Selects the shared memory implementation used for the main shared memory region."), + NULL + }, + &shared_memory_type, + DEFAULT_SHARED_MEMORY_TYPE, shared_memory_options, + NULL, NULL, NULL + }, + { {"wal_sync_method", PGC_SIGHUP, WAL_SETTINGS, gettext_noop("Selects the method used for forcing WAL updates to disk."), diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 1fa02d2c93..6f9764ae21 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -129,6 +129,12 @@ #maintenance_work_mem = 64MB # min 1MB #autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem #max_stack_depth = 2MB # min 100kB +#shared_memory_type = mmap # the default is the first option + # supported by the operating system: + # mmap + # sysv + # windows + # (change requires restart) #dynamic_shared_memory_type = posix # the default is the first option # supported by the operating system: # posix diff --git a/src/include/storage/pg_shmem.h b/src/include/storage/pg_shmem.h index 6b1e040251..86a2d32d56 100644 --- a/src/include/storage/pg_shmem.h +++ b/src/include/storage/pg_shmem.h @@ -52,6 +52,18 @@ typedef enum HUGE_PAGES_TRY } HugePagesType; +/* Possible values for shared_memory_type */ +typedef enum +{ + SHMEM_TYPE_WINDOWS, + SHMEM_TYPE_SYSV, + SHMEM_TYPE_MMAP +} PGShmemType; + +/* GUC variables */ +extern int shared_memory_type; +extern int huge_pages; + #ifndef WIN32 extern unsigned long UsedShmemSegID; #else @@ -59,6 +71,14 @@ extern HANDLE UsedShmemSegID; #endif extern void *UsedShmemSegAddr; +#if !defined(WIN32) && !defined(EXEC_BACKEND) +#define DEFAULT_SHARED_MEMORY_TYPE SHMEM_TYPE_MMAP +#elif !defined(WIN32) +#define DEFAULT_SHARED_MEMORY_TYPE SHMEM_TYPE_SYSV +#else +#define DEFAULT_SHARED_MEMORY_TYPE SHMEM_TYPE_WINDOWS +#endif + #ifdef EXEC_BACKEND extern void PGSharedMemoryReAttach(void); extern void PGSharedMemoryNoReAttach(void); -- 2.19.1
0002-Add-huge-page-support-for-AIX.patch
Description: Binary data
