Hi hackers,
We’ve been thinking about how to make WAL archiving faster.
This topic was previously discussed in [1], and we’ve taken a first step by
implementing the attached patch, which adds support for archiving multiple
WAL files in one go.
The idea is straightforward: instead of invoking the archive command or
callback once per WAL file, we allow passing a batch of files. The patch
introduces support for new placeholders:
-
%F – list of WAL file names
-
%P – list of their full paths
-
%N – number of files
Since PostgreSQL already reads multiple files into memory and caches them
before archiving, this change avoids repeated fork() calls and reduces
overhead in high-throughput setups.
Of course, there are trade-offs. After discussing with Andrey Borodin, we
noted that if even one file in the batch fails to archive, we currently
have to retry the whole batch. While it’s technically possible to return a
list of successfully archived files, that would complicate the API and
introduce messy edge cases.
So we’re also exploring a more flexible idea: an asynchronous archiver mode.
The idea is to have PostgreSQL write WAL file names (marked .ready) into a
FIFO or pipe, and let an archive process or library asynchronously consume
and archive them. It would send back confirmations (or failures) through
another pipe, allowing PostgreSQL to retry failed files as needed. This
could decouple archiving from the archiver loop and open the door to more
efficient and parallel implementations.
We’d appreciate feedback on both directions:
-
Do you think the idea in the attached patch — batching WAL files for
archiving — is viable? Is it something worth pursuing?
-
What do you think about the async archiver concept? Would it fit
PostgreSQL’s architecture and operational expectations?
Thanks,
Stepan Neretin
[1]
https://www.postgresql.org/message-id/flat/BC335D75-105B-403F-9473-976C8BBC32E3%40yandex-team.ru#d45caa9d1075734567164f73371baf00
From 08104dcdd3295a48827e1d58c4c2382620267f5d Mon Sep 17 00:00:00 2001
From: Stepan Neretin <slp...@gmail.com>
Date: Mon, 21 Jul 2025 14:13:51 +0700
Subject: [PATCH] Add support for multi-file archiving in archive modules
This patch introduces optional support for archiving multiple WAL files
at once via a new archive_files_cb callback in archive modules.
A new GUC archive_multi enables this behavior.
The shell and basic archive modules have been updated to support the new callback.
---
contrib/basic_archive/basic_archive.c | 22 ++
src/backend/archive/shell_archive.c | 159 ++++++++----
src/backend/postmaster/pgarch.c | 300 +++++++++++++++--------
src/backend/utils/misc/guc_tables.c | 9 +
src/include/archive/archive_module.h | 18 ++
src/test/perl/PostgreSQL/Test/Cluster.pm | 47 ++--
src/test/recovery/t/002_archiving.pl | 1 +
7 files changed, 396 insertions(+), 160 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 4a8b8c7ac29..0f009e510ec 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -46,6 +46,7 @@ static char *archive_directory = NULL;
static bool basic_archive_configured(ArchiveModuleState *state);
static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static bool basic_archive_files(ArchiveModuleState *state, char **files, char **paths, int nfiles);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
@@ -53,6 +54,7 @@ static const ArchiveModuleCallbacks basic_archive_callbacks = {
.startup_cb = NULL,
.check_configured_cb = basic_archive_configured,
.archive_file_cb = basic_archive_file,
+ .archive_files_cb = basic_archive_files,
.shutdown_cb = NULL
};
@@ -229,6 +231,26 @@ basic_archive_file(ArchiveModuleState *state, const char *file, const char *path
return true;
}
+/*
++ * Archive multiple files.
++ */
+static bool
+basic_archive_files(ArchiveModuleState *state, char **files, char **paths, int nfiles)
+{
+ bool result = true;
+
+ for (int i = 0; i < nfiles; i++)
+ {
+ if (!basic_archive_file(state, files[i], paths[i]))
+ {
+ ereport(WARNING, (errmsg("failed to archive file \"%s\"", files[i])));
+ result = false;
+ }
+ }
+
+ return result;
+}
+
/*
* compare_files
*
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..9e8bf95c6c4 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -27,12 +27,17 @@ static bool shell_archive_configured(ArchiveModuleState *state);
static bool shell_archive_file(ArchiveModuleState *state,
const char *file,
const char *path);
+static bool shell_archive_files(ArchiveModuleState *state,
+ char **files,
+ char **paths,
+ int nfiles);
static void shell_archive_shutdown(ArchiveModuleState *state);
static const ArchiveModuleCallbacks shell_archive_callbacks = {
.startup_cb = NULL,
.check_configured_cb = shell_archive_configured,
.archive_file_cb = shell_archive_file,
+ .archive_files_cb = shell_archive_files,
.shutdown_cb = shell_archive_shutdown
};
@@ -54,26 +59,10 @@ shell_archive_configured(ArchiveModuleState *state)
}
static bool
-shell_archive_file(ArchiveModuleState *state, const char *file,
- const char *path)
+run_archive_command(const char *xlogarchcmd, const char *context_info, int nfiles)
{
- char *xlogarchcmd;
- char *nativePath = NULL;
- int rc;
-
- if (path)
- {
- nativePath = pstrdup(path);
- make_native_path(nativePath);
- }
-
- xlogarchcmd = replace_percent_placeholders(XLogArchiveCommand,
- "archive_command", "fp",
- file, nativePath);
-
- ereport(DEBUG3,
- (errmsg_internal("executing archive command \"%s\"",
- xlogarchcmd)));
+ int rc;
+ TimestampTz start_time = GetCurrentTimestamp();
fflush(NULL);
pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
@@ -82,59 +71,131 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
if (rc != 0)
{
- /*
- * If either the shell itself, or a called command, died on a signal,
- * abort the archiver. We do this because system() ignores SIGINT and
- * SIGQUIT while waiting; so a signal is very likely something that
- * should have interrupted us too. Also die if the shell got a hard
- * "command not found" type of error. If we overreact it's no big
- * deal, the postmaster will just start the archiver again.
- */
- int lev = wait_result_is_any_signal(rc, true) ? FATAL : LOG;
+ int lev = wait_result_is_any_signal(rc, true) ? FATAL : LOG;
if (WIFEXITED(rc))
{
ereport(lev,
- (errmsg("archive command failed with exit code %d",
- WEXITSTATUS(rc)),
- errdetail("The failed archive command was: %s",
- xlogarchcmd)));
+ (errmsg("archive command failed with exit code %d", WEXITSTATUS(rc)),
+ errdetail("The failed archive command was: %s", xlogarchcmd),
+ context_info ? errhint("%s", context_info) : 0));
}
else if (WIFSIGNALED(rc))
{
#if defined(WIN32)
ereport(lev,
- (errmsg("archive command was terminated by exception 0x%X",
- WTERMSIG(rc)),
- errhint("See C include file \"ntstatus.h\" for a description of the hexadecimal value."),
- errdetail("The failed archive command was: %s",
- xlogarchcmd)));
+ (errmsg("archive command was terminated by exception 0x%X", WTERMSIG(rc)),
+ errhint("See C include file \"ntstatus.h\" for a description of the hexadecimal value."),
+ errdetail("The failed archive command was: %s", xlogarchcmd),
+ context_info ? errhint("%s", context_info) : 0));
#else
ereport(lev,
- (errmsg("archive command was terminated by signal %d: %s",
- WTERMSIG(rc), pg_strsignal(WTERMSIG(rc))),
- errdetail("The failed archive command was: %s",
- xlogarchcmd)));
+ (errmsg("archive command was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc))),
+ errdetail("The failed archive command was: %s", xlogarchcmd),
+ context_info ? errhint("%s", context_info) : 0));
#endif
}
else
{
ereport(lev,
- (errmsg("archive command exited with unrecognized status %d",
- rc),
- errdetail("The failed archive command was: %s",
- xlogarchcmd)));
+ (errmsg("archive command exited with unrecognized status %d", rc),
+ errdetail("The failed archive command was: %s", xlogarchcmd),
+ context_info ? errhint("%s", context_info) : 0));
}
- pfree(xlogarchcmd);
-
return false;
}
- pfree(xlogarchcmd);
- elog(DEBUG1, "archived write-ahead log file \"%s\"", file);
return true;
}
+static bool
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
+{
+ char *xlogarchcmd;
+ char *nativePath = NULL;
+ bool success;
+
+ if (path)
+ {
+ nativePath = pstrdup(path);
+ make_native_path(nativePath);
+ }
+
+ xlogarchcmd = replace_percent_placeholders(XLogArchiveCommand,
+ "archive_command", "fp",
+ file, nativePath);
+
+ ereport(DEBUG3,
+ (errmsg_internal("executing archive command \"%s\"", xlogarchcmd)));
+
+ success = run_archive_command(xlogarchcmd, file, 1);
+
+ if (success)
+ elog(DEBUG1, "archived write-ahead log file \"%s\"", file);
+
+ pfree(xlogarchcmd);
+ if (nativePath)
+ pfree(nativePath);
+
+ return success;
+}
+
+
+static bool
+shell_archive_files(ArchiveModuleState *state, char **files, char **paths, int nfiles)
+{
+ StringInfoData filebuf;
+ StringInfoData pathbuf;
+ char nfiles_str[16];
+ char *xlogarchcmd;
+ bool success;
+
+ initStringInfo(&filebuf);
+ for (int i = 0; i < nfiles; i++)
+ {
+ if (i > 0)
+ appendStringInfoChar(&filebuf, ' ');
+ appendStringInfoString(&filebuf, files[i]);
+ }
+
+ initStringInfo(&pathbuf);
+ for (int i = 0; i < nfiles; i++)
+ {
+ char *nativePath = pstrdup(paths[i]);
+ make_native_path(nativePath);
+
+ if (i > 0)
+ appendStringInfoChar(&pathbuf, ' ');
+ appendStringInfoString(&pathbuf, nativePath);
+
+ pfree(nativePath);
+ }
+
+ snprintf(nfiles_str, sizeof(nfiles_str), "%d", nfiles);
+
+ xlogarchcmd = replace_percent_placeholders(XLogArchiveCommand,
+ "archive_command", "FPN",
+ filebuf.data, pathbuf.data, nfiles_str);
+
+ ereport(DEBUG3,
+ (errmsg_internal("executing multi-file archive command: %s", xlogarchcmd)));
+
+ success = run_archive_command(xlogarchcmd, filebuf.data, nfiles);
+
+ if (success)
+ {
+ for (int i = 0; i < nfiles; i++)
+ elog(DEBUG1, "archived write-ahead log file \"%s\"", files[i]);
+ }
+
+ pfree(xlogarchcmd);
+ pfree(filebuf.data);
+ pfree(pathbuf.data);
+
+ return success;
+}
+
static void
shell_archive_shutdown(ArchiveModuleState *state)
{
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..93c3677e25f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -93,6 +93,7 @@ typedef struct PgArchData
} PgArchData;
char *XLogArchiveLibrary = "";
+bool XLogArchiveMulti = false;
char *arch_module_check_errdetail_string;
@@ -144,6 +145,7 @@ static volatile sig_atomic_t ready_to_stop = false;
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
+static void pgarch_ArchiverCopyLoopMulti(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
@@ -346,7 +348,10 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- pgarch_ArchiverCopyLoop();
+ if (XLogArchiveMulti)
+ pgarch_ArchiverCopyLoopMulti();
+ else
+ pgarch_ArchiverCopyLoop();
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -372,6 +377,167 @@ pgarch_MainLoop(void)
} while (!time_to_stop);
}
+typedef bool (*ArchiveCallbackFn)(void *arg);
+
+static bool
+pgarch_safe_callback(ArchiveCallbackFn fn, void *arg)
+{
+ sigjmp_buf local_sigjmp_buf;
+ MemoryContext oldcontext;
+ bool ret;
+
+ oldcontext = MemoryContextSwitchTo(archive_context);
+
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+ EmitErrorReport();
+ disable_all_timeouts(false);
+ LWLockReleaseAll();
+ ConditionVariableCancelSleep();
+ pgstat_report_wait_end();
+ ReleaseAuxProcessResources(false);
+ AtEOXact_Files(false);
+ AtEOXact_HashTables(false);
+
+ MemoryContextSwitchTo(oldcontext);
+ FlushErrorState();
+ MemoryContextReset(archive_context);
+
+ PG_exception_stack = NULL;
+ RESUME_INTERRUPTS();
+
+ ret = false;
+ }
+ else
+ {
+ PG_exception_stack = &local_sigjmp_buf;
+ ret = fn(arg);
+ PG_exception_stack = NULL;
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(archive_context);
+ }
+
+ return ret;
+}
+
+typedef struct {
+ char **filenames;
+ char **paths;
+ int n;
+} ArchiveFilesArg;
+
+static bool
+archive_files_wrapper(void *arg)
+{
+ ArchiveFilesArg *a = (ArchiveFilesArg *) arg;
+ return ArchiveCallbacks->archive_files_cb(archive_module_state,
+ a->filenames,
+ a->paths,
+ a->n);
+}
+
+/*
+ * pgarch_ArchiverCopyLoopMulti
+ *
+ * Archives multiple outstanding WAL files in one batch using archive_files_cb.
+ */
+static void
+pgarch_ArchiverCopyLoopMulti(void)
+{
+ char *filenames[NUM_FILES_PER_DIRECTORY_SCAN];
+ char *paths[NUM_FILES_PER_DIRECTORY_SCAN];
+ int n;
+ bool ret;
+ ArchiveFilesArg arg;
+
+
+ elog(DEBUG1, "Starting archiver copy(multi)");
+
+ Assert(XLogArchiveMulti);
+ Assert(ArchiveCallbacks->archive_files_cb != NULL);
+
+ while (1)
+ {
+ PgArchForceDirScan();
+ pgarch_readyXlog(NULL);
+ n = 0;
+
+ for (int i = 0; i < arch_files->arch_files_size; i++)
+ {
+ char *arch_file = arch_files->arch_files[i];
+ char xlogpath[MAXPGPATH];
+ struct stat stat_buf;
+
+ snprintf(xlogpath, MAXPGPATH, XLOGDIR "/%s", arch_file);
+
+ if (stat(xlogpath, &stat_buf) != 0 && errno == ENOENT)
+ {
+ char xlogready[MAXPGPATH];
+
+ StatusFilePath(xlogready, arch_file, ".ready");
+ if (unlink(xlogready) == 0)
+ {
+ ereport(WARNING,
+ (errmsg("removed orphan archive status file \"%s\"", xlogready)));
+ }
+ else
+ {
+ ereport(WARNING,
+ (errmsg("could not remove orphan archive status file \"%s\": %m", xlogready)));
+ }
+ continue;
+ }
+
+ filenames[n] = arch_file;
+ paths[n] = psprintf(XLOGDIR "/%s", arch_file);
+ n++;
+ }
+
+ if (n == 0)
+ return;
+
+ if (ShutdownRequestPending || !PostmasterIsAlive())
+ return;
+
+ ProcessPgArchInterrupts();
+
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
+ {
+ ereport(WARNING,
+ (errmsg("\"archive_mode\" enabled, yet archiving is not configured"),
+ arch_module_check_errdetail_string ?
+ errdetail_internal("%s", arch_module_check_errdetail_string) : 0));
+ return;
+ }
+
+ arg.filenames = filenames;
+ arg.paths = paths;
+ arg.n = n;
+
+ ret = pgarch_safe_callback(archive_files_wrapper, &arg);
+
+ if (!ret)
+ {
+ for (int i = 0; i < n; i++)
+ pgstat_report_archiver(filenames[i], 0);
+
+ ereport(WARNING,
+ (errmsg("archiving multiple WAL files failed")));
+ return;
+ }
+
+ for (int i = 0; i < n; i++)
+ {
+ pgarch_archiveDone(filenames[i]);
+ pgstat_report_archiver(filenames[i], 1);
+ }
+ }
+}
+
/*
* pgarch_ArchiverCopyLoop
*
@@ -506,9 +672,19 @@ pgarch_ArchiverCopyLoop(void)
}
}
+typedef struct {
+ const char *xlog;
+ const char *pathname;
+} ArchiveXlogArg;
+
+static bool
+archive_file_wrapper(void *arg)
+{
+ ArchiveXlogArg *a = (ArchiveXlogArg *) arg;
+ return ArchiveCallbacks->archive_file_cb(archive_module_state, a->xlog, a->pathname);
+}
+
/*
- * pgarch_archiveXlog
- *
* Invokes archive_file_cb to copy one archive file to wherever it should go
*
* Returns true if successful
@@ -516,104 +692,22 @@ pgarch_ArchiverCopyLoop(void)
static bool
pgarch_archiveXlog(char *xlog)
{
- sigjmp_buf local_sigjmp_buf;
- MemoryContext oldcontext;
- char pathname[MAXPGPATH];
- char activitymsg[MAXFNAMELEN + 16];
- bool ret;
+ char pathname[MAXPGPATH];
+ char activitymsg[MAXFNAMELEN + 16];
+ ArchiveXlogArg arg;
+ bool ret;
snprintf(pathname, MAXPGPATH, XLOGDIR "/%s", xlog);
-
- /* Report archive activity in PS display */
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- oldcontext = MemoryContextSwitchTo(archive_context);
+ arg.xlog = xlog;
+ arg.pathname = pathname;
- /*
- * Since the archiver operates at the bottom of the exception stack,
- * ERRORs turn into FATALs and cause the archiver process to restart.
- * However, using ereport(ERROR, ...) when there are problems is easy to
- * code and maintain. Therefore, we create our own exception handler to
- * catch ERRORs and return false instead of restarting the archiver
- * whenever there is a failure.
- *
- * We assume ERRORs from the archiving callback are the most common
- * exceptions experienced by the archiver, so we opt to handle exceptions
- * here instead of PgArchiverMain() to avoid reinitializing the archiver
- * too frequently. We could instead add a sigsetjmp() block to
- * PgArchiverMain() and use PG_TRY/PG_CATCH here, but the extra code to
- * avoid the odd archiver restart doesn't seem worth it.
- */
- if (sigsetjmp(local_sigjmp_buf, 1) != 0)
- {
- /* Since not using PG_TRY, must reset error stack by hand */
- error_context_stack = NULL;
+ ret = pgarch_safe_callback(archive_file_wrapper, &arg);
- /* Prevent interrupts while cleaning up */
- HOLD_INTERRUPTS();
-
- /* Report the error to the server log. */
- EmitErrorReport();
-
- /*
- * Try to clean up anything the archive module left behind. We try to
- * cover anything that an archive module could conceivably have left
- * behind, but it is of course possible that modules could be doing
- * unexpected things that require additional cleanup. Module authors
- * should be sure to do any extra required cleanup in a PG_CATCH block
- * within the archiving callback, and they are encouraged to notify
- * the pgsql-hackers mailing list so that we can add it here.
- */
- disable_all_timeouts(false);
- LWLockReleaseAll();
- ConditionVariableCancelSleep();
- pgstat_report_wait_end();
- pgaio_error_cleanup();
- ReleaseAuxProcessResources(false);
- AtEOXact_Files(false);
- AtEOXact_HashTables(false);
-
- /*
- * Return to the original memory context and clear ErrorContext for
- * next time.
- */
- MemoryContextSwitchTo(oldcontext);
- FlushErrorState();
-
- /* Flush any leaked data */
- MemoryContextReset(archive_context);
-
- /* Remove our exception handler */
- PG_exception_stack = NULL;
-
- /* Now we can allow interrupts again */
- RESUME_INTERRUPTS();
-
- /* Report failure so that the archiver retries this file */
- ret = false;
- }
- else
- {
- /* Enable our exception handler */
- PG_exception_stack = &local_sigjmp_buf;
-
- /* Archive the file! */
- ret = ArchiveCallbacks->archive_file_cb(archive_module_state,
- xlog, pathname);
-
- /* Remove our exception handler */
- PG_exception_stack = NULL;
-
- /* Reset our memory context and switch back to the original one */
- MemoryContextSwitchTo(oldcontext);
- MemoryContextReset(archive_context);
- }
-
- if (ret)
- snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
- else
- snprintf(activitymsg, sizeof(activitymsg), "failed on %s", xlog);
+ snprintf(activitymsg, sizeof(activitymsg),
+ ret ? "last was %s" : "failed on %s", xlog);
set_ps_display(activitymsg);
return ret;
@@ -673,7 +767,10 @@ pgarch_readyXlog(char *xlog)
if (stat(status_file, &st) == 0)
{
- strcpy(xlog, arch_file);
+ if (xlog)
+ strcpy(xlog, arch_file);
+ else
+ arch_files->arch_files_size++;
return true;
}
else if (errno != ENOENT)
@@ -763,8 +860,11 @@ pgarch_readyXlog(char *xlog)
arch_files->arch_files[i] = DatumGetCString(binaryheap_remove_first(arch_files->arch_heap));
/* Return the highest priority file. */
- arch_files->arch_files_size--;
- strcpy(xlog, arch_files->arch_files[arch_files->arch_files_size]);
+ if (xlog)
+ {
+ arch_files->arch_files_size--;
+ strcpy(xlog, arch_files->arch_files[arch_files->arch_files_size]);
+ }
return true;
}
@@ -937,6 +1037,12 @@ LoadArchiveLibrary(void)
ArchiveCallbacks = (*archive_init) ();
+ if (XLogArchiveMulti && !ArchiveCallbacks->archive_files_cb)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("archive module does not support multi-file archiving"),
+ errdetail("The parameter \"archive_multi\" is enabled, but the archive module does not define archive_files_cb.")));
+
if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d14b1678e7f..9e6630a4145 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -799,6 +799,15 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"archive_multi", PGC_SIGHUP, WAL_ARCHIVING,
+ gettext_noop("Enables support for archiving multiple WAL files at once."),
+ NULL
+ },
+ &XLogArchiveMulti,
+ false,
+ NULL, NULL, NULL
+ },
{
{"enable_indexscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of index-scan plans."),
diff --git a/src/include/archive/archive_module.h b/src/include/archive/archive_module.h
index 3b17ca7eeca..b31282ba67f 100644
--- a/src/include/archive/archive_module.h
+++ b/src/include/archive/archive_module.h
@@ -16,6 +16,7 @@
* The value of the archive_library GUC.
*/
extern PGDLLIMPORT char *XLogArchiveLibrary;
+extern PGDLLIMPORT bool XLogArchiveMulti;
typedef struct ArchiveModuleState
{
@@ -40,11 +41,28 @@ typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
+
+/*
+ * Optional callback for multi-file archiving.
+ *
+ * Called when multiple WAL segments should be archived at once.
+ * The `files` and `paths` arrays must have `n_files` elements each.
+ *
+ * Return true if all files were archived successfully.
+ */
+typedef bool (*ArchiveFilesCB) (ArchiveModuleState *state,
+ char **files,
+ char **paths,
+ int n_files);
+
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
+
typedef struct ArchiveModuleCallbacks
{
ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
+ ArchiveFilesCB archive_files_cb; /* optional */
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 61f68e0cc2e..ac24bb296b4 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -637,6 +637,7 @@ sub init
$params{allows_streaming} = 0 unless defined $params{allows_streaming};
$params{force_initdb} = 0 unless defined $params{force_initdb};
$params{has_archiving} = 0 unless defined $params{has_archiving};
+ $params{is_multi} = 0 unless defined $params{is_multi};
my $initdb_extra_opts_env = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
if (defined $initdb_extra_opts_env)
@@ -766,7 +767,7 @@ sub init
or die("unable to set permissions for $pgdata/postgresql.conf");
$self->set_replication_conf if $params{allows_streaming};
- $self->enable_archiving if $params{has_archiving};
+ $self->enable_archiving(is_multi => $params{is_multi}) if $params{has_archiving};
return;
}
@@ -1481,30 +1482,48 @@ sub set_standby_mode
# Internal routine to enable archiving
sub enable_archiving
{
- my ($self) = @_;
+ my ($self, %opts) = @_;
+ my $is_multi = $opts{is_multi} // 0;
my $path = $self->archive_dir;
my $name = $self->name;
print "### Enabling WAL archiving for node \"$name\"\n";
- # On Windows, the path specified in the restore command needs to use
- # double back-slashes to work properly and to be able to detect properly
- # the file targeted by the copy command, so the directory value used
- # in this routine, using only one back-slash, need to be properly changed
- # first. Paths also need to be double-quoted to prevent failures where
- # the path contains spaces.
- $path =~ s{\\}{\\\\}g if ($PostgreSQL::Test::Utils::windows_os);
- my $copy_command =
- $PostgreSQL::Test::Utils::windows_os
- ? qq{copy "%p" "$path\\\\%f"}
- : qq{cp "%p" "$path/%f"};
+ # On Windows, adjust slashes and quoting
+ if ($PostgreSQL::Test::Utils::windows_os)
+ {
+ $path =~ s{\\}{\\\\}g;
+ }
- # Enable archive_mode and archive_command on node
+ my ($copy_command);
+
+ if ($PostgreSQL::Test::Utils::windows_os)
+ {
+ if ($is_multi) {
+ $copy_command = qq{copy %P "$path"};
+ } else {
+ $copy_command = qq{copy %p "$path\\\\%f"};
+ }
+ }
+ else
+ {
+ if ($is_multi) {
+ $copy_command = qq{cp %P "$path"};
+ } else {
+ $copy_command = qq{cp %p "$path/%f"};
+ }
+ }
+
+ my $archive_multi_conf = $is_multi ? "archive_multi = true\n" : "";
+
+ # Enable archive_mode, archive_command, and optionally archive_multi on node
$self->append_conf(
'postgresql.conf', qq(
archive_mode = on
archive_command = '$copy_command'
+$archive_multi_conf
));
+
return;
}
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
index 3acdb9ff1eb..a99e422f193 100644
--- a/src/test/recovery/t/002_archiving.pl
+++ b/src/test/recovery/t/002_archiving.pl
@@ -13,6 +13,7 @@ use File::Copy;
my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
has_archiving => 1,
+ is_multi => 1,
allows_streaming => 1);
my $backup_name = 'my_backup';
--
2.48.1
