On 2020-01-24 08:50, Michael Paquier wrote:
On Wed, Jan 22, 2020 at 12:55:30AM +0300, Alexander Korotkov wrote:
On Sun, Jan 19, 2020 at 1:24 PM Michael Paquier <mich...@paquier.xyz>
wrote:
+static int
+RestoreArchivedWALFile(const char *path, const char *xlogfname,
+ off_t expectedSize, const char
*restoreCommand)
Not a fan of putting that to pg_rewind/parsexlog.c. It has nothing
to
do with WAL parsing, and it seems to me that we could have an
argument
for making this available as a frontend-only API in src/common/.
I've put it into src/common/fe_archive.c.
This split makes sense. You have forgotten to update
@pgcommonfrontendfiles in Mkvcbuild.pm for the MSVC build. Still, I
can see that we have a lot of duplication between the code of the
frontend and the backend, which is annoying.. The execution part is
tricky to split because the backend has pre- and post- callbacks, the
interruption handling is not the same and there is the problem of
elog() calls, with elevel that can be changed depending on the backend
configuration. However, I can see one portion of the code which is
fully duplicated and could be refactored out: the construction of the
command to execute, by having in input the restore_command string and
the arguments that we expect to replace in the '%' markers which are
part of the command.
OK, I agree that duplication of this part directly is annoying. I did a
refactoring and moved this restore_command building code into
common/archive. Separate file was needed, since fe_archive is
frontend-only, so I cannot put universal code there. I do not know is
there any better place for it.
If NULL is specified for one of those values,
then the construction routine returns NULL, synonym of failure. And
the result of the routine is the built command. I think that you
would need an extra argument to give space for the error message
generated in the event of an error when building the command.
I did it in a bit different way, but the overall logic is exactly as you
proposed, I hope.
Also I have checked win/linux build in the similar to cfbot CI setup and
now everything runs well.
+++ b/src/include/common/fe_archive.h
+#ifndef ARCHIVE_H
+#define ARCHIVE_H
This should be FE_ARCHIVE_H.
Changed.
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options,
&option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRC:c", long_options,
&option_index)) != -1)
This still includes 'C', but that should not be the case.
+ <application>pg_rewind</application> with the
<literal>-c</literal> or
+ <literal>-C</literal> option to automatically retrieve them from
the WAL
[...]
+ <term><option>-C <replaceable
class="parameter">restore_command</replaceable></option></term>
+ <term><option>--target-restore-command=<replaceable
class="parameter">restore_command</replaceable></option></term>
[...]
+ available, try re-running <application>pg_rewind</application>
with
+ the <option>-c</option> or <option>-C</option> option to search
+ for the missing files in the WAL archive.
Three places in the docs need to be cleaned up.
I have fixed all the above, thanks.
Do we really need to test the new "archive" mode as well for
003_extrafiles and 002_databases? I would imagine that only 001_basic
is enough.
We do not check any unique code paths with it, so I do it only for
001_basic now. I have left it as a test mode, since it will be easy to
turn this on for any other test in the future if needed and it fits well
RewindTest.pm module.
+# Moves WAL files to the temporary location and returns
restore_command
+# to get them back.
+sub move_wal
+{
+ my ($tmp_dir, $master_pgdata) = @_;
+ my $wal_archive_path = "$tmp_dir/master_wal_archive";
+ my $wal_path = "$master_pgdata/pg_wal";
+ my $wal_dir;
+ my $restore_command;
I think that this routine is wrongly designed. First, in order to
copy the contents from one path to another, we have
RecursiveCopy::copy_path, and there is a long history about making
it safe for the TAP tests. Second, there is in
PostgresNode::enable_restoring already a code path doing the
decision-making on how building restore_command. We should keep this
code path unique.
Yeah, thank you for pointing me to the RecursiveCopy::copypath and
especially PostgresNode::enable_restoring, I have modified test to use
them instead. The copypath does not work with existing destination
directories and does not preserve initial permissions, so I had to do
some dirty work to achieve what we need in the test and keep it simple
in the same time. However, I think that using these unified routines is
much better for a future support.
New version is attached. Do you have any other comments or objections?
Regards
--
Alexey
From 7af0b3642f6218c764eee361e013f24bfb43ffbe Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.alek...@gmail.com>
Date: Fri, 31 Jan 2020 20:08:13 +0300
Subject: [PATCH v14] pg_rewind: Add options to restore WAL files from archive
Currently, pg_rewind fails when it could not find required WAL files in the
target data directory. One have to manually figure out which WAL files are
required and copy them back from archive.
This commit implements two new pg_rewind options, which allow pg_rewind to
automatically retrieve missing WAL files from archival storage. The first
option let pg_rewind read restore_command from postgresql.conf, while the
second one specifies restore_command directly.
Discussion: https://postgr.es/m/a3acff50-5a0d-9a2c-b3b2-ee36168955c1%40postgrespro.ru
Author: Alexey Kondratov
Reviewed-by: Andrey Borodin, Alvaro Herrera, Michael Paquier
Reviewed-by: Andres Freund, Alexander Korotkov
---
doc/src/sgml/ref/pg_rewind.sgml | 28 ++++--
src/backend/access/transam/xlogarchive.c | 58 +------------
src/bin/pg_rewind/parsexlog.c | 33 ++++++-
src/bin/pg_rewind/pg_rewind.c | 77 ++++++++++++++--
src/bin/pg_rewind/pg_rewind.h | 6 +-
src/bin/pg_rewind/t/001_basic.pl | 3 +-
src/bin/pg_rewind/t/RewindTest.pm | 64 +++++++++++++-
src/common/Makefile | 2 +
src/common/archive.c | 97 +++++++++++++++++++++
src/common/fe_archive.c | 106 +++++++++++++++++++++++
src/include/common/archive.h | 21 +++++
src/include/common/fe_archive.h | 18 ++++
src/tools/msvc/Mkvcbuild.pm | 8 +-
13 files changed, 440 insertions(+), 81 deletions(-)
create mode 100644 src/common/archive.c
create mode 100644 src/common/fe_archive.c
create mode 100644 src/include/common/archive.h
create mode 100644 src/include/common/fe_archive.h
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 42d29edd4e..8fa0d5aad8 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -66,11 +66,11 @@ PostgreSQL documentation
can be found either on the target timeline, the source timeline, or their common
ancestor. In the typical failover scenario where the target cluster was
shut down soon after the divergence, this is not a problem, but if the
- target cluster ran for a long time after the divergence, the old WAL
- files might no longer be present. In that case, they can be manually
- copied from the WAL archive to the <filename>pg_wal</filename> directory, or
- fetched on startup by configuring <xref linkend="guc-primary-conninfo"/> or
- <xref linkend="guc-restore-command"/>. The use of
+ target cluster ran for a long time after the divergence, its old WAL
+ files might no longer be present. In this case, you can manually copy them
+ from the WAL archive to the <filename>pg_wal</filename> directory, or run
+ <application>pg_rewind</application> with the <literal>-c</literal> option to
+ automatically retrieve them from the WAL archive. The use of
<application>pg_rewind</application> is not limited to failover, e.g. a standby
server can be promoted, run some write transactions, and then rewinded
to become a standby again.
@@ -232,6 +232,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-c</option></term>
+ <term><option>--restore-target-wal</option></term>
+ <listitem>
+ <para>
+ Use the <varname>restore_command</varname> defined in
+ <filename>postgresql.conf</filename> to retrieve WAL files from
+ the WAL archive if these files are no longer available in the
+ <filename>pg_wal</filename> directory of the target cluster.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--debug</option></term>
<listitem>
@@ -318,7 +331,10 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b
history forked off from the target cluster. For each WAL record,
record each data block that was touched. This yields a list of all
the data blocks that were changed in the target cluster, after the
- source cluster forked off.
+ source cluster forked off. If some of the WAL files are no longer
+ available, try re-running <application>pg_rewind</application> with
+ the <option>-c</option> option to search for the missing files in
+ the WAL archive.
</para>
</step>
<step>
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 188b73e752..e71aa2edb8 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -21,6 +21,7 @@
#include "access/xlog.h"
#include "access/xlog_internal.h"
+#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
#include "replication/walsender.h"
@@ -55,9 +56,6 @@ RestoreArchivedFile(char *path, const char *xlogfname,
char xlogpath[MAXPGPATH];
char xlogRestoreCmd[MAXPGPATH];
char lastRestartPointFname[MAXPGPATH];
- char *dp;
- char *endp;
- const char *sp;
int rc;
struct stat stat_buf;
XLogSegNo restartSegNo;
@@ -149,58 +147,8 @@ RestoreArchivedFile(char *path, const char *xlogfname,
else
XLogFileName(lastRestartPointFname, 0, 0L, wal_segment_size);
- /*
- * construct the command to be executed
- */
- dp = xlogRestoreCmd;
- endp = xlogRestoreCmd + MAXPGPATH - 1;
- *endp = '\0';
-
- for (sp = recoveryRestoreCommand; *sp; sp++)
- {
- if (*sp == '%')
- {
- switch (sp[1])
- {
- case 'p':
- /* %p: relative path of target file */
- sp++;
- StrNCpy(dp, xlogpath, endp - dp);
- make_native_path(dp);
- dp += strlen(dp);
- break;
- case 'f':
- /* %f: filename of desired file */
- sp++;
- StrNCpy(dp, xlogfname, endp - dp);
- dp += strlen(dp);
- break;
- case 'r':
- /* %r: filename of last restartpoint */
- sp++;
- StrNCpy(dp, lastRestartPointFname, endp - dp);
- dp += strlen(dp);
- break;
- case '%':
- /* convert %% to a single % */
- sp++;
- if (dp < endp)
- *dp++ = *sp;
- break;
- default:
- /* otherwise treat the % as not special */
- if (dp < endp)
- *dp++ = *sp;
- break;
- }
- }
- else
- {
- if (dp < endp)
- *dp++ = *sp;
- }
- }
- *dp = '\0';
+ (void) ConstructRestoreCommand(recoveryRestoreCommand, xlogpath, xlogfname,
+ lastRestartPointFname, xlogRestoreCmd);
ereport(DEBUG3,
(errmsg_internal("executing restore command \"%s\"",
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index eb61cb8803..08a7fc3560 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -19,6 +19,7 @@
#include "catalog/pg_control.h"
#include "catalog/storage_xlog.h"
#include "commands/dbcommands_xlog.h"
+#include "common/fe_archive.h"
#include "filemap.h"
#include "pg_rewind.h"
@@ -41,6 +42,7 @@ static char xlogfpath[MAXPGPATH];
typedef struct XLogPageReadPrivate
{
+ const char *restoreCommand;
int tliIndex;
} XLogPageReadPrivate;
@@ -55,7 +57,7 @@ static int SimpleXLogPageRead(XLogReaderState *xlogreader,
*/
void
extractPageMap(const char *datadir, XLogRecPtr startpoint, int tliIndex,
- XLogRecPtr endpoint)
+ XLogRecPtr endpoint, const char *restore_command)
{
XLogRecord *record;
XLogReaderState *xlogreader;
@@ -63,6 +65,7 @@ extractPageMap(const char *datadir, XLogRecPtr startpoint, int tliIndex,
XLogPageReadPrivate private;
private.tliIndex = tliIndex;
+ private.restoreCommand = restore_command;
xlogreader = XLogReaderAllocate(WalSegSz, datadir, &SimpleXLogPageRead,
&private);
if (xlogreader == NULL)
@@ -146,7 +149,7 @@ readOneRecord(const char *datadir, XLogRecPtr ptr, int tliIndex)
void
findLastCheckpoint(const char *datadir, XLogRecPtr forkptr, int tliIndex,
XLogRecPtr *lastchkptrec, TimeLineID *lastchkpttli,
- XLogRecPtr *lastchkptredo)
+ XLogRecPtr *lastchkptredo, const char *restoreCommand)
{
/* Walk backwards, starting from the given record */
XLogRecord *record;
@@ -170,6 +173,7 @@ findLastCheckpoint(const char *datadir, XLogRecPtr forkptr, int tliIndex,
}
private.tliIndex = tliIndex;
+ private.restoreCommand = restoreCommand;
xlogreader = XLogReaderAllocate(WalSegSz, datadir, &SimpleXLogPageRead,
&private);
if (xlogreader == NULL)
@@ -281,8 +285,29 @@ SimpleXLogPageRead(XLogReaderState *xlogreader, XLogRecPtr targetPagePtr,
if (xlogreadfd < 0)
{
- pg_log_error("could not open file \"%s\": %m", xlogfpath);
- return -1;
+ /*
+ * If we have no restore_command to execute, then exit.
+ */
+ if (private->restoreCommand == NULL)
+ {
+ pg_log_error("could not open file \"%s\": %m", xlogfpath);
+ return -1;
+ }
+
+ /*
+ * Since we have restore_command to execute, then try to retrieve
+ * missing WAL file from the archive.
+ */
+ xlogreadfd = RestoreArchivedWALFile(xlogreader->segcxt.ws_dir,
+ xlogfname,
+ WalSegSz,
+ private->restoreCommand);
+
+ if (xlogreadfd < 0)
+ return -1;
+ else
+ pg_log_debug("using file \"%s\" restored from archive",
+ xlogfpath);
}
}
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index c6d00bb0ab..f078cd694a 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -53,11 +53,13 @@ int WalSegSz;
char *datadir_target = NULL;
char *datadir_source = NULL;
char *connstr_source = NULL;
+char *restore_command = NULL;
static bool debug = false;
bool showprogress = false;
bool dry_run = false;
bool do_sync = true;
+bool restore_wal = false;
/* Target history */
TimeLineHistoryEntry *targetHistory;
@@ -79,6 +81,8 @@ usage(const char *progname)
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
printf(_(" -R, --write-recovery-conf write configuration for replication\n"
" (requires --source-server)\n"));
+ printf(_(" -c, --restore-target-wal use restore_command in target config\n"));
+ printf(_(" to retrieve WAL files from archive\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -105,6 +109,7 @@ main(int argc, char **argv)
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
{"progress", no_argument, NULL, 'P'},
+ {"restore-target-wal", no_argument, NULL, 'c'},
{"debug", no_argument, NULL, 3},
{NULL, 0, NULL, 0}
};
@@ -143,7 +148,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRc", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -155,6 +160,10 @@ main(int argc, char **argv)
showprogress = true;
break;
+ case 'c':
+ restore_wal = true;
+ break;
+
case 'n':
dry_run = true;
break;
@@ -252,6 +261,65 @@ main(int argc, char **argv)
exit(1);
}
+ if (restore_wal)
+ {
+ int rc;
+ char postgres_exec_path[MAXPGPATH],
+ postgres_cmd[MAXPGPATH],
+ cmd_output[MAXPGPATH];
+ FILE *output_fp;
+
+ /* Find postgres executable. */
+ rc = find_other_exec(argv[0], "postgres",
+ PG_BACKEND_VERSIONSTR,
+ postgres_exec_path);
+
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv[0], full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (rc == -1)
+ pg_log_error("The program \"postgres\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ progname, full_path);
+ else
+ pg_log_error("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ full_path, progname);
+ exit(1);
+ }
+
+ /*
+ * Build a command to execute for restore_command GUC retrieval if
+ * set.
+ */
+ snprintf(postgres_cmd, sizeof(postgres_cmd), "%s -D %s -C restore_command",
+ postgres_exec_path, datadir_target);
+
+ if ((output_fp = popen(postgres_cmd, "r")) == NULL ||
+ fgets(cmd_output, sizeof(cmd_output), output_fp) == NULL)
+ pg_fatal("could not get restore_command using %s: %s",
+ postgres_cmd, strerror(errno));
+
+ pclose(output_fp);
+
+ /* Remove trailing newline */
+ if (strchr(cmd_output, '\n') != NULL)
+ *strchr(cmd_output, '\n') = '\0';
+
+ if (!strcmp(cmd_output, ""))
+ pg_fatal("restore_command is not set on the target cluster");
+
+ restore_command = pg_strdup(cmd_output);
+
+ pg_log_debug("using config variable restore_command=\'%s\'.", restore_command);
+ }
+
umask(pg_mode_mask);
atexit(disconnect_atexit);
@@ -349,9 +417,8 @@ main(int argc, char **argv)
exit(0);
}
- findLastCheckpoint(datadir_target, divergerec,
- lastcommontliIndex,
- &chkptrec, &chkpttli, &chkptredo);
+ findLastCheckpoint(datadir_target, divergerec, lastcommontliIndex,
+ &chkptrec, &chkpttli, &chkptredo, restore_command);
pg_log_info("rewinding from last common checkpoint at %X/%X on timeline %u",
(uint32) (chkptrec >> 32), (uint32) chkptrec,
chkpttli);
@@ -377,7 +444,7 @@ main(int argc, char **argv)
if (showprogress)
pg_log_info("reading WAL in target");
extractPageMap(datadir_target, chkptrec, lastcommontliIndex,
- ControlFile_target.checkPoint);
+ ControlFile_target.checkPoint, restore_command);
filemap_finalize();
if (showprogress)
diff --git a/src/bin/pg_rewind/pg_rewind.h b/src/bin/pg_rewind/pg_rewind.h
index e4e8d23c32..b122ae43e5 100644
--- a/src/bin/pg_rewind/pg_rewind.h
+++ b/src/bin/pg_rewind/pg_rewind.h
@@ -42,11 +42,13 @@ extern uint64 fetch_done;
/* in parsexlog.c */
extern void extractPageMap(const char *datadir, XLogRecPtr startpoint,
- int tliIndex, XLogRecPtr endpoint);
+ int tliIndex, XLogRecPtr endpoint,
+ const char *restoreCommand);
extern void findLastCheckpoint(const char *datadir, XLogRecPtr searchptr,
int tliIndex,
XLogRecPtr *lastchkptrec, TimeLineID *lastchkpttli,
- XLogRecPtr *lastchkptredo);
+ XLogRecPtr *lastchkptredo,
+ const char *restoreCommand);
extern XLogRecPtr readOneRecord(const char *datadir, XLogRecPtr ptr,
int tliIndex);
diff --git a/src/bin/pg_rewind/t/001_basic.pl b/src/bin/pg_rewind/t/001_basic.pl
index 95d8ccfced..d97e437741 100644
--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 15;
+use Test::More tests => 20;
use FindBin;
use lib $FindBin::RealBin;
@@ -171,5 +171,6 @@ in master, before promotion
# Run the test in both modes
run_test('local');
run_test('remote');
+run_test('archive');
exit(0);
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index 82fa220ac8..fd81d5684f 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -38,6 +38,7 @@ use File::Copy;
use File::Path qw(rmtree);
use IPC::Run qw(run);
use PostgresNode;
+use RecursiveCopy;
use TestLib;
use Test::More;
@@ -227,10 +228,23 @@ sub run_pg_rewind
# Append the rewind-specific role to the connection string.
$standby_connstr = "$standby_connstr user=rewind_user";
- # Stop the master and be ready to perform the rewind. The cluster
- # needs recovery to finish once, and pg_rewind makes sure that it
- # happens automatically.
- $node_master->stop('immediate');
+ if ($test_mode eq 'archive')
+ {
+ # We test pg_rewind with restore_command by simply moving all WAL files
+ # to another location. It leads to failed ensureCleanShutdown
+ # execution. Since it is difficult to emulate a situation, when
+ # keeping the last WAL segment is enough for startup recovery, but
+ # not enough for successful pg_rewind run, we run these modes with
+ # --no-ensure-shutdown. So stop the master gracefully.
+ $node_master->stop;
+ }
+ else
+ {
+ # Stop the master and be ready to perform the rewind. The cluster
+ # needs recovery to finish once, and pg_rewind makes sure that it
+ # happens automatically.
+ $node_master->stop('immediate');
+ }
# At this point, the rewind processing is ready to run.
# We now have a very simple scenario with a few diverged WAL record.
@@ -284,6 +298,48 @@ sub run_pg_rewind
$node_standby->safe_psql('postgres',
"ALTER ROLE rewind_user WITH REPLICATION;");
}
+ elsif ($test_mode eq "archive")
+ {
+
+ # Do rewind using a local pgdata as source and
+ # specified directory with target WAL archive.
+ # Old master should be stopped at this point.
+
+ # First, remove archive_dir, since RecursiveCopy::copypath
+ # does not support copying to existing directories. It should
+ # be empty in this test, so it is safe.
+ rmdir($node_master->archive_dir);
+
+ # Move all old master WAL files to the archive.
+ RecursiveCopy::copypath(
+ $node_master->data_dir . "/pg_wal",
+ $node_master->archive_dir
+ );
+ chmod(0700, $node_master->archive_dir);
+
+ # Fast way to remove entire directory content
+ rmtree($node_master->data_dir . "/pg_wal");
+ mkdir($node_master->data_dir . "/pg_wal");
+ chmod(0700, $node_master->data_dir . "/pg_wal");
+
+ # Just to append an appropriate restore_command
+ # to postgresql.conf
+ $node_master->enable_restoring($node_master, 0);
+
+ # Stop the new master and be ready to perform the rewind.
+ $node_standby->stop;
+
+ command_ok(
+ [
+ 'pg_rewind',
+ "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-sync", "--no-ensure-shutdown",
+ "-c"
+ ],
+ 'pg_rewind archive_conf');
+ }
else
{
diff --git a/src/common/Makefile b/src/common/Makefile
index e757fb7399..43702dc309 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -47,6 +47,7 @@ LIBS += $(PTHREAD_LIBS)
# If you add objects here, see also src/tools/msvc/Mkvcbuild.pm
OBJS_COMMON = \
+ archive.o \
base64.o \
config_info.o \
controldata_utils.o \
@@ -87,6 +88,7 @@ endif
# (Mkvcbuild.pm has a copy of this list, too)
OBJS_FRONTEND = \
$(OBJS_COMMON) \
+ fe_archive.o \
fe_memutils.o \
file_utils.o \
logging.o \
diff --git a/src/common/archive.c b/src/common/archive.c
new file mode 100644
index 0000000000..d57768a949
--- /dev/null
+++ b/src/common/archive.c
@@ -0,0 +1,97 @@
+/*-------------------------------------------------------------------------
+ *
+ * archive.c
+ * Common WAL archive routines
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/common/archive.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "common/archive.h"
+
+/*
+ * Constructs restore_command from template with %p, %f and %r aliases.
+ * Returns 0 if restore_command was successfuly built.
+ *
+ * If any of the required arguments is NULL, but corresponding alias is
+ * met, then -1 code will be returned.
+ */
+int
+ConstructRestoreCommand(const char *restoreCommand,
+ const char *xlogpath,
+ const char *xlogfname,
+ const char *lastRestartPointFname,
+ char *result)
+{
+ char *dp,
+ *endp;
+ const char *sp;
+
+ /*
+ * Construct the command to be executed.
+ */
+ dp = result;
+ endp = result + MAXPGPATH - 1;
+ *endp = '\0';
+
+ for (sp = restoreCommand; *sp; sp++)
+ {
+ if (*sp == '%')
+ {
+ switch (sp[1])
+ {
+ case 'p':
+ /* %p: relative path of target file */
+ if (xlogpath == NULL)
+ return -1;
+ sp++;
+ StrNCpy(dp, xlogpath, endp - dp);
+ make_native_path(dp);
+ dp += strlen(dp);
+ break;
+ case 'f':
+ /* %f: filename of desired file */
+ if (xlogfname == NULL)
+ return -1;
+ sp++;
+ StrNCpy(dp, xlogfname, endp - dp);
+ dp += strlen(dp);
+ break;
+ case 'r':
+ /* %r: filename of last restartpoint */
+ if (lastRestartPointFname == NULL)
+ return -1;
+ sp++;
+ StrNCpy(dp, lastRestartPointFname, endp - dp);
+ dp += strlen(dp);
+ break;
+ case '%':
+ /* convert %% to a single % */
+ sp++;
+ if (dp < endp)
+ *dp++ = *sp;
+ break;
+ default:
+ /* otherwise treat the % as not special */
+ if (dp < endp)
+ *dp++ = *sp;
+ break;
+ }
+ }
+ else
+ {
+ if (dp < endp)
+ *dp++ = *sp;
+ }
+ }
+ *dp = '\0';
+
+ return 0;
+}
diff --git a/src/common/fe_archive.c b/src/common/fe_archive.c
new file mode 100644
index 0000000000..389d389d0d
--- /dev/null
+++ b/src/common/fe_archive.c
@@ -0,0 +1,106 @@
+/*-------------------------------------------------------------------------
+ *
+ * fe_archive.c
+ * Routines to access WAL archive from frontend
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/common/fe_archive.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include <unistd.h>
+#include <sys/stat.h>
+
+#include "access/xlog_internal.h"
+#include "common/archive.h"
+#include "common/fe_archive.h"
+#include "common/logging.h"
+
+
+/* logging support */
+#define pg_fatal(...) do { pg_log_fatal(__VA_ARGS__); exit(1); } while(0)
+
+/*
+ * Attempt to retrieve the specified file from off-line archival storage.
+ * If successful return a file descriptor of the restored WAL file, else
+ * return -1.
+ *
+ * For fixed-size files, the caller may pass the expected size as an
+ * additional crosscheck on successful recovery. If the file size is not
+ * known, set expectedSize = 0.
+ */
+int
+RestoreArchivedWALFile(const char *path, const char *xlogfname,
+ off_t expectedSize, const char *restoreCommand)
+{
+ char xlogpath[MAXPGPATH],
+ xlogRestoreCmd[MAXPGPATH];
+ int rc,
+ xlogfd;
+ struct stat stat_buf;
+
+ snprintf(xlogpath, MAXPGPATH, "%s/" XLOGDIR "/%s", path, xlogfname);
+
+ rc = ConstructRestoreCommand(restoreCommand, xlogpath, xlogfname, NULL, xlogRestoreCmd);
+
+ if (rc < 0)
+ pg_fatal("restore_command with %%r alias cannot be used.");
+
+ /*
+ * Execute restore_command, which should copy the missing WAL file from
+ * archival storage.
+ */
+ rc = system(xlogRestoreCmd);
+
+ if (rc == 0)
+ {
+ /*
+ * Command apparently succeeded, but let's make sure the file is
+ * really there now and has the correct size.
+ */
+ if (stat(xlogpath, &stat_buf) == 0)
+ {
+ if (expectedSize > 0 && stat_buf.st_size != expectedSize)
+ {
+ pg_log_error("archive file \"%s\" has wrong size: %lu instead of %lu, %s",
+ xlogfname, (unsigned long) stat_buf.st_size,
+ (unsigned long) expectedSize, strerror(errno));
+ }
+ else
+ {
+ xlogfd = open(xlogpath, O_RDONLY | PG_BINARY, 0);
+
+ if (xlogfd < 0)
+ pg_log_error("could not open file \"%s\" restored from archive: %s\n",
+ xlogpath, strerror(errno));
+ else
+ return xlogfd;
+ }
+ }
+ else
+ {
+ /* Stat failed */
+ pg_log_error("could not stat file \"%s\" restored from archive: %s",
+ xlogpath, strerror(errno));
+ }
+ }
+
+ /*
+ * If the failure was due to any sort of signal, then it will be
+ * misleading to return message 'could not restore file...' and propagate
+ * result to the upper levels. We should exit right now.
+ */
+ if (wait_result_is_any_signal(rc, false))
+ pg_fatal("restore_command failed due to the signal: %s",
+ wait_result_to_str(rc));
+
+ pg_log_error("could not restore file \"%s\" from archive\n",
+ xlogfname);
+ return -1;
+}
diff --git a/src/include/common/archive.h b/src/include/common/archive.h
new file mode 100644
index 0000000000..032761425b
--- /dev/null
+++ b/src/include/common/archive.h
@@ -0,0 +1,21 @@
+/*-------------------------------------------------------------------------
+ *
+ * archive.h
+ * Common WAL archive routines
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ *
+ * src/include/common/archive.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef ARCHIVE_H
+#define ARCHIVE_H
+
+extern int ConstructRestoreCommand(const char *restoreCommand,
+ const char *xlogpath,
+ const char *xlogfname,
+ const char *lastRestartPointFname,
+ char *result);
+
+#endif /* ARCHIVE_H */
diff --git a/src/include/common/fe_archive.h b/src/include/common/fe_archive.h
new file mode 100644
index 0000000000..a24e9f383b
--- /dev/null
+++ b/src/include/common/fe_archive.h
@@ -0,0 +1,18 @@
+/*-------------------------------------------------------------------------
+ *
+ * fe_archive.h
+ * Routines to access WAL archive from frontend
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ *
+ * src/include/common/fe_archive.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FE_ARCHIVE_H
+#define FE_ARCHIVE_H
+
+extern int RestoreArchivedWALFile(const char *path, const char *xlogfname,
+ off_t expectedSize, const char *restoreCommand);
+
+#endif /* FE_ARCHIVE_H */
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index a43e31c60e..0e298d3fdc 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -120,8 +120,8 @@ sub mkvcbuild
}
our @pgcommonallfiles = qw(
- base64.c config_info.c controldata_utils.c d2s.c encnames.c exec.c
- f2s.c file_perm.c ip.c jsonapi.c
+ archive.c base64.c config_info.c controldata_utils.c d2s.c encnames.c
+ exec.c f2s.c file_perm.c ip.c jsonapi.c
keywords.c kwlookup.c link-canary.c md5.c
pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c
saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c username.c
@@ -138,8 +138,8 @@ sub mkvcbuild
}
our @pgcommonfrontendfiles = (
- @pgcommonallfiles, qw(fe_memutils.c file_utils.c
- logging.c restricted_token.c));
+ @pgcommonallfiles, qw(fe_archive.c fe_memutils.c
+ file_utils.c logging.c restricted_token.c));
our @pgcommonbkndfiles = @pgcommonallfiles;
--
2.19.1
