Setting up a standby instance is still quite complicated. You need to
run pg_basebackup with all the right options. You need to make sure
pg_basebackup has the right permissions for the target directories. The
created instance has to be integrated into the operating system's start
scripts. There is this slightly awkward business of the --recovery-conf
option and how it interacts with other features. And you should
probably run pg_basebackup under screen. And then how do you get
notified when it's done. And when it's done you have to log back in and
finish up. Too many steps.
My idea is that the postmaster can launch a base backup worker, wait
till it's done, then proceed with the rest of the startup. initdb gets
a special option to create a "minimal" data directory with only a few
files, directories, and the usual configuration files. Then you create
a $PGDATA/basebackup.signal, start the postmaster as normal. It sees
the signal file, launches an auxiliary process that runs the base
backup, then proceeds with normal startup in standby mode.
This makes a whole bunch of things much nicer: The connection
information for where to get the base backup from comes from
postgresql.conf, so you only need to specify it in one place.
pg_basebackup is completely out of the picture; no need to deal with
command-line options, --recovery-conf, screen, monitoring for
completion, etc. If something fails, the base backup process can
automatically be restarted (maybe). Operating system integration is
much easier: You only call initdb and then pg_ctl or postgres, as you
are already doing. Automated deployment systems don't need to wait for
pg_basebackup to finish: You only call initdb, then start the server,
and then you're done -- waiting for the base backup to finish can be
done by the regular monitoring system.
Attached is a very hackish patch to implement this. It works like this:
# (assuming you have a primary already running somewhere)
initdb -D data2 --minimal
$EDITOR data2/postgresql.conf # set primary_conninfo
pg_ctl -D data2 start
(Curious side note: If you don’t set primary_conninfo in these steps,
then libpq defaults apply, so the default behavior might end up being
that a given instance attempts to replicate from itself.)
It works for basic cases. It's missing tablespace support, proper
fsyncing, progress reporting, probably more. Those would be pretty
straightforward I think. The interesting bit is the delicate ordering
of the postmaster startup: Normally, the pg_control file is read quite
early, but if starting from a minimal data directory, we need to wait
until the base backup is done. There is also the question what you do
if the base backup fails halfway through. Currently you probably need
to delete the whole data directory and start again with initdb. Better
might be a way to start again and overwrite any existing files, but that
can clearly also be dangerous. All this needs some careful analysis,
but I think it's doable.
Any thoughts?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
From 1cf36db2514b04428570496fc9d1145591fda0fc Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pe...@eisentraut.org>
Date: Sat, 29 Jun 2019 21:52:21 +0200
Subject: [PATCH v1] Base backup client as auxiliary backend process
---
src/backend/access/transam/xlog.c | 14 +-
src/backend/bootstrap/bootstrap.c | 9 +
src/backend/postmaster/pgstat.c | 6 +
src/backend/postmaster/postmaster.c | 95 +++++-
src/backend/replication/Makefile | 2 +-
src/backend/replication/basebackup_client.c | 33 ++
.../libpqwalreceiver/libpqwalreceiver.c | 308 ++++++++++++++++++
src/bin/initdb/initdb.c | 39 ++-
src/include/access/xlog.h | 4 +
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 1 +
src/include/replication/basebackup_client.h | 1 +
src/include/replication/walreceiver.h | 4 +
13 files changed, 502 insertions(+), 16 deletions(-)
create mode 100644 src/backend/replication/basebackup_client.c
create mode 100644 src/include/replication/basebackup_client.h
diff --git a/src/backend/access/transam/xlog.c
b/src/backend/access/transam/xlog.c
index e08320e829..da97970703 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -905,7 +905,6 @@ static XLogRecord *ReadCheckpointRecord(XLogReaderState
*xlogreader,
XLogRecPtr RecPtr, int whichChkpti, bool report);
static bool rescanLatestTimeLine(void);
static void WriteControlFile(void);
-static void ReadControlFile(void);
static char *str_time(pg_time_t tnow);
static bool CheckForStandbyTrigger(void);
@@ -4572,7 +4571,7 @@ WriteControlFile(void)
XLOG_CONTROL_FILE)));
}
-static void
+void
ReadControlFile(void)
{
pg_crc32c crc;
@@ -6209,13 +6208,11 @@ StartupXLOG(void)
CurrentResourceOwner = AuxProcessResourceOwner;
/*
- * Verify XLOG status looks valid.
+ * Check that contents look valid.
*/
- if (ControlFile->state < DB_SHUTDOWNED ||
- ControlFile->state > DB_IN_PRODUCTION ||
- !XRecOffIsValid(ControlFile->checkPoint))
+ if (!XRecOffIsValid(ControlFile->checkPoint))
ereport(FATAL,
- (errmsg("control file contains invalid data")));
+ (errmsg("control file contains invalid
checkpoint location")));
if (ControlFile->state == DB_SHUTDOWNED)
{
@@ -6248,6 +6245,9 @@ StartupXLOG(void)
ereport(LOG,
(errmsg("database system was interrupted; last
known up at %s",
str_time(ControlFile->time))));
+ else
+ ereport(FATAL,
+ (errmsg("control file contains invalid database
cluster state")));
/* This is just to allow attaching to startup process with a debugger */
#ifdef XLOG_REPLAY_DELAY
diff --git a/src/backend/bootstrap/bootstrap.c
b/src/backend/bootstrap/bootstrap.c
index 43627ab8f4..57769c160c 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -36,6 +36,7 @@
#include "postmaster/bgwriter.h"
#include "postmaster/startup.h"
#include "postmaster/walwriter.h"
+#include "replication/basebackup_client.h"
#include "replication/walreceiver.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
@@ -326,6 +327,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
statmsg = pgstat_get_backend_desc(B_STARTUP);
break;
+ case BaseBackupProcess:
+ statmsg =
pgstat_get_backend_desc(B_BASE_BACKUP);
+ break;
case BgWriterProcess:
statmsg = pgstat_get_backend_desc(B_BG_WRITER);
break;
@@ -451,6 +455,11 @@ AuxiliaryProcessMain(int argc, char *argv[])
StartupProcessMain();
proc_exit(1); /* should never return */
+ case BaseBackupProcess:
+ /* don't set signals, basebackup has its own agenda */
+ BaseBackupMain();
+ proc_exit(1); /* should never return */
+
case BgWriterProcess:
/* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index b4f2b28b51..6664932183 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -2934,6 +2934,9 @@ pgstat_bestart(void)
case StartupProcess:
lbeentry.st_backendType = B_STARTUP;
break;
+ case BaseBackupProcess:
+ lbeentry.st_backendType = B_BASE_BACKUP;
+ break;
case BgWriterProcess:
lbeentry.st_backendType = B_BG_WRITER;
break;
@@ -4289,6 +4292,9 @@ pgstat_get_backend_desc(BackendType backendType)
case B_BG_WORKER:
backendDesc = "background worker";
break;
+ case B_BASE_BACKUP:
+ backendDesc = "base backup";
+ break;
case B_BG_WRITER:
backendDesc = "background writer";
break;
diff --git a/src/backend/postmaster/postmaster.c
b/src/backend/postmaster/postmaster.c
index 688ad439ed..287b233399 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -248,6 +248,7 @@ bool restart_after_crash = true;
/* PIDs of special child processes; 0 when not running */
static pid_t StartupPID = 0,
+ BaseBackupPID = 0,
BgWriterPID = 0,
CheckpointerPID = 0,
WalWriterPID = 0,
@@ -539,6 +540,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartBaseBackup() StartChildProcess(BaseBackupProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -572,6 +574,8 @@ PostmasterMain(int argc, char *argv[])
bool listen_addr_saved = false;
int i;
char *output_config_variable = NULL;
+ struct stat stat_buf;
+ bool basebackup_signal_file_found = false;
InitProcessGlobals();
@@ -877,12 +881,27 @@ PostmasterMain(int argc, char *argv[])
/* Verify that DataDir looks reasonable */
checkDataDir();
- /* Check that pg_control exists */
- checkControlFile();
-
/* And switch working directory into it */
ChangeToDataDir();
+ if (stat(BASEBACKUP_SIGNAL_FILE, &stat_buf) == 0)
+ {
+ int fd;
+
+ fd = BasicOpenFilePerm(STANDBY_SIGNAL_FILE, O_RDWR | PG_BINARY,
+ S_IRUSR | S_IWUSR);
+ if (fd >= 0)
+ {
+ (void) pg_fsync(fd);
+ close(fd);
+ }
+ basebackup_signal_file_found = true;
+ }
+
+ /* Check that pg_control exists */
+ if (!basebackup_signal_file_found)
+ checkControlFile();
+
/*
* Check for invalid combinations of GUC settings.
*/
@@ -961,7 +980,8 @@ PostmasterMain(int argc, char *argv[])
* processes will inherit the correct function pointer and not need to
* repeat the test.
*/
- LocalProcessControlFile(false);
+ if (!basebackup_signal_file_found)
+ LocalProcessControlFile(false);
/*
* Initialize SSL library, if specified.
@@ -1363,6 +1383,33 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ if (basebackup_signal_file_found)
+ {
+ BaseBackupPID = StartBaseBackup();
+
+ /*
+ * XXX wait until done
+ */
+ while (BaseBackupPID != 0)
+ {
+ PG_SETMASK(&UnBlockSig);
+ sleep(2);
+ PG_SETMASK(&BlockSig);
+ }
+
+ /*
+ * Base backup done, now signal standby mode. XXX Is there a
use for
+ * switching into (non-standby) recovery here? How would that
be
+ * configured?
+ */
+ durable_rename(BASEBACKUP_SIGNAL_FILE, STANDBY_SIGNAL_FILE,
FATAL);
+
+ /*
+ * Reread the control file that came in with the base backup.
+ */
+ ReadControlFile();
+ }
+
/*
* We're ready to rock and roll...
*/
@@ -2631,6 +2678,8 @@ SIGHUP_handler(SIGNAL_ARGS)
SignalChildren(SIGHUP);
if (StartupPID != 0)
signal_child(StartupPID, SIGHUP);
+ if (BaseBackupPID != 0)
+ signal_child(BaseBackupPID, SIGHUP);
if (BgWriterPID != 0)
signal_child(BgWriterPID, SIGHUP);
if (CheckpointerPID != 0)
@@ -2782,6 +2831,8 @@ pmdie(SIGNAL_ARGS)
if (StartupPID != 0)
signal_child(StartupPID, SIGTERM);
+ if (BaseBackupPID != 0)
+ signal_child(BaseBackupPID, SIGTERM);
if (BgWriterPID != 0)
signal_child(BgWriterPID, SIGTERM);
if (WalReceiverPID != 0)
@@ -2997,6 +3048,22 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the base backup process?
+ */
+ if (pid == BaseBackupPID)
+ {
+ BaseBackupPID = 0;
+ if (EXIT_STATUS_0(exitstatus))
+ ;
+ else if (EXIT_STATUS_1(exitstatus))
+ elog(FATAL, "base backup failed");
+ else
+ HandleChildCrash(pid, exitstatus,
+ _("base backup
process"));
+ continue;
+ }
+
/*
* Was it the bgwriter? Normal exit can be ignored; we'll
start a new
* one at the next iteration of the postmaster's main loop, if
@@ -3516,6 +3583,18 @@ HandleChildCrash(int pid, int exitstatus, const char
*procname)
StartupStatus = STARTUP_SIGNALED;
}
+ /* Take care of the base backup process too */
+ if (pid == BaseBackupPID)
+ BaseBackupPID = 0;
+ else if (BaseBackupPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ?
"SIGSTOP" : "SIGQUIT"),
+ (int)
BaseBackupPID)));
+ signal_child(BaseBackupPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/* Take care of the bgwriter too */
if (pid == BgWriterPID)
BgWriterPID = 0;
@@ -3750,6 +3829,7 @@ PostmasterStateMachine(void)
if (CountChildren(BACKEND_TYPE_NORMAL | BACKEND_TYPE_WORKER) ==
0 &&
StartupPID == 0 &&
WalReceiverPID == 0 &&
+ BaseBackupPID == 0 &&
BgWriterPID == 0 &&
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
@@ -3844,6 +3924,7 @@ PostmasterStateMachine(void)
/* These other guys should be dead already */
Assert(StartupPID == 0);
Assert(WalReceiverPID == 0);
+ Assert(BaseBackupPID == 0);
Assert(BgWriterPID == 0);
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
@@ -4027,6 +4108,8 @@ TerminateChildren(int signal)
if (signal == SIGQUIT || signal == SIGKILL)
StartupStatus = STARTUP_SIGNALED;
}
+ if (BaseBackupPID != 0)
+ signal_child(BgWriterPID, signal);
if (BgWriterPID != 0)
signal_child(BgWriterPID, signal);
if (CheckpointerPID != 0)
@@ -5400,6 +5483,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup
process: %m")));
break;
+ case BaseBackupProcess:
+ ereport(LOG,
+ (errmsg("could not fork base
backup process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork
background writer process: %m")));
diff --git a/src/backend/replication/Makefile b/src/backend/replication/Makefile
index 562b55fbaa..748093100a 100644
--- a/src/backend/replication/Makefile
+++ b/src/backend/replication/Makefile
@@ -14,7 +14,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
-OBJS = walsender.o walreceiverfuncs.o walreceiver.o basebackup.o \
+OBJS = walsender.o walreceiverfuncs.o walreceiver.o basebackup.o
basebackup_client.o \
repl_gram.o slot.o slotfuncs.o syncrep.o syncrep_gram.o
SUBDIRS = logical
diff --git a/src/backend/replication/basebackup_client.c
b/src/backend/replication/basebackup_client.c
new file mode 100644
index 0000000000..4a75a6091f
--- /dev/null
+++ b/src/backend/replication/basebackup_client.c
@@ -0,0 +1,33 @@
+#include "postgres.h"
+
+#include <unistd.h>
+
+#include "replication/basebackup_client.h"
+#include "replication/walreceiver.h"
+#include "storage/ipc.h"
+#include "utils/guc.h"
+
+void
+BaseBackupMain(void)
+{
+ WalReceiverConn *wrconn = NULL;
+ char *err;
+
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
+ if (WalReceiverFunctions == NULL)
+ elog(ERROR, "libpqwalreceiver didn't initialize correctly");
+
+ /* Establish the connection to the primary */
+ wrconn = walrcv_connect(PrimaryConnInfo, false, cluster_name[0] ?
cluster_name : "basebackup", &err);
+ if (!wrconn)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary
server: %s", err)));
+
+ walrcv_base_backup(wrconn);
+
+ walrcv_disconnect(wrconn);
+
+ elog(LOG, "base backup completed");
+ proc_exit(0);
+}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a920..b72d849fde 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -17,8 +17,10 @@
#include "postgres.h"
#include <unistd.h>
+#include <sys/stat.h>
#include <sys/time.h>
+#include "common/string.h"
#include "libpq-fe.h"
#include "pqexpbuffer.h"
#include "access/xlog.h"
@@ -27,6 +29,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "pgtar.h"
#include "replication/walreceiver.h"
#include "utils/builtins.h"
#include "utils/memutils.h"
@@ -61,6 +64,7 @@ static int libpqrcv_server_version(WalReceiverConn *conn);
static void libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
TimeLineID tli, char **filename,
char **content, int *len);
+static void libpqrcv_base_backup(WalReceiverConn *conn);
static bool libpqrcv_startstreaming(WalReceiverConn *conn,
const
WalRcvStreamOptions *options);
static void libpqrcv_endstreaming(WalReceiverConn *conn,
@@ -88,6 +92,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
libpqrcv_identify_system,
libpqrcv_server_version,
libpqrcv_readtimelinehistoryfile,
+ libpqrcv_base_backup,
libpqrcv_startstreaming,
libpqrcv_endstreaming,
libpqrcv_receive,
@@ -356,6 +361,309 @@ libpqrcv_server_version(WalReceiverConn *conn)
return PQserverVersion(conn->streamConn);
}
+/*
+ * XXX copied from pg_basebackup.c
+ */
+static void
+ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
+{
+ char current_path[MAXPGPATH];
+ char filename[MAXPGPATH];
+ pgoff_t current_len_left = 0;
+ int current_padding = 0;
+ char *copybuf = NULL;
+ FILE *file = NULL;
+
+ strlcpy(current_path, DataDir, sizeof(current_path));
+
+ /*
+ * Get the COPY data
+ */
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ ereport(ERROR,
+ (errmsg("could not get COPY data stream: %s",
+ PQerrorMessage(conn))));
+
+ while (1)
+ {
+ int r;
+
+ if (copybuf != NULL)
+ {
+ PQfreemem(copybuf);
+ copybuf = NULL;
+ }
+
+ r = PQgetCopyData(conn, ©buf, 0);
+
+ if (r == -1)
+ {
+ /*
+ * End of chunk
+ */
+ if (file)
+ fclose(file);
+
+ break;
+ }
+ else if (r == -2)
+ {
+ ereport(ERROR,
+ (errmsg("could not read COPY data: %s",
+ PQerrorMessage(conn))));
+ }
+
+ if (file == NULL)
+ {
+ int filemode;
+
+ /*
+ * No current file, so this must be the header for a
new file
+ */
+ if (r != 512)
+ ereport(ERROR,
+ (errmsg("invalid tar block
header size: %d", r)));
+
+ current_len_left = read_tar_number(©buf[124], 12);
+
+ /* Set permissions on the file */
+ filemode = read_tar_number(©buf[100], 8);
+
+ /*
+ * All files are padded up to 512 bytes
+ */
+ current_padding =
+ ((current_len_left + 511) & ~511) -
current_len_left;
+
+ /*
+ * First part of header is zero terminated filename
+ */
+ snprintf(filename, sizeof(filename), "%s/%s",
current_path,
+ copybuf);
+ if (filename[strlen(filename) - 1] == '/')
+ {
+ /*
+ * Ends in a slash means directory or symlink
to directory
+ */
+ if (copybuf[156] == '5')
+ {
+ /*
+ * Directory
+ */
+ filename[strlen(filename) - 1] = '\0';
/* Remove trailing slash */
+ if (MakePGDirectory(filename) != 0)
+ {
+ if (errno != EEXIST)
+ {
+ elog(ERROR, "could not
create directory \"%s\": %m",
+ filename);
+ }
+ }
+#ifndef WIN32
+ if (chmod(filename, (mode_t) filemode))
+ elog(ERROR, "could not set
permissions on directory \"%s\": %m",
+ filename);
+#endif
+ }
+ else if (copybuf[156] == '2')
+ {
+ /*
+ * Symbolic link
+ *
+ * It's most likely a link in pg_tblspc
directory, to the
+ * location of a tablespace. Apply any
tablespace mapping
+ * given on the command line
(--tablespace-mapping). (We
+ * blindly apply the mapping without
checking that the
+ * link really is inside pg_tblspc. We
don't expect there
+ * to be other symlinks in a data
directory, but if there
+ * are, you can call it an undocumented
feature that you
+ * can map them too.)
+ */
+#ifdef TODO
+ filename[strlen(filename) - 1] = '\0';
/* Remove trailing slash */
+
+ mapped_tblspc_path =
get_tablespace_mapping(©buf[157]);
+ if (symlink(mapped_tblspc_path,
filename) != 0)
+ {
+ pg_log_error("could not create
symbolic link from \"%s\" to \"%s\": %m",
+
filename, mapped_tblspc_path);
+ exit(1);
+ }
+#endif
+ }
+ else
+ {
+ elog(ERROR, "unrecognized link
indicator \"%c\"",
+ copybuf[156]);
+ }
+ continue; /* directory or link
handled */
+ }
+
+ /*
+ * regular file
+ */
+ file = fopen(filename, "wb");
+ if (!file)
+ elog(ERROR, "could not create file \"%s\": %m",
filename);
+
+#ifndef WIN32
+ if (chmod(filename, (mode_t) filemode))
+ elog(ERROR, "could not set permissions on file
\"%s\": %m",
+ filename);
+#endif
+
+ if (current_len_left == 0)
+ {
+ /*
+ * Done with this file, next one will be a new
tar header
+ */
+ fclose(file);
+ file = NULL;
+ continue;
+ }
+ } /* new file */
+ else
+ {
+ /*
+ * Continuing blocks in existing file
+ */
+ if (current_len_left == 0 && r == current_padding)
+ {
+ /*
+ * Received the padding block for this file,
ignore it and
+ * close the file, then move on to the next tar
header.
+ */
+ fclose(file);
+ file = NULL;
+ continue;
+ }
+
+ if (fwrite(copybuf, r, 1, file) != 1)
+ elog(ERROR, "could not write to file \"%s\":
%m", filename);
+
+ current_len_left -= r;
+ if (current_len_left == 0 && current_padding == 0)
+ {
+ /*
+ * Received the last block, and there is no
padding to be
+ * expected. Close the file and move on to the
next tar
+ * header.
+ */
+ fclose(file);
+ file = NULL;
+ continue;
+ }
+ } /* continuing
data in existing file */
+ } /* loop over
all data blocks */
+
+ if (file != NULL)
+ elog(ERROR, "COPY stream ended before last file was finished");
+
+ if (copybuf != NULL)
+ PQfreemem(copybuf);
+}
+
+/*
+ * Make base backup from remote and write to local disk.
+ */
+static void
+libpqrcv_base_backup(WalReceiverConn *conn)
+{
+ PGresult *res;
+
+ elog(LOG, "initiating base backup, waiting for remote checkpoint to
complete");
+
+ if (PQsendQuery(conn->streamConn, "BASE_BACKUP") == 0)
+ ereport(ERROR,
+ (errmsg("could not start base backup on remote
server: %s",
+
pchomp(PQerrorMessage(conn->streamConn)))));
+
+ /*
+ * First result set: WAL start position and timeline ID; we skip it.
+ */
+ res = PQgetResult(conn->streamConn);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("could not start base backup on remote
server: %s",
+
pchomp(PQerrorMessage(conn->streamConn)))));
+ }
+
+ ereport(LOG,
+ (errmsg("remote checkpoint completed")));
+
+ PQclear(res);
+
+ /*
+ * Second result set: tablespace information
+ */
+ res = PQgetResult(conn->streamConn);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("could not get backup header: %s",
+
pchomp(PQerrorMessage(conn->streamConn)))));
+ }
+ if (PQntuples(res) < 1)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("no data returned from server")));
+ }
+
+ /*
+ * Start receiving chunks
+ */
+ for (int i = 0; i < PQntuples(res); i++)
+ {
+ ReceiveAndUnpackTarFile(conn->streamConn, res, i);
+ }
+
+ PQclear(res);
+
+ /*
+ * Final result set: WAL end position and timeline ID
+ */
+ res = PQgetResult(conn->streamConn);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("could not get write-ahead log end
position from server: %s",
+
pchomp(PQerrorMessage(conn->streamConn)))));
+ }
+ if (PQntuples(res) != 1)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("no write-ahead log end position
returned from server")));
+ }
+ PQclear(res);
+
+ res = PQgetResult(conn->streamConn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+#ifdef TODO
+ const char *sqlstate = PQresultErrorField(res,
PG_DIAG_SQLSTATE);
+
+ if (sqlstate &&
+ strcmp(sqlstate, ERRCODE_DATA_CORRUPTED) == 0)
+ {
+ elog(ERROR, "checksum error occurred");
+ }
+ else
+#endif
+ {
+ elog(ERROR, "final receive failed: %s",
+ pchomp(PQerrorMessage(conn->streamConn)));
+ }
+ }
+ PQclear(res);
+}
+
/*
* Start streaming WAL data from given streaming options.
*
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index ad5cd4194a..4c4ec78095 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -136,6 +136,7 @@ static char *pwfilename = NULL;
static char *superuser_password = NULL;
static const char *authmethodhost = NULL;
static const char *authmethodlocal = NULL;
+static bool minimal = false;
static bool debug = false;
static bool noclean = false;
static bool do_sync = true;
@@ -2962,6 +2963,22 @@ initialize_data_directory(void)
/* Now create all the text config files */
setup_config();
+ /*
+ * If minimal data directory requested, write basebackup.signal, and
then
+ * we are done here.
+ */
+ if (minimal)
+ {
+ char *path;
+ char *lines[1] = {NULL};
+
+ path = psprintf("%s/basebackup.signal", pg_data);
+ writefile(path, lines);
+ free(path);
+
+ return;
+ }
+
/* Bootstrap template1 */
bootstrap_template1();
@@ -3053,6 +3070,7 @@ main(int argc, char *argv[])
{"wal-segsize", required_argument, NULL, 12},
{"data-checksums", no_argument, NULL, 'k'},
{"allow-group-access", no_argument, NULL, 'g'},
+ {"minimal", no_argument, NULL, 'm'},
{NULL, 0, NULL, 0}
};
@@ -3094,7 +3112,7 @@ main(int argc, char *argv[])
/* process command-line options */
- while ((c = getopt_long(argc, argv, "dD:E:kL:nNU:WA:sST:X:g",
long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "dD:E:kL:mnNU:WA:sST:X:g",
long_options, &option_index)) != -1)
{
switch (c)
{
@@ -3149,6 +3167,9 @@ main(int argc, char *argv[])
case 'L':
share_path = pg_strdup(optarg);
break;
+ case 'm':
+ minimal = true;
+ break;
case 1:
locale = pg_strdup(optarg);
break;
@@ -3361,9 +3382,19 @@ main(int argc, char *argv[])
/* translator: This is a placeholder in a shell command. */
appendPQExpBuffer(start_db_cmd, " -l %s start", _("logfile"));
- printf(_("\nSuccess. You can now start the database server using:\n\n"
- " %s\n\n"),
- start_db_cmd->data);
+ if (!minimal)
+ {
+ printf(_("\nSuccess. You can now start the database server
using:\n\n"
+ " %s\n\n"),
+ start_db_cmd->data);
+ }
+ else
+ {
+ printf(_("\nSo far so good. Now configure the replication
connection in\n"
+ "postgresql.conf, and then start the database
server using:\n\n"
+ " %s\n\n"),
+ start_db_cmd->data);
+ }
destroyPQExpBuffer(start_db_cmd);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 237f4e0350..5f081923b2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -299,6 +299,9 @@ extern Size XLOGShmemSize(void);
extern void XLOGShmemInit(void);
extern void BootStrapXLOG(void);
extern void LocalProcessControlFile(bool reset);
+#ifndef FRONTEND
+extern void ReadControlFile(void);
+#endif
extern void StartupXLOG(void);
extern void ShutdownXLOG(int code, Datum arg);
extern void InitXLOGAccess(void);
@@ -354,6 +357,7 @@ extern void do_pg_abort_backup(void);
extern SessionBackupState get_backup_status(void);
/* File path names (all relative to $PGDATA) */
+#define BASEBACKUP_SIGNAL_FILE "basebackup.signal"
#define RECOVERY_SIGNAL_FILE "recovery.signal"
#define STANDBY_SIGNAL_FILE "standby.signal"
#define BACKUP_LABEL_FILE "backup_label"
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 61a24c2e3c..1f40d33290 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -398,6 +398,7 @@ typedef enum
CheckerProcess = 0,
BootstrapProcess,
StartupProcess,
+ BaseBackupProcess,
BgWriterProcess,
CheckpointerProcess,
WalWriterProcess,
@@ -410,6 +411,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType ==
StartupProcess)
+#define AmBaseBackupProcess() (MyAuxProcType == BaseBackupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
#define AmCheckpointerProcess() (MyAuxProcType ==
CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 0a3ad3a188..e5c385306e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -722,6 +722,7 @@ typedef enum BackendType
B_AUTOVAC_LAUNCHER,
B_AUTOVAC_WORKER,
B_BACKEND,
+ B_BASE_BACKUP,
B_BG_WORKER,
B_BG_WRITER,
B_CHECKPOINTER,
diff --git a/src/include/replication/basebackup_client.h
b/src/include/replication/basebackup_client.h
new file mode 100644
index 0000000000..dcd10b96f2
--- /dev/null
+++ b/src/include/replication/basebackup_client.h
@@ -0,0 +1 @@
+extern void BaseBackupMain(void);
diff --git a/src/include/replication/walreceiver.h
b/src/include/replication/walreceiver.h
index 86a8130051..bf895b1494 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -215,6 +215,7 @@ typedef void (*walrcv_readtimelinehistoryfile_fn)
(WalReceiverConn *conn,
TimeLineID tli,
char **filename,
char **content, int *size);
+typedef void (*walrcv_base_backup_fn) (WalReceiverConn *conn);
typedef bool (*walrcv_startstreaming_fn) (WalReceiverConn *conn,
const WalRcvStreamOptions *options);
typedef void (*walrcv_endstreaming_fn) (WalReceiverConn *conn,
@@ -242,6 +243,7 @@ typedef struct WalReceiverFunctionsType
walrcv_identify_system_fn walrcv_identify_system;
walrcv_server_version_fn walrcv_server_version;
walrcv_readtimelinehistoryfile_fn walrcv_readtimelinehistoryfile;
+ walrcv_base_backup_fn walrcv_base_backup;
walrcv_startstreaming_fn walrcv_startstreaming;
walrcv_endstreaming_fn walrcv_endstreaming;
walrcv_receive_fn walrcv_receive;
@@ -267,6 +269,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType
*WalReceiverFunctions;
WalReceiverFunctions->walrcv_server_version(conn)
#define walrcv_readtimelinehistoryfile(conn, tli, filename, content, size) \
WalReceiverFunctions->walrcv_readtimelinehistoryfile(conn, tli,
filename, content, size)
+#define walrcv_base_backup(conn) \
+ WalReceiverFunctions->walrcv_base_backup(conn)
#define walrcv_startstreaming(conn, options) \
WalReceiverFunctions->walrcv_startstreaming(conn, options)
#define walrcv_endstreaming(conn, next_tli) \
base-commit: c0faa727507ed34db0d02769d21bbaaf9605e2e4
--
2.22.0
