Hi,
I'd propose the attached WIP patch which allows us to enable WAL archiving
even in standby. The patch adds "always" as the valid value of archive_mode.
If it's set to "always", the archiver is started when the server is in standby
mode and all the WAL files that walreceiver wrote to the disk are archived by
using archive_command. Then, even after the server is promoted to master,
the archiver keeps archiving WAL files. The patch doesn't change the meanings
of the setting values "on" and "off" of archive_mode.
I think that this feature is useful for the case, e.g., where large database
needs to be replicated between remote servers. Imagine the situation where
the replicated database gets corrupted completely in the remote standby.
How should we address this problematic situation and restart the standby?
One approach is to take a fresh backup from the master and restore it onto
the standby. But since the database is large and there is long distance
between two servers, this approach might take a surprisingly long time.
Another approach is to restore the backup which was taken from the standby
before. But most of many WAL files which the backup needs might exist only
in the master (because WAL archiving cannot be enabled in the standby) and
they need to be transfered from the master to the standby via long-distance
network. So I think that this approach also would take a fairly long time.
To shorten that time, you may think that archive_command in the master can
be set so that it transfers WAL files from the master to the standby's
archival storage. I agree that this setting can accelerate the database restore
process. But this causes every WAL files to be transfered between remote
servers twice (one is by streaming replication, another is by archive_command),
and which is a waste of network bandwidth.
Enabling WAL archiving in standby is one of solutions for this situation. We
can expect that most of WAL files that are required for the backup taken from
the standby would exist in the standby's archival storage.
Back to the patch. If archive_mode is set to "always", archive_command is
always used to archive WAL files even during recovery. Do we need to separate
the command into two for master and standby, respectively? We can add
something like standby_archive_command parameter which is used to archive
only WAL files walreceiver writes. The other WAL files are archived by
archive_command. I'm not sure if it's really worth separating the command
that way. Is there any use case?
The patch doesn't allow us to enable WAL archiving *only* during recovery.
Should we support yet another archive_mode like "standby" which allows
the archiver to be running only during recovery, but makes it end just after
the server is promoted to master? I'm not sure if there is really use case for
that.
I've not included the update of document in the patch yet. If we agree to
support this feature, I will do the remaining work.
Regards,
--
Fujii Masao
*** a/src/backend/access/transam/xlog.c
--- b/src/backend/access/transam/xlog.c
***************
*** 81,87 **** int CheckPointSegments = 3;
int wal_keep_segments = 0;
int XLOGbuffers = -1;
int XLogArchiveTimeout = 0;
! bool XLogArchiveMode = false;
char *XLogArchiveCommand = NULL;
bool EnableHotStandby = false;
bool fullPageWrites = true;
--- 81,87 ----
int wal_keep_segments = 0;
int XLOGbuffers = -1;
int XLogArchiveTimeout = 0;
! int XLogArchiveMode = ARCHIVE_MODE_OFF;
char *XLogArchiveCommand = NULL;
bool EnableHotStandby = false;
bool fullPageWrites = true;
*** a/src/backend/postmaster/postmaster.c
--- b/src/backend/postmaster/postmaster.c
***************
*** 89,94 ****
--- 89,95 ----
#include "access/transam.h"
#include "access/xlog.h"
+ #include "access/xlogarchive.h"
#include "bootstrap/bootstrap.h"
#include "catalog/pg_control.h"
#include "lib/ilist.h"
***************
*** 823,831 **** PostmasterMain(int argc, char *argv[])
write_stderr("%s: max_wal_senders must be less than max_connections\n", progname);
ExitPostmaster(1);
}
! if (XLogArchiveMode && wal_level == WAL_LEVEL_MINIMAL)
ereport(ERROR,
! (errmsg("WAL archival (archive_mode=on) requires wal_level \"archive\", \"hot_standby\", or \"logical\"")));
if (max_wal_senders > 0 && wal_level == WAL_LEVEL_MINIMAL)
ereport(ERROR,
(errmsg("WAL streaming (max_wal_senders > 0) requires wal_level \"archive\", \"hot_standby\", or \"logical\"")));
--- 824,832 ----
write_stderr("%s: max_wal_senders must be less than max_connections\n", progname);
ExitPostmaster(1);
}
! if (XLogArchiveMode > ARCHIVE_MODE_OFF && wal_level == WAL_LEVEL_MINIMAL)
ereport(ERROR,
! (errmsg("WAL archival (archive_mode=on/always) requires wal_level \"archive\", \"hot_standby\", or \"logical\"")));
if (max_wal_senders > 0 && wal_level == WAL_LEVEL_MINIMAL)
ereport(ERROR,
(errmsg("WAL streaming (max_wal_senders > 0) requires wal_level \"archive\", \"hot_standby\", or \"logical\"")));
***************
*** 1617,1624 **** ServerLoop(void)
start_autovac_launcher = false; /* signal processed */
}
! /* If we have lost the archiver, try to start a new one */
! if (XLogArchivingActive() && PgArchPID == 0 && pmState == PM_RUN)
PgArchPID = pgarch_start();
/* If we have lost the stats collector, try to start a new one */
--- 1618,1634 ----
start_autovac_launcher = false; /* signal processed */
}
! /*
! * If we have lost the archiver, try to start a new one.
! *
! * If WAL archiving is enabled always, we try to start a new archiver
! * even during recovery.
! */
! if (PgArchPID == 0 &&
! ((XLogArchivingActive() && pmState == PM_RUN) ||
! (XLogArchivingActiveAlways() &&
! (pmState == PM_RECOVERY ||
! pmState == PM_HOT_STANDBY))))
PgArchPID = pgarch_start();
/* If we have lost the stats collector, try to start a new one */
***************
*** 4780,4785 **** sigusr1_handler(SIGNAL_ARGS)
--- 4790,4803 ----
Assert(BgWriterPID == 0);
BgWriterPID = StartBackgroundWriter();
+ /*
+ * Try to start an archiver even during recovery if archiving is
+ * enabled always.
+ */
+ Assert(PgArchPID == 0);
+ if (XLogArchivingActiveAlways())
+ PgArchPID = pgarch_start();
+
pmState = PM_RECOVERY;
}
if (CheckPostmasterSignal(PMSIGNAL_BEGIN_HOT_STANDBY) &&
*** a/src/backend/replication/walreceiver.c
--- b/src/backend/replication/walreceiver.c
***************
*** 547,557 **** WalReceiverMain(void)
XLogFileNameP(recvFileTLI, recvSegNo))));
/*
! * Create .done file forcibly to prevent the streamed segment from
! * being archived later.
*/
XLogFileName(xlogfname, recvFileTLI, recvSegNo);
! XLogArchiveForceDone(xlogfname);
}
recvFile = -1;
--- 547,563 ----
XLogFileNameP(recvFileTLI, recvSegNo))));
/*
! * Notify the archiver that the WAL segment is ready to
! * copy to archival storage even during recovery
! * if WAL archiving is enabled always. Otherwise,
! * create .done file forcibly to prevent the streamed segment
! * from being archived later.
*/
XLogFileName(xlogfname, recvFileTLI, recvSegNo);
! if (XLogArchivingActiveAlways())
! XLogArchiveNotify(xlogfname);
! else
! XLogArchiveForceDone(xlogfname);
}
recvFile = -1;
***************
*** 904,914 **** XLogWalRcvWrite(char *buf, Size nbytes, XLogRecPtr recptr)
XLogFileNameP(recvFileTLI, recvSegNo))));
/*
! * Create .done file forcibly to prevent the streamed segment
* from being archived later.
*/
XLogFileName(xlogfname, recvFileTLI, recvSegNo);
! XLogArchiveForceDone(xlogfname);
}
recvFile = -1;
--- 910,926 ----
XLogFileNameP(recvFileTLI, recvSegNo))));
/*
! * Notify the archiver that the WAL segment is ready to
! * copy to archival storage even during recovery
! * if WAL archiving is enabled always. Otherwise,
! * create .done file forcibly to prevent the streamed segment
* from being archived later.
*/
XLogFileName(xlogfname, recvFileTLI, recvSegNo);
! if (XLogArchivingActiveAlways())
! XLogArchiveNotify(xlogfname);
! else
! XLogArchiveForceDone(xlogfname);
}
recvFile = -1;
*** a/src/backend/utils/misc/guc.c
--- b/src/backend/utils/misc/guc.c
***************
*** 401,406 **** static const struct config_enum_entry huge_pages_options[] = {
--- 401,423 ----
};
/*
+ * Although only "on", "off", and "always" are documented,
+ * we accept all the likely variants of "on" and "off".
+ */
+ static const struct config_enum_entry archive_mode_options[] = {
+ {"always", ARCHIVE_MODE_ALWAYS, false},
+ {"on", ARCHIVE_MODE_ON, false},
+ {"off", ARCHIVE_MODE_OFF, false},
+ {"true", ARCHIVE_MODE_ON, true},
+ {"false", ARCHIVE_MODE_OFF, true},
+ {"yes", ARCHIVE_MODE_ON, true},
+ {"no", ARCHIVE_MODE_OFF, true},
+ {"1", ARCHIVE_MODE_ON, true},
+ {"0", ARCHIVE_MODE_OFF, true},
+ {NULL, 0, false}
+ };
+
+ /*
* Options for enum values stored in other modules
*/
extern const struct config_enum_entry wal_level_options[];
***************
*** 1429,1444 **** static struct config_bool ConfigureNamesBool[] =
},
{
- {"archive_mode", PGC_POSTMASTER, WAL_ARCHIVING,
- gettext_noop("Allows archiving of WAL files using archive_command."),
- NULL
- },
- &XLogArchiveMode,
- false,
- NULL, NULL, NULL
- },
-
- {
{"hot_standby", PGC_POSTMASTER, REPLICATION_STANDBY,
gettext_noop("Allows connections and queries during recovery."),
NULL
--- 1446,1451 ----
***************
*** 3421,3426 **** static struct config_enum ConfigureNamesEnum[] =
--- 3428,3443 ----
},
{
+ {"archive_mode", PGC_POSTMASTER, WAL_ARCHIVING,
+ gettext_noop("Allows archiving of WAL files using archive command."),
+ NULL
+ },
+ &XLogArchiveMode,
+ ARCHIVE_MODE_OFF, archive_mode_options,
+ NULL, NULL, NULL
+ },
+
+ {
{"trace_recovery_messages", PGC_SIGHUP, DEVELOPER_OPTIONS,
gettext_noop("Enables logging of recovery-related debugging information."),
gettext_noop("Each level includes all the levels that follow it. The later"
*** a/src/backend/utils/misc/postgresql.conf.sample
--- b/src/backend/utils/misc/postgresql.conf.sample
***************
*** 202,208 ****
# - Archiving -
! #archive_mode = off # allows archiving to be done
# (change requires restart)
#archive_command = '' # command to use to archive a logfile segment
# placeholders: %p = path of file to archive
--- 202,208 ----
# - Archiving -
! #archive_mode = off # allows archiving to be done; off, on, or always
# (change requires restart)
#archive_command = '' # command to use to archive a logfile segment
# placeholders: %p = path of file to archive
*** a/src/include/access/xlog.h
--- b/src/include/access/xlog.h
***************
*** 12,17 ****
--- 12,18 ----
#define XLOG_H
#include "access/rmgr.h"
+ #include "access/xlogarchive.h"
#include "access/xlogdefs.h"
#include "datatype/timestamp.h"
#include "lib/stringinfo.h"
***************
*** 188,194 **** extern int CheckPointSegments;
extern int wal_keep_segments;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
! extern bool XLogArchiveMode;
extern char *XLogArchiveCommand;
extern bool EnableHotStandby;
extern bool fullPageWrites;
--- 189,195 ----
extern int wal_keep_segments;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
! extern int XLogArchiveMode;
extern char *XLogArchiveCommand;
extern bool EnableHotStandby;
extern bool fullPageWrites;
***************
*** 206,212 **** typedef enum WalLevel
} WalLevel;
extern int wal_level;
! #define XLogArchivingActive() (XLogArchiveMode && wal_level >= WAL_LEVEL_ARCHIVE)
#define XLogArchiveCommandSet() (XLogArchiveCommand[0] != '\0')
/*
--- 207,216 ----
} WalLevel;
extern int wal_level;
! #define XLogArchivingActive() (XLogArchiveMode > ARCHIVE_MODE_OFF \
! && wal_level >= WAL_LEVEL_ARCHIVE)
! #define XLogArchivingActiveAlways() (XLogArchiveMode == ARCHIVE_MODE_ALWAYS \
! && wal_level >= WAL_LEVEL_ARCHIVE)
#define XLogArchiveCommandSet() (XLogArchiveCommand[0] != '\0')
/*
*** /dev/null
--- b/src/include/access/xlogarchive.h
***************
*** 0 ****
--- 1,21 ----
+ /*
+ * xlogarchive.h
+ *
+ * Functions for archiving WAL files and restoring from the archive.
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/xlogarchive.h
+ */
+ #ifndef XLOG_ARCHIVE_H
+ #define XLOG_ARCHIVE_H
+
+ typedef enum
+ {
+ ARCHIVE_MODE_OFF, /* disabled*/
+ ARCHIVE_MODE_ON, /*enabled while server is running normally */
+ ARCHIVE_MODE_ALWAYS /* enabled always (even during recovery) */
+ } ArchiveModeLevel;
+
+ #endif /* XLOG_ARCHIVE_H */
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
