On 11/21/19 1:46 PM, Peter Eisentraut wrote:
On 2019-11-08 05:00, Grigory Smolkin wrote:Attached new patch revision, now end of available WAL is defined as the fact of missing required WAL. In case of standby, the end of WAL is defined as 2 consecutive switches of WAL source, that didn`t provided requested record. In case of streaming standby, each switch of WAL source is forced after 3 failed attempts to get new data from walreceiver.All constants are taken off the top of my head and serves as proof of concept.Well, this is now changing the meaning of the patch quite a bit. I'm on board with making the existing default behavior explicit. (This is similar to how we added recovery_target_timeline = 'current' in PG12.) Still not a fan of the name yet, but that's trivial.If, however, you want to change the default behavior or introduce a new behavior, as you are suggesting here, that should be a separate discussion.
No, default behavior is not to be changed. As I previously mentioned, it may break the backward compatibility for 3rd party backup and replication applications.
At Fri, 8 Nov 2019 16:08:47 +0300, Grigory Smolkin<g.smol...@postgrespro.ru> wrote inOn 11/8/19 7:00 AM, Grigory Smolkin wrote:On 11/7/19 4:36 PM, Grigory Smolkin wrote:I gave it some thought and now think that prohibiting recovery_target 'latest' and standby was a bad idea. All recovery_targets follow the same pattern of usage, so recovery_target "latest" also must be capable of working in standby mode. All recovery_targets have a clear deterministic 'target' where recovery should end. In case of recovery_target "latest" this target is the end of available WAL, therefore the end of available WAL must be more clearly defined. I will work on it. Thank you for a feedback.Attached new patch revision, now end of available WAL is defined as the fact of missing required WAL. In case of standby, the end of WAL is defined as 2 consecutive switches of WAL source, that didn`t provided requested record. In case of streaming standby, each switch of WAL source is forced after 3 failed attempts to get new data from walreceiver. All constants are taken off the top of my head and serves as proof of concept. TAP test is updated.Attached new revision, it contains some minor refactoring.Thanks for the new patch. I found that it needs more than I thought, but it seems a bit too complicated and less stable. As the patch does, WaitForWALToBecomeAvailable needs to exit when avaiable sources are exhausted. However, we don't need to count failures to do that. It is enough that the function have two more exit point. When streaming timeout fires, and when we found that streaming is not set up when archive/wal failed. In my opinion, it is better that we have less dependency to global variables in such deep levels in a call hierachy. Such nformation can be stored in XLogPageReadPrivate.
Many thanks! It looks much better that my approach with global variables. I`ve tested it and have some thoughts/concerns:1. Recovery should report the exact reason why it has been forced to stop. In case of recovering to the end of WAL, standby promotion request received during recovery could be mistaken for reaching the end of WAL and reported as such. To avoid this, I think that reachedEndOfWal variable should be introduced.
In attached patch it is added as a global variable, but maybe something more clever may be devised. I was not sure that reachedEndOfWal could be placed in XLogPageReadPrivate. Because we need to access it at the higher level than ReadRecord(), and I was under impression placing it in XLogPageReadPrivate could violate abstraction level of XLogPageReadPrivate.
2. During the testing, I`ve stumbled upon assertion failure in case of recovering in standby mode to the the end of WAL coupled with recovery_target_action as "promote", caused by the WAL source in state machine not been changed after reaching the recovery target (script to reproduce is attached):
2019-12-07 13:45:54.468 MSK [23067] LOG: starting PostgreSQL 13devel on x86_64-pc-linux-gnu, compiled by gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1), 64-bit 2019-12-07 13:45:54.468 MSK [23067] LOG: listening on IPv4 address "127.0.0.1", port 15433 2019-12-07 13:45:54.470 MSK [23067] LOG: listening on Unix socket "/tmp/.s.PGSQL.15433" 2019-12-07 13:45:54.475 MSK [23068] LOG: database system was interrupted; last known up at 2019-12-07 13:45:49 MSK cp: cannot stat '/home/gsmol/task/13_devel/archive/00000002.history': No such file or directory
2019-12-07 13:45:54.602 MSK [23068] LOG: entering standby mode2019-12-07 13:45:54.614 MSK [23068] LOG: restored log file "000000010000000000000002" from archive
2019-12-07 13:45:54.679 MSK [23068] LOG: redo starts at 0/20000282019-12-07 13:45:54.682 MSK [23068] LOG: consistent recovery state reached at 0/2000100 2019-12-07 13:45:54.682 MSK [23067] LOG: database system is ready to accept read only connections 2019-12-07 13:45:54.711 MSK [23068] LOG: restored log file "000000010000000000000003" from archive 2019-12-07 13:45:54.891 MSK [23068] LOG: restored log file "000000010000000000000004" from archive 2019-12-07 13:45:55.046 MSK [23068] LOG: restored log file "000000010000000000000005" from archive 2019-12-07 13:45:55.210 MSK [23068] LOG: restored log file "000000010000000000000006" from archive 2019-12-07 13:45:55.377 MSK [23068] LOG: restored log file "000000010000000000000007" from archive 2019-12-07 13:45:55.566 MSK [23068] LOG: restored log file "000000010000000000000008" from archive 2019-12-07 13:45:55.737 MSK [23068] LOG: restored log file "000000010000000000000009" from archive cp: cannot stat '/home/gsmol/task/13_devel/archive/00000001000000000000000A': No such file or directory 2019-12-07 13:45:56.233 MSK [23083] LOG: started streaming WAL from primary at 0/A000000 on timeline 1 2019-12-07 13:45:56.365 MSK [23068] LOG: recovery stopping after reaching the end of available WAL
2019-12-07 13:45:56.365 MSK [23068] LOG: redo done at 0/9FFC6702019-12-07 13:45:56.365 MSK [23068] LOG: last completed transaction was at log time 2019-12-07 13:45:53.627746+03 2019-12-07 13:45:56.365 MSK [23083] FATAL: terminating walreceiver process due to administrator command
TRAP: FailedAssertion("StandbyMode", File: "xlog.c", Line: 12032)
postgres: startup waiting for
00000001000000000000000A(ExceptionalCondition+0xa8)[0xa88b55]
postgres: startup waiting for 00000001000000000000000A[0x573417] postgres: startup waiting for 00000001000000000000000A[0x572b68] postgres: startup waiting for 00000001000000000000000A[0x579066]postgres: startup waiting for 00000001000000000000000A(XLogReadRecord+0xe3)[0x5788ac]
postgres: startup waiting for 00000001000000000000000A[0x5651f8]postgres: startup waiting for 00000001000000000000000A(StartupXLOG+0x23aa)[0x56b26e] postgres: startup waiting for 00000001000000000000000A(StartupProcessMain+0xc7)[0x8642a1] postgres: startup waiting for 00000001000000000000000A(AuxiliaryProcessMain+0x5b8)[0x5802ad]
postgres: startup waiting for 00000001000000000000000A[0x863175]postgres: startup waiting for 00000001000000000000000A(PostmasterMain+0x1214)[0x85e0a4]
postgres: startup waiting for 00000001000000000000000A[0x76f247] /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f8867958f33]postgres: startup waiting for 00000001000000000000000A(_start+0x2e)[0x47afee]
#0 0x00007f886796ce75 in raise () from /lib64/libc.so.6 #1 0x00007f8867957895 in abort () from /lib64/libc.so.6#2 0x0000000000a88b82 in ExceptionalCondition (conditionName=0xb24acc "StandbyMode", errorType=0xb208a7 "FailedAssertion",
fileName=0xb208a0 "xlog.c", lineNumber=12032) at assert.c:67#3 0x0000000000573417 in WaitForWALToBecomeAvailable (RecPtr=151003136, randAccess=true, fetching_ckpt=false, tliRecPtr=167757424,
return_on_eow=true) at xlog.c:12032#4 0x0000000000572b68 in XLogPageRead (xlogreader=0xf08ed8, targetPagePtr=150994944, reqLen=8192, targetRecPtr=167757424,
readBuf=0xf37d50 "\002\321\005") at xlog.c:11611#5 0x0000000000579066 in ReadPageInternal (state=0xf08ed8, pageptr=167755776, reqLen=1672) at xlogreader.c:579 #6 0x00000000005788ac in XLogReadRecord (state=0xf08ed8, RecPtr=167757424, errormsg=0x7fff1f5cdeb8) at xlogreader.c:300 #7 0x00000000005651f8 in ReadRecord (xlogreader=0xf08ed8, RecPtr=167757424, emode=22, fetching_ckpt=false) at xlog.c:4271
#8 0x000000000056b26e in StartupXLOG () at xlog.c:7373 #9 0x00000000008642a1 in StartupProcessMain () at startup.c:196#10 0x00000000005802ad in AuxiliaryProcessMain (argc=2, argv=0x7fff1f5cea80) at bootstrap.c:451 #11 0x0000000000863175 in StartChildProcess (type=StartupProcess) at postmaster.c:5461 #12 0x000000000085e0a4 in PostmasterMain (argc=3, argv=0xf082b0) at postmaster.c:1392
#13 0x000000000076f247 in main (argc=3, argv=0xf082b0) at main.c:210 (gdb) frame 8 #8 0x000000000056b26e in StartupXLOG () at xlog.c:7373 7373 record = ReadRecord(xlogreader, LastRec, PANIC, false); (gdb) p StandbyMode $1 = false (gdb) list 7368 7369 /*7370 * Re-fetch the last valid or last applied record, so we can identify the 7371 * exact endpoint of what we consider the valid portion of WAL.
7372 */ 7373 record = ReadRecord(xlogreader, LastRec, PANIC, false); 7374 EndOfLog = EndRecPtr; 7375 7376 /*7377 * EndOfLogTLI is the TLI in the filename of the XLOG segment containing
Both issues are fixed in the new patch version. Any review and thoughts on the matters would be much appreciated.
I think The doc needs to exiplain on the difference between default and latest.
Sure, I will work on it.
Please find the attached, which illustrates the first two points of the aboves. regards.
-- Grigory Smolkin Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
recovery_target_latest.sh
Description: application/shellscript
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4ec13f3311..bc904a09bd 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -3385,21 +3385,19 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
<variablelist>
<varlistentry id="guc-recovery-target" xreflabel="recovery_target">
- <term><varname>recovery_target</varname><literal> = 'immediate'</literal>
+ <term><varname>recovery_target</varname> (<type>string</type>)
<indexterm>
<primary><varname>recovery_target</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- This parameter specifies that recovery should end as soon as a
- consistent state is reached, i.e. as early as possible. When restoring
- from an online backup, this means the point where taking the backup
- ended.
- </para>
- <para>
- Technically, this is a string parameter, but <literal>'immediate'</literal>
- is currently the only allowed value.
+ This parameter determines how far recovery should proceed. The value
+ <literal>immediate</literal> means that recovery should end as soon as a consistent
+ state is reached, i.e. as early as possible. When restoring from an online
+ backup, this means the point where taking the backup ended.
+ The second possible value <literal>latest</literal> means that recovery
+ should proceed to the end of the available WAL log.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 5f0ee50092..db75587f1e 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -808,6 +808,7 @@ typedef struct XLogPageReadPrivate
int emode;
bool fetching_ckpt; /* are we fetching a checkpoint record? */
bool randAccess;
+ bool return_on_eow; /* returns when reaching End of WAL */
} XLogPageReadPrivate;
/*
@@ -843,6 +844,9 @@ static bool updateMinRecoveryPoint = true;
*/
bool reachedConsistency = false;
+/* Have we reached End of WAL? Used only in case of RECOVERY_TARGET_LATEST */
+bool reachedEndOfWal = false;
+
static bool InRedo = false;
/* Have we launched bgwriter during recovery? */
@@ -887,7 +891,9 @@ static int XLogFileReadAnyTLI(XLogSegNo segno, int emode, int source);
static int XLogPageRead(XLogReaderState *xlogreader, XLogRecPtr targetPagePtr,
int reqLen, XLogRecPtr targetRecPtr, char *readBuf);
static bool WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
- bool fetching_ckpt, XLogRecPtr tliRecPtr);
+ bool fetching_ckpt,
+ XLogRecPtr tliRecPtr,
+ bool return_on_eow);
static int emode_for_corrupt_record(int emode, XLogRecPtr RecPtr);
static void XLogFileClose(void);
static void PreallocXlogFiles(XLogRecPtr endptr);
@@ -4253,6 +4259,7 @@ ReadRecord(XLogReaderState *xlogreader, XLogRecPtr RecPtr, int emode,
private->fetching_ckpt = fetching_ckpt;
private->emode = emode;
private->randAccess = (RecPtr != InvalidXLogRecPtr);
+ private->return_on_eow = (recoveryTarget == RECOVERY_TARGET_LATEST);
/* This is the first attempt to read this page. */
lastSourceFailed = false;
@@ -4371,8 +4378,12 @@ ReadRecord(XLogReaderState *xlogreader, XLogRecPtr RecPtr, int emode,
continue;
}
- /* In standby mode, loop back to retry. Otherwise, give up. */
- if (StandbyMode && !CheckForStandbyTrigger())
+ /*
+ * In standby mode, loop back to retry. Otherwise, give up.
+ * If we told to return on end of WAL, also give up.
+ */
+ if (StandbyMode && !CheckForStandbyTrigger() &&
+ !private->return_on_eow)
continue;
else
return NULL;
@@ -6342,6 +6353,9 @@ StartupXLOG(void)
else if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE)
ereport(LOG,
(errmsg("starting point-in-time recovery to earliest consistent point")));
+ else if (recoveryTarget == RECOVERY_TARGET_LATEST)
+ ereport(LOG,
+ (errmsg("starting point-in-time recovery until all available WAL is applied")));
else
ereport(LOG,
(errmsg("starting archive recovery")));
@@ -7254,6 +7268,13 @@ StartupXLOG(void)
* end of main redo apply loop
*/
+ if (recoveryTarget == RECOVERY_TARGET_LATEST && reachedEndOfWal)
+ {
+ ereport(LOG,
+ (errmsg("recovery stopping after reaching the end of available WAL")));
+ reachedStopPoint = true;
+ }
+
if (reachedStopPoint)
{
if (!reachedConsistency)
@@ -7456,6 +7477,8 @@ StartupXLOG(void)
recoveryStopName);
else if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE)
snprintf(reason, sizeof(reason), "reached consistency");
+ else if (recoveryTarget == RECOVERY_TARGET_LATEST)
+ snprintf(reason, sizeof(reason), "end of WAL");
else
snprintf(reason, sizeof(reason), "no recovery target specified");
@@ -11588,7 +11611,8 @@ retry:
if (!WaitForWALToBecomeAvailable(targetPagePtr + reqLen,
private->randAccess,
private->fetching_ckpt,
- targetRecPtr))
+ targetRecPtr,
+ private->return_on_eow))
{
if (readFile >= 0)
close(readFile);
@@ -11728,7 +11752,9 @@ next_record_is_invalid:
*
* If the record is not immediately available, the function returns false
* if we're not in standby mode. In standby mode, waits for it to become
- * available.
+ * available looping over all sources if return_on_eow is false. Otherwise the
+ * function returns false if the record is not immediately available on all
+ * sources.
*
* When the requested record becomes available, the function opens the file
* containing it (if not open already), and returns true. When end of standby
@@ -11737,7 +11763,8 @@ next_record_is_invalid:
*/
static bool
WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
- bool fetching_ckpt, XLogRecPtr tliRecPtr)
+ bool fetching_ckpt, XLogRecPtr tliRecPtr,
+ bool return_on_eow)
{
static TimestampTz last_fail_time = 0;
TimestampTz now;
@@ -11845,6 +11872,15 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
receivedUpto = 0;
}
+ /*
+ * If we don't get into streaming, this is the end of WAL
+ */
+ else if (return_on_eow)
+ {
+ reachedEndOfWal = true;
+ return false;
+ }
+
/*
* Move to XLOG_FROM_STREAM state in either case. We'll
* get immediate failure if we didn't launch walreceiver,
@@ -12107,6 +12143,18 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
WL_EXIT_ON_PM_DEATH,
5000L, WAIT_EVENT_RECOVERY_WAL_ALL);
ResetLatch(&XLogCtl->recoveryWakeupLatch);
+
+ /*
+ * This is not exactly the end of WAL, but assume the
+ * primary has no more record to send.
+ */
+ if (return_on_eow)
+ {
+ reachedEndOfWal = true;
+ currentSource = XLOG_FROM_PG_WAL;
+ return false;
+ }
+
break;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 5fccc9683e..3238e4ce98 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3545,7 +3545,10 @@ static struct config_string ConfigureNamesString[] =
{
{"recovery_target", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
- gettext_noop("Set to \"immediate\" to end recovery as soon as a consistent state is reached."),
+ gettext_noop("Set to \"immediate\" to end recovery as "
+ "soon as a consistent state is reached or "
+ " to \"latest\" to end recovery after "
+ "replaying all available WAL in the archive."),
NULL
},
&recovery_target_string,
@@ -11650,9 +11653,10 @@ error_multiple_recovery_targets(void)
static bool
check_recovery_target(char **newval, void **extra, GucSource source)
{
- if (strcmp(*newval, "immediate") != 0 && strcmp(*newval, "") != 0)
+ if (strcmp(*newval, "immediate") != 0 && strcmp(*newval, "latest") != 0 &&
+ strcmp(*newval, "") != 0)
{
- GUC_check_errdetail("The only allowed value is \"immediate\".");
+ GUC_check_errdetail("The only allowed values are \"immediate\" and \"latest\".");
return false;
}
return true;
@@ -11662,11 +11666,17 @@ static void
assign_recovery_target(const char *newval, void *extra)
{
if (recoveryTarget != RECOVERY_TARGET_UNSET &&
- recoveryTarget != RECOVERY_TARGET_IMMEDIATE)
+ recoveryTarget != RECOVERY_TARGET_IMMEDIATE &&
+ recoveryTarget != RECOVERY_TARGET_LATEST)
error_multiple_recovery_targets();
if (newval && strcmp(newval, "") != 0)
- recoveryTarget = RECOVERY_TARGET_IMMEDIATE;
+ {
+ if (strcmp(newval, "immediate") == 0)
+ recoveryTarget = RECOVERY_TARGET_IMMEDIATE;
+ else if (strcmp(newval, "latest") == 0)
+ recoveryTarget = RECOVERY_TARGET_LATEST;
+ }
else
recoveryTarget = RECOVERY_TARGET_UNSET;
}
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252aad..55e7e4bd2c 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -84,7 +84,8 @@ typedef enum
RECOVERY_TARGET_TIME,
RECOVERY_TARGET_NAME,
RECOVERY_TARGET_LSN,
- RECOVERY_TARGET_IMMEDIATE
+ RECOVERY_TARGET_IMMEDIATE,
+ RECOVERY_TARGET_LATEST
} RecoveryTargetType;
/*
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
index d8fbd50011..70fb79a8fc 100644
--- a/src/test/recovery/t/003_recovery_targets.pl
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -3,7 +3,8 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 8;
+use Test::More tests => 15;
+#use Time::HiRes qw(usleep);
# Create and test a standby from given backup, with a certain recovery target.
# Choose $until_lsn later than the transaction commit that causes the row
@@ -26,6 +27,8 @@ sub test_recovery_standby
$node_standby->append_conf('postgresql.conf', qq($param_item));
}
+ $node_standby->append_conf('postgresql.conf', "recovery_target_action = promote");
+
$node_standby->start;
# Wait until standby has replayed enough data
@@ -39,6 +42,15 @@ sub test_recovery_standby
$node_standby->safe_psql('postgres', "SELECT count(*) FROM tab_int");
is($result, qq($num_rows), "check standby content for $test_name");
+ # check that mode was promoted
+ if ($test_name ne 'multiple overriding settings')
+ {
+ $node_standby->safe_psql('postgres', "SELECT pg_sleep(1)");
+ my $is_in_recovery =
+ $node_standby->safe_psql('postgres', "SELECT pg_is_in_recovery()");
+ is($is_in_recovery, 'f', "check that standby was promoted for $test_name");
+ }
+
# Stop standby node
$node_standby->teardown_node;
@@ -77,7 +89,7 @@ my $lsn3 =
$node_master->safe_psql('postgres', "SELECT pg_current_wal_lsn();");
my $recovery_time = $node_master->safe_psql('postgres', "SELECT now()");
-# Even more data, this time with a recovery target name
+# More data, with a recovery target name
$node_master->safe_psql('postgres',
"INSERT INTO tab_int VALUES (generate_series(3001,4000))");
my $recovery_name = "my_target";
@@ -86,14 +98,17 @@ my $lsn4 =
$node_master->safe_psql('postgres',
"SELECT pg_create_restore_point('$recovery_name');");
-# And now for a recovery target LSN
+# Even more data, this time with a recovery target LSN
$node_master->safe_psql('postgres',
"INSERT INTO tab_int VALUES (generate_series(4001,5000))");
my $lsn5 = my $recovery_lsn =
$node_master->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
+# And now for a recovery target 'latest'
$node_master->safe_psql('postgres',
"INSERT INTO tab_int VALUES (generate_series(5001,6000))");
+my $lsn6 =
+ $node_master->safe_psql('postgres', "SELECT pg_current_wal_lsn();");
# Force archiving of WAL file
$node_master->safe_psql('postgres', "SELECT pg_switch_wal()");
@@ -114,6 +129,9 @@ test_recovery_standby('name', 'standby_4', $node_master, \@recovery_params,
@recovery_params = ("recovery_target_lsn = '$recovery_lsn'");
test_recovery_standby('LSN', 'standby_5', $node_master, \@recovery_params,
"5000", $lsn5);
+@recovery_params = ("recovery_target = 'latest'");
+test_recovery_standby('latest target', 'standby_6', $node_master, \@recovery_params,
+ "6000", $lsn6);
# Multiple targets
#
@@ -126,9 +144,9 @@ test_recovery_standby('LSN', 'standby_5', $node_master, \@recovery_params,
"recovery_target_name = ''",
"recovery_target_time = '$recovery_time'");
test_recovery_standby('multiple overriding settings',
- 'standby_6', $node_master, \@recovery_params, "3000", $lsn3);
+ 'standby_7', $node_master, \@recovery_params, "3000", $lsn3);
-my $node_standby = get_new_node('standby_7');
+my $node_standby = get_new_node('standby_8');
$node_standby->init_from_backup($node_master, 'my_backup',
has_restoring => 1);
$node_standby->append_conf(
