Hi,
We have addressed the O(n^2) problem which involves directory scan for
archiving individual WAL files by maintaining a WAL counter to identify
the next WAL file in a sequence.
WAL archiver scans the status directory to identify the next WAL file
which needs to be archived. This directory scan can be minimized by
maintaining the log segment number of the current file which is being
archived
and incrementing it by '1' to get the next WAL file in a sequence. Archiver
can check the availability of the next file in status directory and in case
if the
file is not available then it should fall-back to directory scan to get the
oldest
WAL file.
Please find attached patch v1.
Thanks,
Dipesh
On Fri, May 7, 2021 at 1:31 AM Andres Freund <and...@anarazel.de> wrote:
> Hi,
>
> On 2021-05-06 21:23:36 +0200, Hannu Krosing wrote:
> > How are you envisioning the shared-memory signaling should work in the
> > original sample case, where the archiver had been failing for half a
> > year ?
>
> If we leave history files and gaps in the .ready sequence aside for a
> second, we really only need an LSN or segment number describing the
> current "archive position". Then we can iterate over the segments
> between the "archive position" and the flush position (which we already
> know). Even if we needed to keep statting .ready/.done files (to handle
> gaps due to archive command mucking around with .ready/done), it'd still
> be a lot cheaper than what we do today. It probably would even still be
> cheaper if we just statted all potentially relevant timeline history
> files all the time to send them first.
>
>
> > Or should we perhaps have a system table for ready-to-archive WAL
> > files to get around limitation sof file system to return just the
> > needed files with ORDER BY ... LIMIT as we already know how to make
> > lookups in database fast ?
>
> Archiving needs to work on a standby so that doesn't seem like an
> option.
>
> Regards,
>
> Andres Freund
>
>
>
From ce5591c7ef574a1b039a5de807eb0c566a3456e1 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit <dipesh.pan...@enterprisedb.com>
Date: Wed, 30 Jun 2021 14:05:58 +0530
Subject: [PATCH] mitigate directory scan for WAL archiver
WAL archiver scans the status directory to identify the next WAL file
which needs to be archived. This directory scan can be minimized by
maintaining the log segment number of current file which is being
archived and incrementing it by '1' to get the next WAL file.
Archiver can check the availability of next file and in case if the
file is not available then it should fall-back to directory scan to
get the oldest WAL file.
---
src/backend/postmaster/pgarch.c | 63 ++++++++++++++++++++++++++++++++++++-----
1 file changed, 56 insertions(+), 7 deletions(-)
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 74a7d7c..eb219d7 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -87,6 +87,12 @@ static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
/*
+ * Log segment number and timeline ID to get next WAL file in a sequence.
+ */
+static XLogSegNo nextLogSegNo = 0;
+static TimeLineID curFileTLI = 0;
+
+/*
* Flags set by interrupt handlers for later service in the main loop.
*/
static volatile sig_atomic_t ready_to_stop = false;
@@ -411,6 +417,10 @@ pgarch_ArchiverCopyLoop(void)
/* successful */
pgarch_archiveDone(xlog);
+ /* Increment log segment number to point to the next WAL file */
+ if (!IsTLHistoryFileName(xlog))
+ nextLogSegNo++;
+
/*
* Tell the collector about the WAL file that we successfully
* archived
@@ -596,29 +606,55 @@ pgarch_archiveXlog(char *xlog)
* larger ID; the net result being that past timelines are given higher
* priority for archiving. This seems okay, or at least not obviously worth
* changing.
+ *
+ * WAL files are generated in a specific order of log segment number. The
+ * directory scan for each WAL file can be minimized by identifying the next
+ * WAL file in the sequence. This can be achieved by maintaining log segment
+ * number and timeline ID corresponding to WAL file currently being archived.
+ * The log segment number of current WAL file can be incremented by '1' upon
+ * successful archival to point to the next WAL file.
*/
static bool
pgarch_readyXlog(char *xlog)
{
- /*
- * open xlog status directory and read through list of xlogs that have the
- * .ready suffix, looking for earliest file. It is possible to optimise
- * this code, though only a single file is expected on the vast majority
- * of calls, so....
- */
+ char basename[MAX_XFN_CHARS + 1];
+ char xlogready[MAXPGPATH];
char XLogArchiveStatusDir[MAXPGPATH];
DIR *rldir;
struct dirent *rlde;
+ struct stat st;
bool found = false;
bool historyFound = false;
+ /*
+ * Log segment number already points to the next file in the sequence
+ * (as part of successful archival of the previous file). Generate the path
+ * for status file.
+ */
+ XLogFileName(basename, curFileTLI, nextLogSegNo, wal_segment_size);
+ StatusFilePath(xlogready, basename, ".ready");
+
+ /* Check availability of the status file */
+ if (stat(xlogready, &st) == 0)
+ {
+ strcpy(xlog, basename);
+ return true;
+ }
+
+ /*
+ * Fall-back to directory scan
+ *
+ * open xlog status directory and read through list of xlogs that have the
+ * .ready suffix, looking for earliest file. It is possible to optimise
+ * this code, though only a single file is expected on the vast majority
+ * of calls, so....
+ */
snprintf(XLogArchiveStatusDir, MAXPGPATH, XLOGDIR "/archive_status");
rldir = AllocateDir(XLogArchiveStatusDir);
while ((rlde = ReadDir(rldir, XLogArchiveStatusDir)) != NULL)
{
int basenamelen = (int) strlen(rlde->d_name) - 6;
- char basename[MAX_XFN_CHARS + 1];
bool ishistory;
/* Ignore entries with unexpected number of characters */
@@ -661,6 +697,19 @@ pgarch_readyXlog(char *xlog)
strcpy(xlog, basename);
}
}
+
+ /*
+ * Found the oldest WAL, reset timeline ID and log segment number to generate
+ * the next WAL file in the sequence.
+ */
+ if (found && !historyFound)
+ {
+ XLogFromFileName(xlog, &curFileTLI, &nextLogSegNo, wal_segment_size);
+ ereport(LOG,
+ (errmsg("directory scan to archive write-ahead log file \"%s\"",
+ xlog)));
+ }
+
FreeDir(rldir);
return found;
--
1.8.3.1
