On Thu, Mar 23, 2023 at 04:35:46PM -0400, Tom Lane wrote:
> So I'm still pretty desperately unhappy with count_leaf_partitions.
> I don't like expending significant cost purely for progress tracking
> purposes, I don't like the undocumented assumption that NoLock is
> safe, and what's more, if it is safe then we've already traversed
> the inheritance tree to lock everything so in principle we could
> have the count already. However, it does seem like getting that
> knowledge from point A to point B would be a mess in most places.
>
> One thing we could do to reduce the cost (and improve the safety)
> is to forget the idea of checking the relkinds and just set the
> PARTITIONS_TOTAL count to list_length() of the find_all_inheritors
> result.
Actually list_length() minus 1 ...
> We've already agreed that it's okay if the PARTITIONS_DONE
> count never reaches PARTITIONS_TOTAL, so this would just be taking
> that idea further. (Or we could increment PARTITIONS_DONE for
> non-leaf partitions when we visit them, thus making that TOTAL
> more nearly correct.)
Yes, I think that's actually more correct. If TOTAL is set without
regard to relkind, then DONE ought to be set the same way.
I updated the documentation to indicate that the counters include the
intermediate partitioned rels, but I wonder if it's better to say
nothing and leave that undefined.
> Furthermore, as things stand it's not hard
> for PARTITIONS_TOTAL to be zero --- there's at least one such case
> in the regression tests --- and that seems just weird to me.
I don't know why it'd seem weird. postgres doesn't create partitions
automatically, so by default there are none. If we create a table but
never load any data, it'll have no partitions. Also, the TOTAL=0 case
won't go away just because we start counting intermediate partitions.
> By the by, this is awful code:
>
> + if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
>
> Consult the definition of RELKIND_HAS_STORAGE to see why.
> But I want to get rid of that rather than fixing it.
Good point, but I'd burden-shift the blame to RELKIND_HAS_STORAGE().
BTW, I promoted myself to a co-author of the patch. My interest here is
to resolve this hoping to allow the CIC patch to progress.
--
Justin
>From b48837a934ef1381d26ef62bbaf66f5e52b0cd6d Mon Sep 17 00:00:00 2001
From: Ilya Gladyshev <ilya.v.gladys...@gmail.com>
Date: Tue, 31 Jan 2023 19:13:07 +0400
Subject: [PATCH 1/3] fix CREATE INDEX progress report with nested partitions
The progress reporting was added in v12 (ab0dfc961) but the original
patch didn't seem to consider the possibility of nested partitioning.
When called recursively, DefineIndex() would clobber the number of
completed partitions, and it was possible to end up with the TOTAL
counter greater than the DONE counter.
This clarifies/re-defines that the progress reporting counts both direct
and indirect children, as well as intermediate partitioned tables:
- The TOTAL counter is set once at the start of the command.
- For indexes which are newly-built, the recursively-called
DefineIndex() increments the DONE counter.
- For pre-existing indexes which are ATTACHed rather than built,
DefineIndex() increments the DONE counter, unless the attached index is
partitioned, in which case progress report is not updated.
Author: Ilya Gladyshev, Justin Pryzby
Reviewed-By: Tomas Vondra, Dean Rasheed, Alvaro Herrera, Matthias van de Meent, Tom Lane
Discussion: https://www.postgresql.org/message-id/flat/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com
---
doc/src/sgml/monitoring.sgml | 10 +++-
src/backend/bootstrap/bootparse.y | 2 +
src/backend/commands/indexcmds.c | 60 +++++++++++++++++--
src/backend/commands/tablecmds.c | 4 +-
src/backend/tcop/utility.c | 6 +-
src/backend/utils/activity/backend_progress.c | 28 +++++++++
src/include/commands/defrem.h | 1 +
src/include/utils/backend_progress.h | 1 +
8 files changed, 103 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 7ab4424bf13..cf88dca53f3 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6929,7 +6929,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the total number of partitions on which the index is to be created.
+ the total number of partitions on which the index is to be created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, and includes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
@@ -6940,7 +6943,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the number of partitions on which the index has been created.
+ the number of partitions on which the index has been created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, and includes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index 86804bb598e..81a1b7bfec3 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -306,6 +306,7 @@ Boot_DeclareIndexStmt:
$4,
InvalidOid,
InvalidOid,
+ -1,
false,
false,
false,
@@ -358,6 +359,7 @@ Boot_DeclareUniqueIndexStmt:
$5,
InvalidOid,
InvalidOid,
+ -1,
false,
false,
false,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index ff48f44c66f..edb65d4cacb 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -521,6 +521,7 @@ WaitForOlderSnapshots(TransactionId limitXmin, bool progress)
* case the caller had better have checked it earlier.
* 'skip_build': make the catalog entries but don't create the index files
* 'quiet': suppress the NOTICE chatter ordinarily provided for constraints.
+ * 'total_parts': total number of direct and indirect partitions
*
* Returns the object address of the created index.
*/
@@ -530,6 +531,7 @@ DefineIndex(Oid relationId,
Oid indexRelationId,
Oid parentIndexId,
Oid parentConstraintId,
+ int total_parts,
bool is_alter_table,
bool check_rights,
bool check_not_in_use,
@@ -1225,8 +1227,30 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
- nparts);
+ /*
+ * Set the total number of partitions at the start of the command;
+ * but don't update it when being called recursively.
+ */
+ if (!OidIsValid(parentIndexId))
+ {
+ /*
+ * When called by ProcessUtilitySlow(), the number of
+ * partitions is passed in as an optimization. This should
+ * count partitions the same way. Subtract one since
+ * find_all_inheritors() includes the rel itself.
+ */
+ if (total_parts < 0)
+ {
+ List *childs = find_all_inheritors(relationId,
+ NoLock, NULL);
+
+ total_parts = list_length(childs) - 1;
+ list_free(childs);
+ }
+
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ total_parts);
+ }
/* Make a local copy of partdesc->oids[], just for safety */
memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
@@ -1432,14 +1456,23 @@ DefineIndex(Oid relationId,
InvalidOid, /* no predefined OID */
indexRelationId, /* this is our child */
createdConstraintId,
+ -1,
is_alter_table, check_rights, check_not_in_use,
skip_build, quiet);
SetUserIdAndSecContext(child_save_userid,
child_save_sec_context);
}
+ else
+ {
+ /*
+ * If a pre-existing index was ATTACHed, the progress
+ * report is updated here. A partitioned index is
+ * likewise counted when attached, but not its partitions,
+ * since that's expensive, and ATTACH is fast anyway.
+ */
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
+ }
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
- i + 1);
free_attrmap(attmap);
}
@@ -1479,6 +1512,16 @@ DefineIndex(Oid relationId,
table_close(rel, NoLock);
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ {
+ /*
+ * Update progress for a partitioned index itself; the
+ * recursively-called function will have updated the counter for
+ * its child indexes.
+ */
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
+ }
+
return address;
}
@@ -1490,9 +1533,16 @@ DefineIndex(Oid relationId,
/* Close the heap and we're done, in the non-concurrent case */
table_close(rel, NoLock);
- /* If this is the top-level index, we're done. */
+ /*
+ * If this is the top-level index, the command is done. When called
+ * recursively for child tables, the done partition counter is
+ * incremented now, rather than in the caller, to provide fine-grained
+ * progress reporting in the case of intermediate partitioning.
+ */
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
return address;
}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3e2c5f797cd..cc74d703f91 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -1216,6 +1216,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
InvalidOid,
RelationGetRelid(idxRel),
constraintOid,
+ -1,
false, false, false, false, false);
index_close(idxRel, AccessShareLock);
@@ -8640,6 +8641,7 @@ ATExecAddIndex(AlteredTableInfo *tab, Relation rel,
InvalidOid, /* no predefined OID */
InvalidOid, /* no parent index */
InvalidOid, /* no parent constraint */
+ -1, /* total_parts unknown */
true, /* is_alter_table */
check_rights,
false, /* check_not_in_use - we did it already */
@@ -18105,7 +18107,7 @@ AttachPartitionEnsureIndexes(Relation rel, Relation attachrel)
&conOid);
DefineIndex(RelationGetRelid(attachrel), stmt, InvalidOid,
RelationGetRelid(idxRel),
- conOid,
+ conOid, -1,
true, false, false, false, false);
}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index eada7353639..2dc92c72cea 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -1465,6 +1465,7 @@ ProcessUtilitySlow(ParseState *pstate,
Oid relid;
LOCKMODE lockmode;
bool is_alter_table;
+ int nparts = 0;
if (stmt->concurrent)
PreventInTransactionBlock(isTopLevel,
@@ -1503,9 +1504,11 @@ ProcessUtilitySlow(ParseState *pstate,
List *inheritors = NIL;
inheritors = find_all_inheritors(relid, lockmode, NULL);
+ nparts = list_length(inheritors) - 1;
foreach(lc, inheritors)
{
- char relkind = get_rel_relkind(lfirst_oid(lc));
+ Oid partrelid = lfirst_oid(lc);
+ char relkind = get_rel_relkind(partrelid);
if (relkind != RELKIND_RELATION &&
relkind != RELKIND_MATVIEW &&
@@ -1548,6 +1551,7 @@ ProcessUtilitySlow(ParseState *pstate,
InvalidOid, /* no predefined OID */
InvalidOid, /* no parent index */
InvalidOid, /* no parent constraint */
+ nparts,
is_alter_table,
true, /* check_rights */
true, /* check_not_in_use */
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index d96af812b19..2a9994b98fd 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -58,6 +58,34 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*-----------
+ * pgstat_progress_incr_param() -
+ *
+ * Increment index'th member in st_progress_param[] of the current backend.
+ *-----------
+ */
+void
+pgstat_progress_incr_param(int index, int64 incr)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ int64 val;
+
+ Assert(index >= 0 && index < PGSTAT_NUM_PROGRESS_PARAM);
+
+ if (!beentry || !pgstat_track_activities)
+ return;
+
+ /*
+ * Because no other process should write to this backend's own status, we
+ * can read its value from shared memory without needing to loop to ensure
+ * its consistency.
+ */
+ val = beentry->st_progress_param[index];
+ val += incr;
+
+ pgstat_progress_update_param(index, val);
+}
+
/*-----------
* pgstat_progress_update_multi_param() -
*
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 4f7f87fc62c..478203ed4c4 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -29,6 +29,7 @@ extern ObjectAddress DefineIndex(Oid relationId,
Oid indexRelationId,
Oid parentIndexId,
Oid parentConstraintId,
+ int total_parts,
bool is_alter_table,
bool check_rights,
bool check_not_in_use,
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 005e5d75ab6..a84752ade99 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -36,6 +36,7 @@ typedef enum ProgressCommandType
extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
Oid relid);
extern void pgstat_progress_update_param(int index, int64 val);
+extern void pgstat_progress_incr_param(int index, int64 incr);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
--
2.34.1
>From b9bfc805aa17e07ab7a92b6478daa9ff09b2b8e6 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Wed, 1 Feb 2023 10:23:53 -0600
Subject: [PATCH 2/3] assertions for progress reporting
---
src/backend/commands/analyze.c | 10 ++-
src/backend/utils/activity/backend_progress.c | 84 +++++++++++++++++++
2 files changed, 92 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 65750958bb2..3bfc941aa2c 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1145,6 +1145,12 @@ acquire_sample_rows(Relation onerel, int elevel,
TableScanDesc scan;
BlockNumber nblocks;
BlockNumber blksdone = 0;
+ int64 progress_vals[2] = {0};
+ int const progress_inds[2] = {
+ PROGRESS_ANALYZE_BLOCKS_DONE,
+ PROGRESS_ANALYZE_BLOCKS_TOTAL
+ };
+
#ifdef USE_PREFETCH
int prefetch_maximum = 0; /* blocks to prefetch if enabled */
BlockSamplerData prefetch_bs;
@@ -1169,8 +1175,8 @@ acquire_sample_rows(Relation onerel, int elevel,
#endif
/* Report sampling block numbers */
- pgstat_progress_update_param(PROGRESS_ANALYZE_BLOCKS_TOTAL,
- nblocks);
+ progress_vals[1] = nblocks;
+ pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
/* Prepare for sampling rows */
reservoir_init_selection_state(&rstate, targrows);
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 2a9994b98fd..63f9482b175 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -10,6 +10,7 @@
*/
#include "postgres.h"
+#include "commands/progress.h"
#include "port/atomics.h" /* for memory barriers */
#include "utils/backend_progress.h"
#include "utils/backend_status.h"
@@ -37,6 +38,85 @@ pgstat_progress_start_command(ProgressCommandType cmdtype, Oid relid)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Check for consistency of progress data (current < total).
+ *
+ * Check during pgstat_progress_updates_*() rather than only from
+ * pgstat_progress_end_command() to catch issues with uninitialized/stale data
+ * from previous progress commands.
+ *
+ * If a command fails due to interrupt or error, the values may be less than
+ * the expected final value.
+ */
+static void
+pgstat_progress_asserts(void)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ volatile int64 *a = beentry->st_progress_param;
+
+ switch (beentry->st_progress_command)
+ {
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_VACUUMED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_NUM_DEAD_TUPLES] <=
+ a[PROGRESS_VACUUM_MAX_DEAD_TUPLES]);
+ break;
+
+ case PROGRESS_COMMAND_ANALYZE:
+ Assert(a[PROGRESS_ANALYZE_BLOCKS_DONE] <=
+ a[PROGRESS_ANALYZE_BLOCKS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_EXT_STATS_COMPUTED] <=
+ a[PROGRESS_ANALYZE_EXT_STATS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_CHILD_TABLES_DONE] <=
+ a[PROGRESS_ANALYZE_CHILD_TABLES_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ Assert(a[PROGRESS_CLUSTER_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_CLUSTER_TOTAL_HEAP_BLKS]);
+ /* FALLTHROUGH */
+ /* ..because CLUSTER rebuilds indexes */
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ Assert(a[PROGRESS_CREATEIDX_TUPLES_DONE] <=
+ a[PROGRESS_CREATEIDX_TUPLES_TOTAL]);
+ Assert(a[PROGRESS_CREATEIDX_PARTITIONS_DONE] <=
+ a[PROGRESS_CREATEIDX_PARTITIONS_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ /* progress reporting is optional for these */
+ if (a[PROGRESS_BASEBACKUP_BACKUP_TOTAL] >= 0)
+ {
+ Assert(a[PROGRESS_BASEBACKUP_BACKUP_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_BACKUP_TOTAL]);
+ Assert(a[PROGRESS_BASEBACKUP_TBLSPC_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_TBLSPC_TOTAL]);
+ }
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+#if 0
+ //This currently fails file_fdw tests, since pgstat_prorgress evidently fails
+ // to support simultaneous copy commands, as happens during JOIN.
+ /* bytes progress is not available in all cases */
+ if (a[PROGRESS_COPY_BYTES_TOTAL] > 0)
+ //Assert(a[PROGRESS_COPY_BYTES_PROCESSED] <= a[PROGRESS_COPY_BYTES_TOTAL]);
+ if (a[PROGRESS_COPY_BYTES_PROCESSED] > a[PROGRESS_COPY_BYTES_TOTAL])
+ elog(WARNING, "PROGRESS_COPY_BYTES_PROCESSED %ld %ld",
+ a[PROGRESS_COPY_BYTES_PROCESSED],
+ a[PROGRESS_COPY_BYTES_TOTAL]);
+#endif
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break; /* Do nothing */
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -56,6 +136,8 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
@@ -113,6 +195,8 @@ pgstat_progress_update_multi_param(int nparam, const int *index,
}
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
--
2.34.1
>From 68403c5123a0b475affbf89626f7364adce1df24 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sat, 21 Jan 2023 21:41:04 -0600
Subject: [PATCH 3/3] f! also assert that progress values don't go backwards
and the total is constant
pgstat_progress_update_multi_param() is being abused as a way to update
a value which might otherwise violate the assertion.
See also:
https://www.postgresql.org/message-id/CA%2BTgmoYSvEP3weQaCPGf6%2BDXLy2__JbJUYtoXyWP%3DqHcyGbihA%40mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 37 +++++++++
src/backend/storage/lmgr/lmgr.c | 24 +++---
src/backend/utils/activity/backend_progress.c | 81 +++++++++++++++++++
3 files changed, 130 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f38..61cfbf6a17a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1042,6 +1042,13 @@ lazy_scan_heap(LVRelState *vacrel)
/* Forget the LP_DEAD items that we just vacuumed */
dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -2199,6 +2206,13 @@ lazy_vacuum(LVRelState *vacrel)
{
Assert(!vacrel->do_index_cleanup);
vacrel->dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
return;
}
@@ -2301,6 +2315,13 @@ lazy_vacuum(LVRelState *vacrel)
* vacuum)
*/
vacrel->dead_items->num_items = 0;
+
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
}
/*
@@ -2414,12 +2435,23 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+#if 0
+ const int progress_inds[] = {
+ PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+ };
+ const int64 progress_vals[] = {
+ PROGRESS_VACUUM_PHASE_VACUUM_HEAP,
+ 0,
+ };
+#endif
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
Assert(vacrel->num_index_scans > 0);
/* Report that we are now vacuuming the heap */
+ //pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
@@ -3190,7 +3222,12 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
dead_items->max_items = max_items;
dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
vacrel->dead_items = dead_items;
}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index ee9b89a6726..a3acdce5ff5 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -912,6 +912,13 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
int total = 0;
int done = 0;
+ const int progress_index[] = {
+ PROGRESS_WAITFOR_TOTAL,
+ PROGRESS_WAITFOR_DONE,
+ PROGRESS_WAITFOR_CURRENT_PID
+ };
+ const int64 progress_values[] = {0, 0, 0};
+
/* Done if no locks to wait for */
if (locktags == NIL)
return;
@@ -930,7 +937,10 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
}
if (progress)
+ {
+ pgstat_progress_update_multi_param(3, progress_index, progress_values);
pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+ }
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
@@ -960,19 +970,9 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
}
}
+
if (progress)
- {
- const int index[] = {
- PROGRESS_WAITFOR_TOTAL,
- PROGRESS_WAITFOR_DONE,
- PROGRESS_WAITFOR_CURRENT_PID
- };
- const int64 values[] = {
- 0, 0, 0
- };
-
- pgstat_progress_update_multi_param(3, index, values);
- }
+ pgstat_progress_update_multi_param(3, progress_index, progress_values);
list_free_deep(holders);
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 63f9482b175..41ad6884521 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -117,6 +117,78 @@ pgstat_progress_asserts(void)
}
}
+/*
+ * Check that newval >= oldval, and that when total is not being set twice.
+ */
+static void
+pgstat_progress_assert_forward_progress(int command, int index,
+ int64 oldval, int64 newval)
+{
+ switch (command)
+ {
+ case PROGRESS_COMMAND_ANALYZE:
+ /*
+ * phase goes backwards for inheritance tables, which are sampled
+ * twice
+ */
+ if (index != PROGRESS_ANALYZE_CURRENT_CHILD_TABLE_RELID &&
+ index != PROGRESS_ANALYZE_PHASE)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_ANALYZE_BLOCKS_TOTAL ||
+ index == PROGRESS_ANALYZE_EXT_STATS_TOTAL ||
+ index == PROGRESS_ANALYZE_CHILD_TABLES_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ if (index != PROGRESS_CLUSTER_INDEX_RELID)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_CLUSTER_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ if (index != PROGRESS_CREATEIDX_INDEX_OID &&
+ index != PROGRESS_CREATEIDX_SUBPHASE &&
+ index != PROGRESS_WAITFOR_CURRENT_PID &&
+ index != PROGRESS_CREATEIDX_ACCESS_METHOD_OID)
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_CREATEIDX_TUPLES_TOTAL ||
+ index == PROGRESS_CREATEIDX_PARTITIONS_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL &&
+ oldval == 0 && newval == -1)
+ return; /* Do nothing: this is the initial "null"
+ * state before the size is estimated */
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL ||
+ index == PROGRESS_BASEBACKUP_TBLSPC_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_COPY_BYTES_TOTAL)
+ Assert(oldval == 0);
+ break;
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_VACUUM_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ if (index == PROGRESS_VACUUM_MAX_DEAD_TUPLES)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break;
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -133,6 +205,15 @@ pgstat_progress_update_param(int index, int64 val)
if (!beentry || !pgstat_track_activities)
return;
+ if (index != PROGRESS_SCAN_BLOCKS_DONE)
+ {
+ /* Check that progress does not go backwards */
+ int64 oldval = beentry->st_progress_param[index];
+
+ pgstat_progress_assert_forward_progress(beentry->st_progress_command,
+ index, oldval, val);
+ }
+
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
--
2.34.1
