On Mon, 22 Feb 2021 at 05:49, Bharath Rupireddy
<bharath.rupireddyforpostg...@gmail.com> wrote:
>
> On Mon, Feb 22, 2021 at 12:40 AM Matthias van de Meent
> <boekewurm+postg...@gmail.com> wrote:
> >
> > On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy
> > <bharath.rupireddyforpostg...@gmail.com> wrote:
> > >
> > > For COPY TO the name "source_type" column and for COPY FROM the name
> > > "destination_type" makes sense. To have a combined column name for
> > > both, how about naming that column as "io_type"?
> >
> > Thank you, that's way better! PFA what I believe is a finalized
> > patchset v9, utilizing io_type terminology instead of io_target.
>
> Thanks for the patches. I reviewed them.
>
> 0001 - I think there's a bug. See COPY TO stdout doesn't print
> io_type as "STDIO".
Fixed in attached
> 0003 - patch:
> I'm doubtful if the "bytes_total": 79 i.e. test file size will be the
> same across different platforms and file systems types, if true, then
> the below tests will not be stable. Do we also want to exclude the
> bytes_total from the output, just to be on the safer side? Thoughts?
I'm fairly certain that input files of the regression tests are
considered 'binary files' to the test framework and that contents
don't change between different architectures or OSes. I also think
that any POSIX-compliant file system would report anything but the
size of the file contents, i.e. the size of the blob that is the file,
and that is correctly reported here. Other than that, if bytes_total
wouldn't be stable, then bytes_processed wouldn't make sense either.
For STDIN / STDOUT you might also have a point (different input
methods might have different length encodings for the specified
input), but insofar that I understand the test framework and the
expected git configurations, the tests run using UTF-8 / ascii only,
with a single style of newlines[+]. Sadly, I cannot provide examples
nor outputs for other test framework settings due to my lack of
experience with running the tests with non-standard settings.
Note, I'm happy to be proven wrong here, in which case I don't
disagree, but according to my limited knowledge, these outputs should
be stable.
With regards,
Matthias van de Meent
[+] Except when explicitly configured to run using non-standard
configurations, in which case there are different expected output
values to be configured for that configuration.
From 3729dd30ac10af14b056db6c41d44f8c6b39b444 Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <boekewurm@gmail.com>
Date: Fri, 12 Feb 2021 14:06:44 +0100
Subject: [PATCH v10 1/3] Add progress-reported components for COPY progress
reporting
The command, io target and excluded tuple count (by the
COPY ... FROM ... WHERE -clause) are now reported in the pg_stat_progress_copy
view. Additionally, the column lines_processed is renamed to tuples_processed
to disambiguate the meaning of this column in cases of CSV and BINARY copies
and to stay consistent with regards to names in the pg_stat_progress_*-views.
Of special interest is the reporting of io_type, with which we can
distinguish logical replications' initial table synchronization workers'
progress without having to join the pg_stat_activity view.
---
doc/src/sgml/monitoring.sgml | 37 ++++++++++++++++++++++++++--
src/backend/catalog/system_views.sql | 11 ++++++++-
src/backend/commands/copyfrom.c | 27 +++++++++++++++++---
src/backend/commands/copyto.c | 18 ++++++++++++--
src/include/commands/progress.h | 15 ++++++++++-
src/test/regress/expected/rules.out | 15 ++++++++++-
6 files changed, 113 insertions(+), 10 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 3513e127b7..a21bee69d8 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6546,6 +6546,29 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>command</structfield> <type>text</type>
+ </para>
+ <para>
+ The command that is running: <literal>COPY FROM</literal>, or
+ <literal>COPY TO</literal>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>io_type</structfield> <type>text</type>
+ </para>
+ <para>
+ The io type that the data is read from or written to:
+ <literal>FILE</literal>, <literal>PROGRAM</literal>,
+ <literal>STDIO</literal> (for <command>COPY FROM STDIN</command> and
+ <command>COPY TO STDOUT</command>), or <literal>CALLBACK</literal>
+ (used in the table synchronization background worker).
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>bytes_processed</structfield> <type>bigint</type>
@@ -6567,10 +6590,20 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>lines_processed</structfield> <type>bigint</type>
+ <structfield>tuples_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of tuples already processed by <command>COPY</command> command.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_excluded</structfield> <type>bigint</type>
</para>
<para>
- Number of lines already processed by <command>COPY</command> command.
+ Number of tuples not processed because they were excluded by the
+ <command>WHERE</command> clause of the <command>COPY</command> command.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index fa58afd9d7..f96ad0a5dc 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1129,9 +1129,18 @@ CREATE VIEW pg_stat_progress_copy AS
SELECT
S.pid AS pid, S.datid AS datid, D.datname AS datname,
S.relid AS relid,
+ CASE S.param5 WHEN 1 THEN 'COPY FROM'
+ WHEN 2 THEN 'COPY TO'
+ END AS command,
+ CASE S.param6 WHEN 1 THEN 'FILE'
+ WHEN 2 THEN 'PROGRAM'
+ WHEN 3 THEN 'STDIO'
+ WHEN 4 THEN 'CALLBACK'
+ END AS io_type,
S.param1 AS bytes_processed,
S.param2 AS bytes_total,
- S.param3 AS lines_processed
+ S.param3 AS tuples_processed,
+ S.param4 AS tuples_excluded
FROM pg_stat_get_progress_info('COPY') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 796ca7b3f7..36828246c1 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -539,7 +539,8 @@ CopyFrom(CopyFromState cstate)
BulkInsertState bistate = NULL;
CopyInsertMethod insertMethod;
CopyMultiInsertInfo multiInsertInfo = {0}; /* pacify compiler */
- uint64 processed = 0;
+ int64 processed = 0;
+ int64 excluded = 0;
bool has_before_insert_row_trig;
bool has_instead_insert_row_trig;
bool leafpart_use_multi_insert = false;
@@ -869,7 +870,11 @@ CopyFrom(CopyFromState cstate)
econtext->ecxt_scantuple = myslot;
/* Skip items that don't match COPY's WHERE clause */
if (!ExecQual(cstate->qualexpr, econtext))
+ {
+ /* Report that this tuple was filtered out by the WHERE clause */
+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_EXCLUDED, ++excluded);
continue;
+ }
}
/* Determine the partition to insert the tuple into */
@@ -1107,7 +1112,7 @@ CopyFrom(CopyFromState cstate)
* for counting tuples inserted by an INSERT command. Update
* progress of the COPY command as well.
*/
- pgstat_progress_update_param(PROGRESS_COPY_LINES_PROCESSED, ++processed);
+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED, ++processed);
}
}
@@ -1200,6 +1205,16 @@ BeginCopyFrom(ParseState *pstate,
ExprState **defexprs;
MemoryContext oldcontext;
bool volatile_defexprs;
+ const int progress_cols[] = {
+ PROGRESS_COPY_COMMAND,
+ PROGRESS_COPY_IO_TYPE,
+ PROGRESS_COPY_BYTES_TOTAL
+ };
+ int64 progress_vals[] = {
+ PROGRESS_COPY_COMMAND_FROM,
+ 0,
+ 0
+ };
/* Allocate workspace and zero all fields */
cstate = (CopyFromStateData *) palloc0(sizeof(CopyFromStateData));
@@ -1437,11 +1452,13 @@ BeginCopyFrom(ParseState *pstate,
if (data_source_cb)
{
+ progress_vals[1] = PROGRESS_COPY_IO_TYPE_CALLBACK;
cstate->copy_src = COPY_CALLBACK;
cstate->data_source_cb = data_source_cb;
}
else if (pipe)
{
+ progress_vals[1] = PROGRESS_COPY_IO_TYPE_STDIO;
Assert(!is_program); /* the grammar does not allow this */
if (whereToSendOutput == DestRemote)
ReceiveCopyBegin(cstate);
@@ -1454,6 +1471,7 @@ BeginCopyFrom(ParseState *pstate,
if (cstate->is_program)
{
+ progress_vals[1] = PROGRESS_COPY_IO_TYPE_PROGRAM;
cstate->copy_file = OpenPipeStream(cstate->filename, PG_BINARY_R);
if (cstate->copy_file == NULL)
ereport(ERROR,
@@ -1465,6 +1483,7 @@ BeginCopyFrom(ParseState *pstate,
{
struct stat st;
+ progress_vals[1] = PROGRESS_COPY_IO_TYPE_FILE;
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
if (cstate->copy_file == NULL)
{
@@ -1491,10 +1510,12 @@ BeginCopyFrom(ParseState *pstate,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("\"%s\" is a directory", cstate->filename)));
- pgstat_progress_update_param(PROGRESS_COPY_BYTES_TOTAL, st.st_size);
+ progress_vals[2] = st.st_size;
}
}
+ pgstat_progress_update_multi_param(3, progress_cols, progress_vals);
+
if (cstate->opts.binary)
{
/* Read and verify binary header */
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index e04ec1e331..606f6f190c 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -394,6 +394,14 @@ BeginCopyTo(ParseState *pstate,
TupleDesc tupDesc;
int num_phys_attrs;
MemoryContext oldcontext;
+ const int progress_cols[] = {
+ PROGRESS_COPY_COMMAND,
+ PROGRESS_COPY_IO_TYPE
+ };
+ int64 progress_vals[] = {
+ PROGRESS_COPY_COMMAND_TO,
+ 0
+ };
if (rel != NULL && rel->rd_rel->relkind != RELKIND_RELATION)
{
@@ -700,6 +708,8 @@ BeginCopyTo(ParseState *pstate,
if (pipe)
{
+ progress_vals[1] = PROGRESS_COPY_IO_TYPE_STDIO;
+
Assert(!is_program); /* the grammar does not allow this */
if (whereToSendOutput != DestRemote)
cstate->copy_file = stdout;
@@ -711,6 +721,7 @@ BeginCopyTo(ParseState *pstate,
if (is_program)
{
+ progress_vals[1] = PROGRESS_COPY_IO_TYPE_PROGRAM;
cstate->copy_file = OpenPipeStream(cstate->filename, PG_BINARY_W);
if (cstate->copy_file == NULL)
ereport(ERROR,
@@ -723,6 +734,7 @@ BeginCopyTo(ParseState *pstate,
mode_t oumask; /* Pre-existing umask value */
struct stat st;
+ progress_vals[1] = PROGRESS_COPY_IO_TYPE_FILE;
/*
* Prevent write to relative path ... too easy to shoot oneself in
* the foot by overwriting a database file ...
@@ -772,6 +784,8 @@ BeginCopyTo(ParseState *pstate,
/* initialize progress */
pgstat_progress_start_command(PROGRESS_COMMAND_COPY,
cstate->rel ? RelationGetRelid(cstate->rel) : InvalidOid);
+ pgstat_progress_update_multi_param(2, progress_cols, progress_vals);
+
cstate->bytes_processed = 0;
MemoryContextSwitchTo(oldcontext);
@@ -954,7 +968,7 @@ CopyTo(CopyToState cstate)
CopyOneRowTo(cstate, slot);
/* Increment amount of processed tuples and update the progress */
- pgstat_progress_update_param(PROGRESS_COPY_LINES_PROCESSED, ++processed);
+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED, ++processed);
}
ExecDropSingleTupleTableSlot(slot);
@@ -1321,7 +1335,7 @@ copy_dest_receive(TupleTableSlot *slot, DestReceiver *self)
CopyOneRowTo(cstate, slot);
/* Increment amount of processed tuples and update the progress */
- pgstat_progress_update_param(PROGRESS_COPY_LINES_PROCESSED, ++myState->processed);
+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED, ++myState->processed);
return true;
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 95ec5d02e9..508b562d92 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -136,6 +136,19 @@
/* Commands of PROGRESS_COPY */
#define PROGRESS_COPY_BYTES_PROCESSED 0
#define PROGRESS_COPY_BYTES_TOTAL 1
-#define PROGRESS_COPY_LINES_PROCESSED 2
+#define PROGRESS_COPY_TUPLES_PROCESSED 2
+#define PROGRESS_COPY_TUPLES_EXCLUDED 3
+#define PROGRESS_COPY_COMMAND 4
+#define PROGRESS_COPY_IO_TYPE 5
+
+/* Commands of PROGRESS_COPY_COMMAND */
+#define PROGRESS_COPY_COMMAND_FROM 1
+#define PROGRESS_COPY_COMMAND_TO 2
+
+/* Types of PROGRESS_COPY_INOUT_TYPE */
+#define PROGRESS_COPY_IO_TYPE_FILE 1
+#define PROGRESS_COPY_IO_TYPE_PROGRAM 2
+#define PROGRESS_COPY_IO_TYPE_STDIO 3
+#define PROGRESS_COPY_IO_TYPE_CALLBACK 4
#endif
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 10a1f34ebc..a6ae73bd09 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1949,9 +1949,22 @@ pg_stat_progress_copy| SELECT s.pid,
s.datid,
d.datname,
s.relid,
+ CASE s.param5
+ WHEN 1 THEN 'COPY FROM'::text
+ WHEN 2 THEN 'COPY TO'::text
+ ELSE NULL::text
+ END AS command,
+ CASE s.param6
+ WHEN 1 THEN 'FILE'::text
+ WHEN 2 THEN 'PROGRAM'::text
+ WHEN 3 THEN 'STDIO'::text
+ WHEN 4 THEN 'CALLBACK'::text
+ ELSE NULL::text
+ END AS io_type,
s.param1 AS bytes_processed,
s.param2 AS bytes_total,
- s.param3 AS lines_processed
+ s.param3 AS tuples_processed,
+ s.param4 AS tuples_excluded
FROM (pg_stat_get_progress_info('COPY'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_create_index| SELECT s.pid,
--
2.20.1
From 82dfa694f48874d23f25a6d2673fefdd09010e9e Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <boekewurm@gmail.com>
Date: Fri, 5 Feb 2021 23:11:50 +0100
Subject: [PATCH v10 2/3] Add backlinks to progress reporting documentation
Previously, for most progress-reported features, the only place the
feature was mentioned is in the progress reporting document itself.
This makes the progress reporting more discoverable from the reported
commands.
---
doc/src/sgml/ref/analyze.sgml | 7 +++++++
doc/src/sgml/ref/cluster.sgml | 6 ++++++
doc/src/sgml/ref/copy.sgml | 14 ++++++++++++++
doc/src/sgml/ref/create_index.sgml | 7 +++++++
doc/src/sgml/ref/pg_basebackup.sgml | 1 +
doc/src/sgml/ref/reindex.sgml | 7 +++++++
doc/src/sgml/ref/vacuum.sgml | 11 +++++++++++
7 files changed, 53 insertions(+)
diff --git a/doc/src/sgml/ref/analyze.sgml b/doc/src/sgml/ref/analyze.sgml
index 7d816c87c6..e44a5e07c1 100644
--- a/doc/src/sgml/ref/analyze.sgml
+++ b/doc/src/sgml/ref/analyze.sgml
@@ -273,6 +273,12 @@ ANALYZE [ VERBOSE ] [ <replaceable class="parameter">table_and_columns</replacea
will not record new statistics for that table. Any existing statistics
will be retained.
</para>
+
+ <para>
+ Each backend running <command>ANALYZE</command> will report its progress to
+ the <structname>pg_stat_progress_analyze</structname> view. See
+ <xref linkend="analyze-progress-reporting"/> for details.
+ </para>
</refsect1>
<refsect1>
@@ -291,6 +297,7 @@ ANALYZE [ VERBOSE ] [ <replaceable class="parameter">table_and_columns</replacea
<member><xref linkend="app-vacuumdb"/></member>
<member><xref linkend="runtime-config-resource-vacuum-cost"/></member>
<member><xref linkend="autovacuum"/></member>
+ <member><xref linkend="analyze-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/cluster.sgml b/doc/src/sgml/ref/cluster.sgml
index 5dd21a0189..6dcf886e54 100644
--- a/doc/src/sgml/ref/cluster.sgml
+++ b/doc/src/sgml/ref/cluster.sgml
@@ -192,6 +192,11 @@ CLUSTER [VERBOSE]
are periodically reclustered.
</para>
+ <para>
+ Each backend running <command>CLUSTER</command> will report its progress to
+ the <structname>pg_stat_progress_cluster</structname> view. See
+ <xref linkend="cluster-progress-reporting"/> for details.
+ </para>
</refsect1>
<refsect1>
@@ -242,6 +247,7 @@ CLUSTER <replaceable class="parameter">index_name</replaceable> ON <replaceable
<simplelist type="inline">
<member><xref linkend="app-clusterdb"/></member>
+ <member><xref linkend="cluster-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 0fca6583af..bd2dd87c52 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -82,6 +82,12 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
specified, data is transmitted via the connection between the
client and the server.
</para>
+
+ <para>
+ Each backend running <command>COPY</command> will report its progress to
+ the <structname>pg_stat_progress_copy</structname> view. See
+ <xref linkend="copy-progress-reporting"/> for details.
+ </para>
</refsect1>
<refsect1>
@@ -1052,4 +1058,12 @@ COPY [ BINARY ] <replaceable class="parameter">table_name</replaceable>
[ WITH NULL AS '<replaceable class="parameter">null_string</replaceable>' ]
</synopsis></para>
</refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="copy-progress-reporting"/></member>
+ </simplelist>
+ </refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/src/sgml/ref/create_index.sgml
index a5271a9f8f..092372694d 100644
--- a/doc/src/sgml/ref/create_index.sgml
+++ b/doc/src/sgml/ref/create_index.sgml
@@ -865,6 +865,12 @@ Indexes:
will interpret it as <literal>USING gist</literal>, to simplify conversion
of old databases to GiST.
</para>
+
+ <para>
+ Each backend running <command>CREATE INDEX</command> will report its
+ progress to the <structname>pg_stat_progress_create_index</structname>
+ view. See <xref linkend="create-index-progress-reporting"/> for details.
+ </para>
</refsect1>
<refsect1>
@@ -978,6 +984,7 @@ CREATE INDEX CONCURRENTLY sales_quantity_index ON sales_table (quantity);
<member><xref linkend="sql-alterindex"/></member>
<member><xref linkend="sql-dropindex"/></member>
<member><xref linkend="sql-reindex"/></member>
+ <member><xref linkend="create-index-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 5754ad5aa6..14cc88a852 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -904,6 +904,7 @@ PostgreSQL documentation
<simplelist type="inline">
<member><xref linkend="app-pgdump"/></member>
+ <member><xref linkend="basebackup-progress-reporting"/></member>
</simplelist>
</refsect1>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index 07795b5737..67cdd83449 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -493,6 +493,12 @@ Indexes:
is reindexed concurrently, those indexes will be skipped. (It is possible
to reindex such indexes without the <command>CONCURRENTLY</command> option.)
</para>
+
+ <para>
+ Each backend running <command>REINDEX</command> will report its progress
+ to the <structname>pg_stat_progress_create_index</structname> view. See
+ <xref linkend="create-index-progress-reporting"/> for details.
+ </para>
</refsect2>
</refsect1>
@@ -551,6 +557,7 @@ REINDEX TABLE CONCURRENTLY my_broken_table;
<member><xref linkend="sql-createindex"/></member>
<member><xref linkend="sql-dropindex"/></member>
<member><xref linkend="app-reindexdb"/></member>
+ <member><xref linkend="create-index-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml
index 4bb624979b..08a40e94f7 100644
--- a/doc/src/sgml/ref/vacuum.sgml
+++ b/doc/src/sgml/ref/vacuum.sgml
@@ -393,6 +393,15 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
information about automatic and manual vacuuming, see
<xref linkend="routine-vacuuming"/>.
</para>
+ <para>
+ Each backend running <command>VACUUM</command> without the
+ <literal>FULL</literal> option will report its progress in the
+ <structname>pg_stat_progress_vacuum</structname> view. Backends running
+ <command>VACUUM</command> with the <literal>FULL</literal> option report
+ progress in the <structname>pg_stat_progress_cluster</structname> instead.
+ See <xref linkend="vacuum-progress-reporting"/> and
+ <xref linkend="cluster-progress-reporting"/> for details.
+ </para>
</refsect1>
<refsect1>
@@ -422,6 +431,8 @@ VACUUM (VERBOSE, ANALYZE) onek;
<member><xref linkend="app-vacuumdb"/></member>
<member><xref linkend="runtime-config-resource-vacuum-cost"/></member>
<member><xref linkend="autovacuum"/></member>
+ <member><xref linkend="vacuum-progress-reporting"/></member>
+ <member><xref linkend="cluster-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
--
2.20.1
From 6062d394a508bf117e666b794a9ff36c1ef0b0d3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Josef=20=C5=A0im=C3=A1nek?= <josef.simanek@gmail.com>
Date: Tue, 9 Feb 2021 13:06:45 +0100
Subject: [PATCH v10 3/3] Add copy progress reporting regression tests.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This tests some basic features of copy progress reporting.
Sadly, we can only request one snapshot of progress_reporting each
transaction / statement, so we can't (easily) get intermediate results without
each result requiring a seperate statement being run.
Author: Josef Šimánek, Matthias van de Meent
---
src/test/regress/input/copy.source | 60 +++++++++++++++++++++++++++++
src/test/regress/output/copy.source | 47 ++++++++++++++++++++++
2 files changed, 107 insertions(+)
diff --git a/src/test/regress/input/copy.source b/src/test/regress/input/copy.source
index a1d529ad36..4f1cbc73d2 100644
--- a/src/test/regress/input/copy.source
+++ b/src/test/regress/input/copy.source
@@ -201,3 +201,63 @@ select * from parted_copytest where b = 1;
select * from parted_copytest where b = 2;
drop table parted_copytest;
+
+--
+-- progress reporting
+--
+
+-- setup
+-- reuse employer datatype, that has a small sized data set
+
+create table progress_reporting (
+ name text,
+ age int4,
+ location point,
+ salary int4,
+ manager name
+);
+
+-- Add a trigger to 'raise info' the contents of pg_stat_progress_copy. This
+-- allows us to test and verify the contents of this view, which would
+-- otherwise require a concurrent session checking that table.
+create function notice_after_progress_reporting() returns trigger AS
+$$
+declare report record;
+begin
+ -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to
+ -- variance in system process ids, and concurrency in runs of tests.
+ -- Additionally, due to the usage of this test in pg_regress, the 'datid'
+ -- also is not consistent between runs.
+ select into report (to_jsonb(r) - '{pid,relid,datid}'::text[]) as value
+ from pg_stat_progress_copy r
+ where pid = pg_backend_pid();
+
+ raise info 'progress: %', report.value::text;
+
+ RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+create trigger check_after_progress_reporting
+ after insert on progress_reporting
+ for each statement
+ execute function notice_after_progress_reporting();
+
+-- reporting of STDIO imports, and correct bytes-processed/tuples-processed reporting
+
+copy progress_reporting from stdin;
+sharon 25 (15,12) 1000 sam
+sam 30 (10,5) 2000 bill
+bill 20 (11,10) 1000 sharon
+\.
+
+-- reporting of FILE imports, and correct reporting of tuples-excluded
+
+copy progress_reporting from '@abs_srcdir@/data/emp.data'
+ where (salary < 2000);
+
+-- cleanup progress_reporting
+
+drop trigger check_after_progress_reporting on progress_reporting;
+drop function notice_after_progress_reporting();
+drop table progress_reporting;
diff --git a/src/test/regress/output/copy.source b/src/test/regress/output/copy.source
index 938d3551da..f3596bdd15 100644
--- a/src/test/regress/output/copy.source
+++ b/src/test/regress/output/copy.source
@@ -165,3 +165,50 @@ select * from parted_copytest where b = 2;
(1 row)
drop table parted_copytest;
+--
+-- progress reporting
+--
+-- setup
+-- reuse employer datatype, that has a small sized data set
+create table progress_reporting (
+ name text,
+ age int4,
+ location point,
+ salary int4,
+ manager name
+);
+-- Add a trigger to 'raise info' the contents of pg_stat_progress_copy. This
+-- allows us to test and verify the contents of this view, which would
+-- otherwise require a concurrent session checking that table.
+create function notice_after_progress_reporting() returns trigger AS
+$$
+declare report record;
+begin
+ -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to
+ -- variance in system process ids, and concurrency in runs of tests.
+ -- Additionally, due to the usage of this test in pg_regress, the 'datid'
+ -- also is not consistent between runs.
+ select into report (to_jsonb(r) - '{pid,relid,datid}'::text[]) as value
+ from pg_stat_progress_copy r
+ where pid = pg_backend_pid();
+
+ raise info 'progress: %', report.value::text;
+
+ RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+create trigger check_after_progress_reporting
+ after insert on progress_reporting
+ for each statement
+ execute function notice_after_progress_reporting();
+-- reporting of STDIO imports, and correct bytes-processed/tuples-processed reporting
+copy progress_reporting from stdin;
+INFO: progress: {"command": "COPY FROM", "datname": "regression", "io_type": "STDIO", "bytes_total": 0, "bytes_processed": 79, "tuples_excluded": 0, "tuples_processed": 3}
+-- reporting of FILE imports, and correct reporting of tuples-excluded
+copy progress_reporting from '@abs_srcdir@/data/emp.data'
+ where (salary < 2000);
+INFO: progress: {"command": "COPY FROM", "datname": "regression", "io_type": "FILE", "bytes_total": 79, "bytes_processed": 79, "tuples_excluded": 1, "tuples_processed": 2}
+-- cleanup progress_reporting
+drop trigger check_after_progress_reporting on progress_reporting;
+drop function notice_after_progress_reporting();
+drop table progress_reporting;
--
2.20.1
