On Sat, Feb 2, 2019 at 7:00 PM Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > > On 2019-Feb-01, Bossart, Nathan wrote: > > > IMHO we could document this feature at a slightly higher level without > > leaving out any really important user-facing behavior. Here's a quick > > attempt to show what I am thinking: > > > > With this option, VACUUM skips all index cleanup behavior and > > only marks tuples as "dead" without reclaiming the storage. > > While this can help reclaim transaction IDs faster to avoid > > transaction ID wraparound (see Section 24.1.5), it will not > > reduce bloat. > > Hmm ... don't we compact out the storage for dead tuples? If we do (and > I think we should) then this wording is not entirely correct.
Yeah, we remove tuple and leave the dead ItemId. So we actually reclaim the almost tuple storage. > > > Note that this option is ignored for tables > > that have no indexes. Also, this option cannot be used in > > conjunction with the FULL option, since FULL requires > > rewriting the table. > > I would remove the "Also," in there, since it seems to me to give the > wrong impression about those two things being similar, but they're not: > one is convenient behavior, the other is a limitation. Agreed. > > > + /* Notify user that DISABLE_INDEX_CLEANUP option is ignored */ > > + if (!vacrelstats->hasindex && (options & > > VACOPT_DISABLE_INDEX_CLEANUP)) > > + ereport(NOTICE, > > + (errmsg("DISABLE_INDEX_CLEANUP is ignored > > because table \"%s\" does not have index", > > + > > RelationGetRelationName(onerel)))); > > > > It might make more sense to emit this in lazy_scan_heap() where we > > determine the value of skip_index_vacuum. > > Why do we need a NOTICE here? I think this case should be silent. > Okay, removed it. On Fri, Feb 1, 2019 at 11:43 PM Bossart, Nathan <bossa...@amazon.com> wrote: > > + if (skip_index_vacuum) > + ereport(elevel, > + (errmsg("\"%s\": marked %.0f row > versions as dead in %u pages", > + > RelationGetRelationName(onerel), > + tups_vacuumed, > vacuumed_pages))); > > IIUC tups_vacuumed will include tuples removed during HOT pruning, so > it could be inaccurate to say all of these tuples have only been > marked "dead." Perhaps we could keep a separate count of tuples > removed via HOT pruning in case we are using DISABLE_INDEX_CLEANUP. > There might be similar problems with the values stored in vacrelstats > that are used at the end of heap_vacuum_rel() (e.g. tuples_deleted). Yeah, tups_vacuumed include such tuples so the message is inaccurate. So I think that we should not change the message but we can report the dead item pointers we left in errdetail. That's more accurate and would help user. postgres(1:1130)=# vacuum (verbose, disable_index_cleanup) test; INFO: vacuuming "public.test" INFO: "test": removed 49 row versions in 1 pages INFO: "test": found 49 removable, 51 nonremovable row versions in 1 out of 1 pages DETAIL: 0 dead row versions cannot be removed yet, oldest xmin: 509 There were 0 unused item pointers. Skipped 0 pages due to buffer pins, 0 frozen pages. 0 pages are entirely empty. 49 tuples are left as dead. CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s. VACUUM Attached the updated patch and the patch for vacuumdb. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
From e309d7829572648157a5d7a8363e876aaef675ef Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 21 Jan 2019 19:07:44 +0900 Subject: [PATCH v6 1/2] Add DISABLE_INDEX_CLEANUP option to VACUUM command --- doc/src/sgml/ref/vacuum.sgml | 21 ++++++++++- src/backend/access/heap/vacuumlazy.c | 68 ++++++++++++++++++++++++++++-------- src/backend/commands/vacuum.c | 8 ++++- src/backend/parser/gram.y | 2 ++ src/include/nodes/parsenodes.h | 4 ++- src/test/regress/expected/vacuum.out | 3 ++ src/test/regress/sql/vacuum.sql | 3 ++ 7 files changed, 92 insertions(+), 17 deletions(-) diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml index fd911f5..f5cde2b 100644 --- a/doc/src/sgml/ref/vacuum.sgml +++ b/doc/src/sgml/ref/vacuum.sgml @@ -31,6 +31,7 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet VERBOSE ANALYZE DISABLE_PAGE_SKIPPING + DISABLE_INDEX_CLEANUP SKIP_LOCKED <phrase>and <replaceable class="parameter">table_and_columns</replaceable> is:</phrase> @@ -161,7 +162,25 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet </listitem> </varlistentry> - <varlistentry> + <varlistentry> + <term><literal>DISABLE_INDEX_CLEANUP</literal></term> + <listitem> + <para> + <command>VACUUM</command> removes dead tuples and prunes HOT-updated + tuples chain for live tuples on table. If the table has any dead tuple + it removes them from both table and indexes for re-use. With this + option <command>VACUUM</command> doesn't completely remove dead tuples + and disables removing dead tuples from indexes. This is suitable for + avoiding transaction ID wraparound (see + <xref linkend="vacuum-for-wraparound"/>) but not sufficient for avoiding + index bloat. This option is ignored if the table doesn't have index. + This cannot be used in conjunction with <literal>FULL</literal> + option. + </para> + </listitem> + </varlistentry> + + <varlistentry> <term><literal>SKIP_LOCKED</literal></term> <listitem> <para> diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 37aa484..9585beb 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -112,7 +112,10 @@ typedef struct LVRelStats { - /* hasindex = true means two-pass strategy; false means one-pass */ + /* + * hasindex = true means two-pass strategy; false means one-pass. But we + * always use the one-pass strategy when index vacuum is disabled. + */ bool hasindex; /* Overall statistics about rel */ BlockNumber old_rel_pages; /* previous value of pg_class.relpages */ @@ -167,7 +170,8 @@ static bool should_attempt_truncation(LVRelStats *vacrelstats); static void lazy_truncate_heap(Relation onerel, LVRelStats *vacrelstats); static BlockNumber count_nondeletable_pages(Relation onerel, LVRelStats *vacrelstats); -static void lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks); +static void lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks, + bool skip_index_vacuum); static void lazy_record_dead_tuple(LVRelStats *vacrelstats, ItemPointer itemptr); static bool lazy_tid_reaped(ItemPointer itemptr, void *state); @@ -485,13 +489,15 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats, live_tuples, /* live tuples (reltuples estimate) */ tups_vacuumed, /* tuples cleaned up by vacuum */ nkeep, /* dead-but-not-removable tuples */ - nunused; /* unused item pointers */ + nunused, /* unused item pointers */ + nleft; /* item pointers we left */ IndexBulkDeleteResult **indstats; int i; PGRUsage ru0; Buffer vmbuffer = InvalidBuffer; BlockNumber next_unskippable_block; bool skipping_blocks; + bool skip_index_vacuum; xl_heap_freeze_tuple *frozen; StringInfoData buf; const int initprog_index[] = { @@ -517,7 +523,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats, empty_pages = vacuumed_pages = 0; next_fsm_block_to_vacuum = (BlockNumber) 0; - num_tuples = live_tuples = tups_vacuumed = nkeep = nunused = 0; + num_tuples = live_tuples = tups_vacuumed = nkeep = nunused = nleft = 0; indstats = (IndexBulkDeleteResult **) palloc0(nindexes * sizeof(IndexBulkDeleteResult *)); @@ -529,7 +535,14 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats, vacrelstats->nonempty_pages = 0; vacrelstats->latestRemovedXid = InvalidTransactionId; - lazy_space_alloc(vacrelstats, nblocks); + /* + * Skip index vacuum if it's requested for table with indexes. In this + * case we use the one-pass strategy and don't remove tuple storage. + */ + skip_index_vacuum = + (options & VACOPT_DISABLE_INDEX_CLEANUP) != 0 && vacrelstats->hasindex; + + lazy_space_alloc(vacrelstats, nblocks, skip_index_vacuum); frozen = palloc(sizeof(xl_heap_freeze_tuple) * MaxHeapTuplesPerPage); /* Report that we're scanning the heap, advertising total # of blocks */ @@ -722,6 +735,8 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats, }; int64 hvp_val[2]; + Assert(!skip_index_vacuum); + /* * Before beginning index vacuuming, we release any pin we may * hold on the visibility map page. This isn't necessary for @@ -1203,12 +1218,28 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats, * If there are no indexes then we can vacuum the page right now * instead of doing a second scan. */ - if (nindexes == 0 && + if ((nindexes == 0 || skip_index_vacuum) && vacrelstats->num_dead_tuples > 0) { - /* Remove tuples from heap */ - lazy_vacuum_page(onerel, blkno, buf, 0, vacrelstats, &vmbuffer); - has_dead_tuples = false; + /* + * Remove tuples from heap if the table has no index. If the table + * has index but index vacuum is disabled, we don't vacuum and forget + * them. The vacrelstats->dead_tuples could have tuples which became + * dead after checked at HOT-pruning time which are handled by + * lazy_vacuum_page() but we don't worry about handling those because + * it's a very rare condition and these would not be a large number. + */ + if (nindexes == 0) + { + Assert(!skip_index_vacuum); + lazy_vacuum_page(onerel, blkno, buf, 0, vacrelstats, &vmbuffer); + has_dead_tuples = false; + } + else + { + /* Keep the number of tuples we left for the report */ + nleft += vacrelstats->num_dead_tuples; + } /* * Forget the now-vacuumed tuples, and press on, but be careful @@ -1373,6 +1404,8 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats, }; int64 hvp_val[2]; + Assert(!skip_index_vacuum); + /* Log cleanup info before we touch indexes */ vacuum_log_cleanup_info(onerel, vacrelstats); @@ -1411,10 +1444,11 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP); /* Do post-vacuum cleanup and statistics update for each index */ - for (i = 0; i < nindexes; i++) - lazy_cleanup_index(Irel[i], indstats[i], vacrelstats); + if (!skip_index_vacuum) + for (i = 0; i < nindexes; i++) + lazy_cleanup_index(Irel[i], indstats[i], vacrelstats); - /* If no indexes, make log report that lazy_vacuum_heap would've made */ + /* Make log report that lazy_vacuum_heap would've made */ if (vacuumed_pages) ereport(elevel, (errmsg("\"%s\": removed %.0f row versions in %u pages", @@ -1443,6 +1477,11 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats, "%u pages are entirely empty.\n", empty_pages), empty_pages); + if (skip_index_vacuum) + appendStringInfo(&buf, ngettext("%.0f tuple is left as dead.\n", + "%.0f tuples are left as dead.\n", + nleft), + nleft); appendStringInfo(&buf, _("%s."), pg_rusage_show(&ru0)); ereport(elevel, @@ -2078,14 +2117,15 @@ count_nondeletable_pages(Relation onerel, LVRelStats *vacrelstats) * See the comments at the head of this file for rationale. */ static void -lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks) +lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks, + bool skip_index_vacuum) { long maxtuples; int vac_work_mem = IsAutoVacuumWorkerProcess() && autovacuum_work_mem != -1 ? autovacuum_work_mem : maintenance_work_mem; - if (vacrelstats->hasindex) + if (vacrelstats->hasindex && !skip_index_vacuum) { maxtuples = (vac_work_mem * 1024L) / sizeof(ItemPointerData); maxtuples = Min(maxtuples, INT_MAX); diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index e91df21..00024dd 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -203,7 +203,8 @@ vacuum(int options, List *relations, VacuumParams *params, stmttype))); /* - * Sanity check DISABLE_PAGE_SKIPPING option. + * Sanity check DISABLE_PAGE_SKIPPING option and DISABLE_INDEX_CLEANUP + * option. */ if ((options & VACOPT_FULL) != 0 && (options & VACOPT_DISABLE_PAGE_SKIPPING) != 0) @@ -211,6 +212,11 @@ vacuum(int options, List *relations, VacuumParams *params, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("VACUUM option DISABLE_PAGE_SKIPPING cannot be used with FULL"))); + if ((options & VACOPT_FULL) != 0 && + (options & VACOPT_DISABLE_INDEX_CLEANUP) != 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("VACUUM option DISABLE_INDEX_CLEANUP cannot be used with FULL"))); /* * Send info about dead objects to the statistics collector, unless we are * in autovacuum --- autovacuum.c does this for itself. diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index c1faf41..12a5057 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -10465,6 +10465,8 @@ vacuum_option_elem: { if (strcmp($1, "disable_page_skipping") == 0) $$ = VACOPT_DISABLE_PAGE_SKIPPING; + else if (strcmp($1, "disable_index_cleanup") == 0) + $$ = VACOPT_DISABLE_INDEX_CLEANUP; else if (strcmp($1, "skip_locked") == 0) $$ = VACOPT_SKIP_LOCKED; else diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 2fe14d7..e79e67c 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -3156,7 +3156,9 @@ typedef enum VacuumOption VACOPT_FULL = 1 << 4, /* FULL (non-concurrent) vacuum */ VACOPT_SKIP_LOCKED = 1 << 5, /* skip if cannot get lock */ VACOPT_SKIPTOAST = 1 << 6, /* don't process the TOAST table, if any */ - VACOPT_DISABLE_PAGE_SKIPPING = 1 << 7 /* don't skip any pages */ + VACOPT_DISABLE_PAGE_SKIPPING = 1 << 7, /* don't skip any pages */ + VACOPT_DISABLE_INDEX_CLEANUP = 1 << 8 /* don't remove dead tuple and + * cleanup indexes */ } VacuumOption; /* diff --git a/src/test/regress/expected/vacuum.out b/src/test/regress/expected/vacuum.out index fa9d663..cffce59 100644 --- a/src/test/regress/expected/vacuum.out +++ b/src/test/regress/expected/vacuum.out @@ -80,6 +80,9 @@ CONTEXT: SQL function "do_analyze" statement 1 SQL function "wrap_do_analyze" statement 1 VACUUM FULL vactst; VACUUM (DISABLE_PAGE_SKIPPING) vaccluster; +VACUUM (DISABLE_INDEX_CLEANUP) vaccluster; +VACUUM (DISABLE_INDEX_CLEANUP) vactst; -- DISABLE_INDEX_CLEANUP is ignored +VACUUM (DISABLE_INDEX_CLEANUP, FREEZE) vaccluster; -- partitioned table CREATE TABLE vacparted (a int, b char) PARTITION BY LIST (a); CREATE TABLE vacparted1 PARTITION OF vacparted FOR VALUES IN (1); diff --git a/src/test/regress/sql/vacuum.sql b/src/test/regress/sql/vacuum.sql index 9defa0d..9c4bdb7 100644 --- a/src/test/regress/sql/vacuum.sql +++ b/src/test/regress/sql/vacuum.sql @@ -61,6 +61,9 @@ VACUUM FULL vaccluster; VACUUM FULL vactst; VACUUM (DISABLE_PAGE_SKIPPING) vaccluster; +VACUUM (DISABLE_INDEX_CLEANUP) vaccluster; +VACUUM (DISABLE_INDEX_CLEANUP) vactst; -- DISABLE_INDEX_CLEANUP is ignored +VACUUM (DISABLE_INDEX_CLEANUP, FREEZE) vaccluster; -- partitioned table CREATE TABLE vacparted (a int, b char) PARTITION BY LIST (a); -- 1.8.3.1
From 81690e88c1c53df19dcd7efdf79c3eef780c1d26 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Fri, 1 Feb 2019 16:02:50 +0100 Subject: [PATCH v6 2/2] Add --diable-index-cleanup option to vacuumdb. --- doc/src/sgml/ref/vacuumdb.sgml | 15 +++++++++++++++ src/bin/scripts/t/100_vacuumdb.pl | 9 ++++++++- src/bin/scripts/vacuumdb.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 52 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml index 41c7f3d..02d6e46 100644 --- a/doc/src/sgml/ref/vacuumdb.sgml +++ b/doc/src/sgml/ref/vacuumdb.sgml @@ -118,6 +118,21 @@ PostgreSQL documentation </varlistentry> <varlistentry> + <term><option>--disable-index-cleanup</option></term> + <listitem> + <para> + Disable index vacuuming and index cleanup. + </para> + <note> + <para> + This option is only available for servers running + <productname>PostgreSQL</productname> 12 and later. + </para> + </note> + </listitem> + </varlistentry> + + <varlistentry> <term><option>-e</option></term> <term><option>--echo</option></term> <listitem> diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl index 7f3a9b1..d227225 100644 --- a/src/bin/scripts/t/100_vacuumdb.pl +++ b/src/bin/scripts/t/100_vacuumdb.pl @@ -3,7 +3,7 @@ use warnings; use PostgresNode; use TestLib; -use Test::More tests => 44; +use Test::More tests => 47; program_help_ok('vacuumdb'); program_version_ok('vacuumdb'); @@ -38,6 +38,10 @@ $node->issues_sql_like( qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\).*;/, 'vacuumdb --disable-page-skipping'); $node->issues_sql_like( + [ 'vacuumdb', '--disable-index-cleanup', 'postgres' ], + qr/statement: VACUUM \(DISABLE_INDEX_CLEANUP\).*;/, + 'vacuumdb --disable-index-cleanup'); +$node->issues_sql_like( [ 'vacuumdb', '--skip-locked', 'postgres' ], qr/statement: VACUUM \(SKIP_LOCKED\).*;/, 'vacuumdb --skip-locked'); @@ -48,6 +52,9 @@ $node->issues_sql_like( $node->command_fails( [ 'vacuumdb', '--analyze-only', '--disable-page-skipping', 'postgres' ], '--analyze-only and --disable-page-skipping specified together'); +$node->command_fails( + [ 'vacuumdb', '--analyze-only', '--disable-index-cleanup', 'postgres' ], + '--analyze-only and --disable-index-cleanup specified together'); $node->command_ok([qw(vacuumdb -Z --table=pg_am dbname=template1)], 'vacuumdb with connection string'); diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c index 5ac41ea..7aaa3d0 100644 --- a/src/bin/scripts/vacuumdb.c +++ b/src/bin/scripts/vacuumdb.c @@ -42,6 +42,7 @@ typedef struct vacuumingOptions bool full; bool freeze; bool disable_page_skipping; + bool disable_index_cleanup; bool skip_locked; int min_xid_age; int min_mxid_age; @@ -117,6 +118,7 @@ main(int argc, char *argv[]) {"skip-locked", no_argument, NULL, 5}, {"min-xid-age", required_argument, NULL, 6}, {"min-mxid-age", required_argument, NULL, 7}, + {"disable-index-cleanup", no_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -244,6 +246,11 @@ main(int argc, char *argv[]) exit(1); } break; + case 8: + { + vacopts.disable_index_cleanup = true; + break; + } default: fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); exit(1); @@ -288,6 +295,12 @@ main(int argc, char *argv[]) progname, "disable-page-skipping"); exit(1); } + if (vacopts.disable_index_cleanup) + { + fprintf(stderr, _("%s: cannot use the \"%s\" option when performing only analyze\n"), + progname, "disable-index-cleanup"); + exit(1); + } /* allow 'and_analyze' with 'analyze_only' */ } @@ -418,6 +431,14 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts, exit(1); } + if (vacopts->disable_index_cleanup && PQserverVersion(conn) < 120000) + { + PQfinish(conn); + fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 12\n"), + progname, "disable-index-cleanup"); + exit(1); + } + if (vacopts->skip_locked && PQserverVersion(conn) < 120000) { PQfinish(conn); @@ -868,6 +889,13 @@ prepare_vacuum_command(PQExpBuffer sql, int serverVersion, appendPQExpBuffer(sql, "%sDISABLE_PAGE_SKIPPING", sep); sep = comma; } + if (vacopts->disable_index_cleanup) + { + /* DISABLE_PAGE_SKIPPING is supported since 12 */ + Assert(serverVersion >= 120000); + appendPQExpBuffer(sql, "%sDISABLE_INDEX_CLEANUP", sep); + sep = comma; + } if (vacopts->skip_locked) { /* SKIP_LOCKED is supported since v12 */ @@ -1221,6 +1249,7 @@ help(const char *progname) printf(_(" -a, --all vacuum all databases\n")); printf(_(" -d, --dbname=DBNAME database to vacuum\n")); printf(_(" --disable-page-skipping disable all page-skipping behavior\n")); + printf(_(" --disable-index-cleanup disable index vacuuming and index clenaup\n")); printf(_(" -e, --echo show the commands being sent to the server\n")); printf(_(" -f, --full do full vacuuming\n")); printf(_(" -F, --freeze freeze row transaction information\n")); -- 1.8.3.1