Re: Single transaction in the tablesync worker?

2021-01-23 Thread Peter Smith
On Sat, Jan 23, 2021 at 11:26 PM Amit Kapila wrote: > > On Sat, Jan 23, 2021 at 4:55 AM Peter Smith wrote: > > > > PSA the v18 patch for the Tablesync Solution1. > > > > Few comments: > = > 1. > - * So the state progression is always: INIT -&g

Re: Single transaction in the tablesync worker?

2021-01-24 Thread Peter Smith
On Sun, Jan 24, 2021 at 5:54 PM Peter Smith wrote: > > 4. > > - /* > > - * To build a slot name for the sync work, we are limited to NAMEDATALEN - > > - * 1 characters. We cut the original slot name to NAMEDATALEN - 28 chars > > - * and append _%u_sync_%u (1

Re: Single transaction in the tablesync worker?

2021-01-24 Thread Peter Smith
apply - DropSubscription - AlterSubscription_refresh * Updates to PG docs. * New TAP test case. Known Issues: * None. --- Kind Regards, Peter Smith. Fujitsu Australia v19-0001-Tablesync-Solution1.patch Description: Binary data v19-0002-Tablesync-extra-logging.patch Description: Binary data

Re: Single transaction in the tablesync worker?

2021-01-24 Thread Peter Smith
> + if (originid != InvalidRepOriginId) > + { > + elog(DEBUG1, "DropSubscription: dropping origin tracking for > \"%s\"", originname); > > I don't think we need this and the DEBUG1 message in > AlterSubscription_refresh. IT is fine to print this information for > background workers like in apply-worker but not sure if need it here. > The DropSubscription drops the origin of apply worker but it doesn't > use such a DEBUG message so I guess we don't it for tablesync origins > as well. > OK. These DEBUG1 logs are removed in the latest patch [v19]. [v19] https://www.postgresql.org/message-id/CAHut%2BPsj7Xm8C1LbqeAbk-3duyS8xXJtL9TiGaeu3P8g272mAA%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-01-25 Thread Peter Smith
state may be lost (currently under investigation). --- Kind Regards, Peter Smith. Fujitsu Australia v20-0002-Tablesync-extra-logging.patch Description: Binary data v20-0001-Tablesync-Solution1.patch Description: Binary data

Re: Single transaction in the tablesync worker?

2021-01-25 Thread Peter Smith
die "Timed out while waiting for subscriber to start sync"; > > Is there a reason why we can't use the existing way to check for > failure in this case? > The TAP test is updated in the latest patch [v20]. [v20] https://www.postgresql.org/message-id/CAHut%2BPuNwSujoL_dwa%3DTtozJ_vF%3DCnJxjgQTCmNBkazd8J1m-A%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-01-25 Thread Peter Smith
On Mon, Jan 25, 2021 at 1:58 PM Amit Kapila wrote: > > On Sun, Jan 24, 2021 at 12:24 PM Peter Smith wrote: > > > > On Sat, Jan 23, 2021 at 11:26 PM Amit Kapila > > wrote: > > > > > > > > > Few comments: > > > = &

Re: Single transaction in the tablesync worker?

2021-01-25 Thread Peter Smith
On Mon, Jan 25, 2021 at 2:54 PM Amit Kapila wrote: > > On Mon, Jan 25, 2021 at 8:23 AM Peter Smith wrote: > > > > On Sat, Jan 23, 2021 at 11:26 PM Amit Kapila > > wrote: > > > > > > 2. > > > @@ -98,11 +102,16 @@ > > > #in

Re: Single transaction in the tablesync worker?

2021-01-25 Thread Peter Smith
On Mon, Jan 25, 2021 at 4:48 PM Amit Kapila wrote: > > On Mon, Jan 25, 2021 at 8:03 AM Peter Smith wrote: > > > > Hi Amit. > > > > PSA the v19 patch for the Tablesync Solution1. > > > > I see one race condition in this patch where we try

Re: Deleting older versions in unique indexes to avoid page splits

2021-01-26 Thread Peter Geoghegan
istic/organic approach has a lot of advantages, especially given the general uncertainty about workload characteristics. Your suspicion of the simple nature of the heuristics actually makes a lot of sense to me. I do get it. -- Peter Geoghegan

pg_replication_origin_drop API potential race condition

2021-01-26 Thread Peter Smith
cing a new "missing_ok" parameter for replorigin_drop. Thoughts? [ak0125] https://www.postgresql.org/message-id/CAA4eK1%2ByeLwBCkTvTdPM-hSk1fr6jT8KJc362CN8zrGztq_JqQ%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-01-26 Thread Peter Smith
On Mon, Jan 25, 2021 at 4:48 PM Amit Kapila wrote: > > On Mon, Jan 25, 2021 at 8:03 AM Peter Smith wrote: > > > > Hi Amit. > > > > PSA the v19 patch for the Tablesync Solution1. > > > > I see one race condition in this patch where we try

Re: FailedAssertion in heap_index_delete_tuples at heapam.c:7220

2021-01-26 Thread Peter Geoghegan
ix is obvious: Bring gistprunepage() in line with _hash_vacuum_one_page(). I'll go push a fix for that now. Thanks for the report! -- Peter Geoghegan

Re: vacuum_cost_page_miss default value and modern hardware

2021-01-27 Thread Peter Geoghegan
On Thu, Jan 21, 2021 at 5:12 PM Peter Geoghegan wrote: > I'm going to go ahead with committing my patch to lower the default > next week. If anybody has any objections to that plan, please speak > up. Just pushed a commit that reduced the default for vacuum_cost_page_miss to 2.

Re: Thoughts on "killed tuples" index hint bits support on standby

2021-01-27 Thread Peter Geoghegan
rly be compelled to consider the bigger picture, including the question of setting hint bits on standbys -- this other person had better not make that harder in the future, for the same reasons (obviously what you want to do here makes sense, it just isn't everything to everybody). This is yet another way in which expanding scope can help. -- Peter Geoghegan

Re: Thoughts on "killed tuples" index hint bits support on standby

2021-01-27 Thread Peter Geoghegan
On Wed, Jan 27, 2021 at 5:24 PM Peter Geoghegan wrote: > The issue here is that it would also be nice to use a "recently dead" > bit on the primary, but if you do that then maybe you're back to the > original problem. OTOH, I also think that we could repurpose the > LP_

Re: Single transaction in the tablesync worker?

2021-01-27 Thread Peter Smith
/message-id/CAHut%2BPuNwSujoL_dwa%3DTtozJ_vF%3DCnJxjgQTCmNBkazd8J1m-A%40mail.gmail.com [ak0127] https://www.postgresql.org/message-id/CAA4eK1LDsj9kw4FbWAw3CMHyVsjafgDum03cYy-wpGmor%3D8-1w%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia v21-0002-Tablesync-extra-logging.patch Description

Re: Single transaction in the tablesync worker?

2021-01-27 Thread Peter Smith
On Wed, Jan 27, 2021 at 2:53 PM Amit Kapila wrote: > > On Sat, Jan 23, 2021 at 5:56 PM Amit Kapila wrote: > > > > On Sat, Jan 23, 2021 at 4:55 AM Peter Smith wrote: > > > > > > PSA the v18 patch for the Tablesync Solution1. > > > > 7. Have you

Re: "has_column_privilege()" issue with attnums and non-existent columns

2021-01-28 Thread Peter Eisentraut
On 2021-01-12 06:53, Ian Lawrence Barwick wrote: postgres=# SELECT has_column_privilege('foo', 999::int2, 'SELECT'); has_column_privilege -- t (1 row) The comment on the relevant code section in "src/backend/utils/adt/acl.c" (related to "column_priv

Re: BUG #16583: merge join on tables with different DB collation behind postgres_fdw fails

2021-01-28 Thread Peter Eisentraut
her they conform to those rules. As you said, the description of the import_collate option is kind of hand-wavy about all this. -- Peter Eisentraut 2ndQuadrant, an EDB company https://www.2ndquadrant.com/

Re: vacuum_cost_page_miss default value and modern hardware

2021-01-28 Thread Peter Geoghegan
On Thu, Jan 28, 2021 at 9:30 AM Robert Haas wrote: > On Thu, Jan 14, 2021 at 8:09 PM Peter Geoghegan wrote: > > So dirty pages are debt that VACUUM can easily create, whereas buffer > > misses are paid directly by VACUUM. It is its own backpressure, for > > the most par

Re: Single transaction in the tablesync worker?

2021-01-29 Thread Peter Smith
(cleanup_at_shutdown function). + Addresses some minor review comments. [ak0128] https://www.postgresql.org/message-id/CAA4eK1LMYXZY1SpzgW-WyFdy%2BFTMZ4BMz1dj0rT2rxGv-zLwFA%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia v22-0001-Tablesync-Solution1.patch Description: Binary data

Re: Single transaction in the tablesync worker?

2021-01-29 Thread Peter Smith
On Thu, Jan 28, 2021 at 9:37 PM Amit Kapila wrote: > > On Thu, Jan 28, 2021 at 12:32 PM Peter Smith wrote: > > > > On Wed, Jan 27, 2021 at 2:53 PM Amit Kapila wrote: > > > > > > On Sat, Jan 23, 2021 at 5:56 PM Amit Kapila > > > wrote: > >

Re: Dumping/restoring fails on inherited generated column

2021-01-29 Thread Peter Eisentraut
to determine whether a column was inherited, but that's not correct. This patch uses the existing logic in flagInhAttrs() to find whether there is a matching parent column with a generation expression. I've added pg_dump test cases here to check the different variations that the code

Re: Should we make Bitmapsets a kind of Node?

2021-01-29 Thread Peter Geoghegan
e that anymore. It's probably much riskier to use 32-bit x86 today than it is to use (say) POWER8, or some other contemporary minority platform. -- Peter Geoghegan

Re: Should we make Bitmapsets a kind of Node?

2021-01-29 Thread Peter Geoghegan
. I suspect that this matters. I am speculating here, of course -- I have to because there is no guidance to work off of. I don't know anybody that still runs Postgres (or anything like it) on a 32-bit platform. I think that Michael Paquier owns a Raspberry Pi zero, but that hardly counts. -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-01-29 Thread Peter Geoghegan
s, based on multiple factors. That will probably be the only practical way to assess how much better (or worse) the patch is compared to master. This patch is more about efficiency and predictability than performance per se. Which is good, because that's where most of the real world problems actually are. -- Peter Geoghegan

Re: Thoughts on "killed tuples" index hint bits support on standby

2021-01-29 Thread Peter Geoghegan
AMs/all-index-AMs divide in how item pointers work would be preserved. > Also, btw, do you know any reason to keep minRecoveryPoint at a low value? Not offhand. -- Peter Geoghegan

Re: Should we make Bitmapsets a kind of Node?

2021-01-29 Thread Peter Geoghegan
. Including even the extremely low power 32-bit chips that are not yet fully obsolete, like the Raspberry Pi Zero's chip. -- Peter Geoghegan

Re: Should we make Bitmapsets a kind of Node?

2021-01-29 Thread Peter Geoghegan
Postgres may well come out ahead, in a certain sense. -- Peter Geoghegan

Re: Should we make Bitmapsets a kind of Node?

2021-01-29 Thread Peter Geoghegan
On Fri, Jan 29, 2021 at 6:44 PM Tom Lane wrote: > Pointer width is interesting, but really it's a solved problem > compared to these. What about USE_FLOAT8_BYVAL? -- Peter Geoghegan

Re: Allow CURRENT_ROLE in GRANTED BY

2021-01-30 Thread Peter Eisentraut
On 2020-12-30 13:43, Simon Riggs wrote: On Thu, 10 Dec 2020 at 18:40, Peter Eisentraut wrote: On 2020-06-24 20:21, Peter Eisentraut wrote: On 2020-06-24 10:12, Vik Fearing wrote: On 6/24/20 8:35 AM, Peter Eisentraut wrote: I was checking some loose ends in SQL conformance, when I noticed

Re: SELECT INTO deprecation

2021-01-30 Thread Peter Eisentraut
On 2020-12-17 02:30, Michael Paquier wrote: On Wed, Dec 16, 2020 at 06:07:08PM +0100, Peter Eisentraut wrote: Right, we would very likely not add it now. But it doesn't seem to cause a lot of ongoing maintenance burden, so if there is a use case, it's not unreasonable to keep it

Re: Add primary keys to system catalogs

2021-01-30 Thread Peter Eisentraut
On 2021-01-21 18:15, Tom Lane wrote: After reading the patch again, I have a couple more nits about comments, which I'll just present as a proposed delta patch. Otherwise it's good. I'll mark it RFC. Committed with your update, thanks. -- Peter Eisentraut 2ndQuadrant, an ED

Re: Is it worth accepting multiple CRLs?

2021-01-30 Thread Peter Eisentraut
On 2021-01-19 09:32, Kyotaro Horiguchi wrote: At Tue, 19 Jan 2021 09:17:34 +0900 (JST), Kyotaro Horiguchi wrote in By the way we can do the same thing on CA file/dir, but I personally think that the benefit from the specify-by-directory for CA files is far less than CRL files. So I'm not going

Re: Thoughts on "killed tuples" index hint bits support on standby

2021-01-30 Thread Peter Geoghegan
7;m unsure. > > BTW, what happens when the page splits on the primary, btw? Does your > > patch "move over" the LP_DEAD bits to each half of the split? > > That part is not changed in any way. Maybe it's okay to assume that it's no loss to throw away hint bits set on the standby, because in practice deletion on the primary will usually do the right thing anyway. But you never said that. I think that you should take an explicit position on this question -- make it a formal part of your overall design. -- Peter Geoghegan

Re: Thoughts on "killed tuples" index hint bits support on standby

2021-01-31 Thread Peter Geoghegan
On Sat, Jan 30, 2021 at 5:39 PM Peter Geoghegan wrote: > If you invent some entirely new category of standby-only hint bit at a > level below the access method code, then you can use it inside access > method code such as nbtree. Maybe you don't have to play games with > minReco

Re: Single transaction in the tablesync worker?

2021-01-31 Thread Peter Smith
ubscriptionRelRelationId, AccessExclusiveLock); > 4. Added/Changed quite a few comments. > @@ -1042,6 +1115,31 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel) } list_free(subworkers); + /* + * Tablesync resource cleanup (slots and origins). The comment is mislea

Re: Single transaction in the tablesync worker?

2021-01-31 Thread Peter Smith
f LogicalRepWorkerId */ - struct StopWorkersData *parent; /* This need not be an immediate - * subtransaction parent */ -} StopWorkersData; Since v23 removes that typedef from the code, don't you also have to remove it from src/tools/pgindent/typedefs.list? Kind Regards, Peter Smith. Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-01-31 Thread Peter Smith
On Mon, Feb 1, 2021 at 1:54 PM Amit Kapila wrote: > > On Mon, Feb 1, 2021 at 6:48 AM Peter Smith wrote: > > > > On Sun, Jan 31, 2021 at 12:19 AM Amit Kapila > > wrote: > > > > > > I have made the below changes in the patch. Let me know what you thin

Re: Single transaction in the tablesync worker?

2021-01-31 Thread Peter Smith
On Mon, Feb 1, 2021 at 3:44 PM Amit Kapila wrote: > > On Mon, Feb 1, 2021 at 9:38 AM Peter Smith wrote: > > > > On Mon, Feb 1, 2021 at 1:54 PM Amit Kapila wrote: > > > > > > On Mon, Feb 1, 2021 at 6:48 AM Peter Smith wrote: > > > > >

Re: Single transaction in the tablesync worker?

2021-01-31 Thread Peter Smith
ERM signal arrives immediately after the slot is dropped, the tablesync will still become SYNCDONE. Is this wrong understanding? But your scenario could still be possible if "die" exited immediately (e.g. only in single user mode?). Kind Regards, Peter Smith. Fujitsu Australia

Re: Add primary keys to system catalogs

2021-02-01 Thread Peter Eisentraut
On 2021-01-30 22:56, Tom Lane wrote: Peter Eisentraut writes: Committed with your update, thanks. Hmm, shouldn't there have been a catversion bump in there? I suppose yes on the grounds that it introduces something new in a freshly initdb-ed database. But I thought it wasn't

Re: SEARCH and CYCLE clauses

2021-02-01 Thread Peter Eisentraut
On 2020-11-25 20:35, Pavel Stehule wrote: I checked this patch, and I didn't find any issue. make check-world passed make doc passed I'll mark it as ready for committer This has been committed. Thanks. -- Peter Eisentraut 2ndQuadrant, an EDB company https://www.2ndquadrant.com/

Re: Thoughts on "killed tuples" index hint bits support on standby

2021-02-01 Thread Peter Geoghegan
fter > applying FPI). So, the masking page during *applying* the FPI is semantically > the same as setting a bit in it 1 nanosecond after. > > And `btree_mask` (and other mask functions) already used for consistency > checks to exclude LP_DEAD. I don't see how that is relevant. btree_mask() is only used by wal_consistency_checking, which is mostly just for Postgres hackers. -- Peter Geoghegan

Typo in tablesync comment

2021-02-01 Thread Peter Smith
PSA a trivial patch to correct what seems like a typo in the tablesync comment. Kind Regards, Peter Smith. Fujitsu Australia fix-tablesync-comment-typo-20210202.patch Description: Binary data

Re: Single transaction in the tablesync worker?

2021-02-01 Thread Peter Smith
gt; Yes, we could restrict the create slot API if you really wanted to. But, IMO it is implausible that a user could "accidentally" clash with the internal tablesync slot name, so in practice maybe this change would not help much but it might make it more difficult to test some scenarios. Kind Regards, Peter Smith. Fujitsu Australia

Re: New IndexAM API controlling index vacuum strategies

2021-02-01 Thread Peter Geoghegan
On Fri, Jan 29, 2021 at 5:26 PM Peter Geoghegan wrote: > It'll be essential to have good instrumentation as we do more > benchmarking. We're probably going to have to make subjective > assessments of benchmark results, based on multiple factors. That will > probably be the

Re: New IndexAM API controlling index vacuum strategies

2021-02-01 Thread Peter Geoghegan
On Mon, Feb 1, 2021 at 7:17 PM Peter Geoghegan wrote: > Here is how the triggering criteria could work: maybe skipping > accessing all indexes during VACUUM happens when less than 1% or > 10,000 of the items from the table are to be removed by VACUUM -- > whichever is greater. Of co

Re: Single transaction in the tablesync worker?

2021-02-01 Thread Peter Smith
ble copy" scenario. [ac0202] https://www.postgresql.org/message-id/CAFPTHDaZw5o%2BwMbv3aveOzuLyz_rqZebXAj59rDKTJbwXFPYgw%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia Test Scenario = OBJECTIVE - Break the connection during table copy (after slot is create

Re: Commitfest 2021-01 is now closed.

2021-02-01 Thread Peter Geoghegan
gt;> > work, please tell me here or by directly emailing me. >> >> I think you did an excellent job. +1 -- Peter Geoghegan

Re: Single transaction in the tablesync worker?

2021-02-02 Thread Peter Smith
and the stack trace details. [ac0202] https://www.postgresql.org/message-id/CAFPTHDYzjaNfzsFHpER9idAPB8v5j%3DSUbWL0AKj5iVy0BKbTpg%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia Test Scenario 1. INSERT data so tablesync should copy something 2. While paused in LogicalRepSyncTa

Improve new hash partition bound check error messages

2021-02-02 Thread Peter Eisentraut
3aed36729da22e1 Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Tue, 2 Feb 2021 11:21:21 +0100 Subject: [PATCH] Improve new hash partition bound check error messages For the error message "every hash partition modulus must be a factor of the next larger modulus", add a detail message

Re: Add primary keys to system catalogs

2021-02-02 Thread Peter Eisentraut
On 2021-02-01 15:24, Tom Lane wrote: Peter Eisentraut writes: On 2021-01-30 22:56, Tom Lane wrote: Hmm, shouldn't there have been a catversion bump in there? I suppose yes on the grounds that it introduces something new in a freshly initdb-ed database. But I thought it wasn't

Re: Add primary keys to system catalogs

2021-02-02 Thread Peter Eisentraut
recursive deletion. Now that we have the refobjversion column, the presence of duplicate dependencies might be even more dubious. I think that could be worth analyzing. -- Peter Eisentraut 2ndQuadrant, an EDB company https://www.2ndquadrant.com/

Re: New IndexAM API controlling index vacuum strategies

2021-02-02 Thread Peter Geoghegan
eter? Possibly. I'm concerned about making any user-visible interface (say a GUC) compatible with an improved version that is smarter about bottom-up index deletion (in particular, one that can vacuum only a subset of the indexes on a table, which now seems too ambitious for Postgres 14). -- Peter Geoghegan

Re: Single transaction in the tablesync worker?

2021-02-02 Thread Peter Smith
On Wed, Feb 3, 2021 at 12:26 AM Amit Kapila wrote: > > On Tue, Feb 2, 2021 at 3:31 PM Peter Smith wrote: > > > > After seeing Ajin's test [ac0202] which did a DROP TABLE, I have also > > tried a simple test where I do a DROP TABLE with very bad timing for > >

Re: Single transaction in the tablesync worker?

2021-02-02 Thread Peter Smith
On Wed, Feb 3, 2021 at 1:34 PM Amit Kapila wrote: > > On Wed, Feb 3, 2021 at 6:38 AM Peter Smith wrote: > > > > On Wed, Feb 3, 2021 at 12:26 AM Amit Kapila wrote: > > > > > > On Tue, Feb 2, 2021 at 3:31 PM Peter Smith wrote: > > > > > >

Re: Typo in tablesync comment

2021-02-02 Thread Peter Smith
.g - *The catalog pg_subscription_rel is used to keep information about *subscribed tables and their state. The catalog holds all states *except SYNCWAIT and CATCHUP which are only in shared memory. - Kind Regards, Peter Smith. Fujitsu Australia

DROP TABLE can crash the replication sync worker

2021-02-03 Thread Peter Smith
bUEWN%2Ba%3D8Pg_njsJKc9Zoz05A_ewJSvjX2MQ%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia v1-0001-DROP-TABLE-breaks-sync-worker-relid.patch Description: Binary data

Re: Single transaction in the tablesync worker?

2021-02-03 Thread Peter Smith
On Wed, Feb 3, 2021 at 2:51 PM Peter Smith wrote: > > On Wed, Feb 3, 2021 at 1:34 PM Amit Kapila wrote: > > > > On Wed, Feb 3, 2021 at 6:38 AM Peter Smith wrote: > > > > > > On Wed, Feb 3, 2021 at 12:26 AM Amit Kapila > > > wrote: > > >

Re: Dumping/restoring fails on inherited generated column

2021-02-03 Thread Peter Eisentraut
On 2021-01-29 17:41, Tom Lane wrote: Peter Eisentraut writes: I've had another go at this, and I've found a solution that appears to address all the issues I'm aware of. It's all very similar to the previously discussed patches. The main difference is that previous patc

pg_dump: Add const decorations

2021-02-03 Thread Peter Eisentraut
From a2ec95a397a1e8a075a56079c6928ba3b1d2b6ce Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Wed, 3 Feb 2021 13:10:24 +0100 Subject: [PATCH] pg_dump: Add const decorations Add const decorations to the *info arguments of the dump* functions, to clarify that they don't modify that argument.

Re: Improve new hash partition bound check error messages

2021-02-03 Thread Peter Eisentraut
On 2021-02-02 13:26, Heikki Linnakangas wrote: How about this? CREATE TABLE fail_part PARTITION OF hash_parted FOR VALUES WITH (MODULUS 25, REMAINDER 3); ERROR: every hash partition modulus must be a factor of the next larger modulus DETAIL: 25 is not divisible by 10, the modulus of existing p

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2021-02-03 Thread Peter Eisentraut
On 2021-01-17 14:38, Magnus Hagander wrote: On Thu, Jan 7, 2021 at 11:53 AM Peter Eisentraut wrote: After pondering this again, I think we can go with initdb --no-instructions, as in your patch. As a minor nitpick, I would leave out the else printf(_("\nSuccess.\n&qu

Re: Dumping/restoring fails on inherited generated column

2021-02-03 Thread Peter Eisentraut
our commands and they restore correctly AFAICT. -- Peter Eisentraut 2ndQuadrant, an EDB company https://www.2ndquadrant.com/

Re: Dumping/restoring fails on inherited generated column

2021-02-03 Thread Peter Eisentraut
dress this. Probably needs another look for column number mapping and all the usual stuff, but the basic idea should be okay. -- Peter Eisentraut 2ndQuadrant, an EDB company https://www.2ndquadrant.com/ From bd6ee462e5ad4a6514940e7e78d40c4f28f3dc3f Mon Sep 17 00:00:00 2001 From: Peter Eisentraut

Re: Typo in tablesync comment

2021-02-03 Thread Peter Smith
On Wed, Feb 3, 2021 at 11:55 PM Amit Kapila wrote: > > On Wed, Feb 3, 2021 at 1:35 PM Michael Paquier wrote: > > > > On Wed, Feb 03, 2021 at 06:52:56PM +1100, Peter Smith wrote: > > > > > Maybe better to rewrite it more drastically? > > > >

Re: DROP TABLE can crash the replication sync worker

2021-02-03 Thread Peter Smith
On Wed, Feb 3, 2021 at 11:49 PM Amit Kapila wrote: > > On Wed, Feb 3, 2021 at 2:53 PM Peter Smith wrote: > > > > Hi Hackers. > > > > As discovered in another thread [master] there is an *existing* bug in > > the PG HEAD code which can happen if a DROP TABLE is

Re: pg_replication_origin_drop API potential race condition

2021-02-03 Thread Peter Smith
ems ok. I wonder if it is better to isolate that locked portion (replyorigin_by_name + replorigin_drop) so that in addition to being called from pg_replication_origin_drop, we can call it internally from PG code to safely drop the origins. Kind Regards, Peter Smith. Fujitsu Australia

Re: pg_replication_origin_drop API potential race condition

2021-02-04 Thread Peter Smith
On Thu, Feb 4, 2021 at 4:43 PM Amit Kapila wrote: > > On Thu, Feb 4, 2021 at 9:57 AM Peter Smith wrote: > > > > On Wed, Feb 3, 2021 at 11:17 PM Amit Kapila wrote: > > > > > > > > How about if we call replorigin_by_name() inside replorigin_drop after >

Re: Single transaction in the tablesync worker?

2021-02-04 Thread Peter Smith
relid, + syncslotname); + + ReplicationSlotDropAtPubNode(wrconn, syncslotname, false /* missing_ok */); + Should this comment also describe why the missing_ok is false for this case? Kind Regards, Peter Smith. Fujitsu Australia

Re: pg_replication_origin_drop API potential race condition

2021-02-04 Thread Peter Smith
On Thu, Feb 4, 2021 at 9:20 PM Amit Kapila wrote: > > On Thu, Feb 4, 2021 at 1:31 PM Peter Smith wrote: > > > > PSA a patch which I think implements what we are talking about. > > > > This doesn't seem correct to me. Have you tested that the patch > re

Re: pg_replication_origin_drop API potential race condition

2021-02-05 Thread Peter Smith
On Fri, Feb 5, 2021 at 6:02 PM Amit Kapila wrote: > > On Fri, Feb 5, 2021 at 9:46 AM Peter Smith wrote: > > > > PSA patch updated per above suggestions. > > > > Thanks, I have tested your patch and before the patch, I was getting > errors like "tuple concurr

Re: Dumping/restoring fails on inherited generated column

2021-02-05 Thread Peter Eisentraut
I figured out how to take account of generation expressions with different column orders. I used the same approach that we use for check constraints. The attached patch is good to go from my perspective. -- Peter Eisentraut 2ndQuadrant, an EDB company https://www.2ndquadrant.com/ From 3079ddc7e1fdf66

Re: New IndexAM API controlling index vacuum strategies

2021-02-05 Thread Peter Geoghegan
can take care of this myself for Postgres 14. > I agreed to cut the scope for PG14. It would be good if we could > improve index vacuum while cutting cut the scope for PG14 and not > expanding the extent of the impact of this issue. Great! Well, if I take care of the _bt_page_recyclable() wraparound/epoch issue in a general kind of way then AFAICT there is no added risk. -- Peter Geoghegan

Re: Single transaction in the tablesync worker?

2021-02-05 Thread Peter Smith
otname name is also unique. Is that understanding not correct? - [1] https://www.postgresql.org/docs/devel/catalog-pg-subscription.html Kind Regards, Peter Smith. Fujitsu Australia

GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-06 Thread Peter Geoghegan
ullXid() is the only symbol name matching "GlobalVisIsRemovable*". Have I missed something? -- Peter Geoghegan

Re: Single transaction in the tablesync worker?

2021-02-06 Thread Peter Smith
; for current callers though (but then maybe don't call the param > missign_ok?). You are right. The way we are using this function has evolved beyond the original intention. Probably renaming the param to something like "error_ok" would be more appropriate now. Kind Regards, Peter Smith. Fujitsu Australia

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-06 Thread Peter Geoghegan
you do it yourself, just in case. -- Peter Geoghegan

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-07 Thread Peter Geoghegan
On Sat, Feb 6, 2021 at 7:41 PM Peter Geoghegan wrote: > Yes, please do. I could do it myself, but better that you do it > yourself, just in case. I went ahead and fixed it myself. Thanks -- Peter Geoghegan

Re: Single transaction in the tablesync worker?

2021-02-07 Thread Peter Smith
On Sun, Feb 7, 2021 at 2:38 PM Peter Smith wrote: > > On Sat, Feb 6, 2021 at 2:10 AM Petr Jelinek > wrote: > > > > Hi, > > > > Some minor comments about code: > > > > > + else if (res->status == WALRCV_ERROR && missing_ok) &

Re: Single transaction in the tablesync worker?

2021-02-07 Thread Peter Smith
iolation # Check if DROP SUBSCRIPTION cleans up slots on the publisher side # when the subscriber is stuck on data copy for constraint But it is not clear to me what was the exact cause of that PK violation. I think you must be relying on data that is leftover from some previous test case but I am not sure which one. Can you make the comment more detailed to say *how* the PK violation is happening - e.g something to say which rows, in which table, and inserted by who? -- Kind Regards, Peter Smith. Fujitsu Australia

small test case for abbrev(cidr)

2021-02-08 Thread Peter Eisentraut
4bba4782c81ec6cab3d85e045abc89a85092e2d0 Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Mon, 8 Feb 2021 15:03:49 +0100 Subject: [PATCH] Add test case for abbrev(cidr) This will in particular add some good test coverage for inet_cidr_ntop.c, which was previously completely uncovered. --- src/test/regress/expected/inet.out

Re: Single transaction in the tablesync worker?

2021-02-08 Thread Peter Smith
On Mon, Feb 8, 2021 at 11:42 AM Peter Smith wrote: > > On Sun, Feb 7, 2021 at 2:38 PM Peter Smith wrote: > > > > On Sat, Feb 6, 2021 at 2:10 AM Petr Jelinek > > wrote: > > > > > > Hi, > > > > > > Some minor comments about code: >

Re: Single transaction in the tablesync worker?

2021-02-08 Thread Peter Smith
; "With current scheme" -> "With the current scheme" 7. Maybe consider to just assign GetSystemIdentifier() to a static instead of calling that function for every slot? static uint64 sysid = GetSystemIdentifier(); IIUC the sysid value is never going to change for a process, right

Re: Single transaction in the tablesync worker?

2021-02-08 Thread Peter Smith
uot;) were probably meant to be ellipsis ("...") Kind Regards, Peter Smith. Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-02-09 Thread Peter Smith
acked onto the end like that. I am wondering if reordering of parent to child might be more natural. e.g sysid + subid + relid gives a more intuitive name IMO. So in this example it would be "pg_sync_6927117142022745645_16397_16389" Thoughts? Kind Regards, Peter Smith Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-02-09 Thread Peter Smith
) { /* be tidy */ list_free(rstates); return; } -- Isn't the V29 code missing doing a table_close(rel, NoLock) there? -- Kind Regards, Peter Smith. Fujitsu Australia

Routine usage information schema tables

2021-02-09 Thread Peter Eisentraut
uot;, where it would become more useful, but I'm sending it now separately to not bloat the other patch further. [0]: https://www.postgresql.org/message-id/flat/1c11f1eb-f00c-43b7-799d-2d44132c0...@2ndquadrant.com From f3a62c680760f57693dac2be2e7ba7c8cfa04fd4 Mon Sep 17 00:00:00 2001 From: Pe

64-bit XIDs in deleted nbtree pages

2021-02-09 Thread Peter Geoghegan
tradictory just by examining the output over time, how things change, etc. [1] https://postgr.es/m/ca+tgmoyd7xpr1dwewwxxiw4-wc1nbjf3rb9d2qgpvyh9ejz...@mail.gmail.com [2] https://postgr.es/m/cah2-wzmkebqpd4mvguptos9bmfvp9mds5crtcosv1rqj3jc...@mail.gmail.com -- Peter Geoghegan From 39ef90d96d0c061b2e537c4cdc989

Re: Single transaction in the tablesync worker?

2021-02-09 Thread Peter Smith
On Tue, Feb 9, 2021 at 8:32 PM Amit Kapila wrote: > > On Tue, Feb 9, 2021 at 12:02 PM Peter Smith wrote: > > > > Here are my feedback comments for the V29 patch. > > > > Thanks. > > > > > 3. > > Previously the tablesync origin name format wa

Re: 64-bit XIDs in deleted nbtree pages

2021-02-09 Thread Peter Geoghegan
On Tue, Feb 9, 2021 at 2:14 PM Peter Geoghegan wrote: > The first patch teaches nbtree to use 64-bit transaction IDs here, and > so makes it impossible to leak deleted nbtree pages. This patch is the > nbtree equivalent of commit 6655a729, which made GiST use 64-bit XIDs > due to exac

Re: Single transaction in the tablesync worker?

2021-02-09 Thread Peter Smith
On Tue, Feb 9, 2021 at 10:38 AM Peter Smith wrote: > > On Mon, Feb 8, 2021 at 11:42 AM Peter Smith wrote: > > > > On Sun, Feb 7, 2021 at 2:38 PM Peter Smith wrote: > > > > > > On Sat, Feb 6, 2021 at 2:10 AM Petr Jelinek > > > wrote: > > > &

Re: 64-bit XIDs in deleted nbtree pages

2021-02-09 Thread Peter Geoghegan
On Tue, Feb 9, 2021 at 5:53 PM Peter Geoghegan wrote: > Here is a plan that allows us to stop storing any kind of XID in the > metapage in all cases: Attached is v2, which deals with the metapage 32-bit XID/btm_oldest_btpo_xact issue using the approach I described earlier. We don't s

Re: New IndexAM API controlling index vacuum strategies

2021-02-09 Thread Peter Geoghegan
I came up with a solution that turns out to not be very invasive. At the same time it has unexpected advantages, liking improving amcheck coverage for deleted pages. -- Peter Geoghegan

Re: 64-bit XIDs in deleted nbtree pages

2021-02-10 Thread Peter Geoghegan
hat actually happens in nbtree. What's a "removed" page? The distinction between all of the different kinds of index pages that might be involved here is just subtle. Again, better to use a precise, descriptive term that nobody fully understands -- because hardly anybody will fully understand it anyway (even including advanced users that go on to find the VACUUM VERBOSE output very useful for whatever reason). -- Peter Geoghegan

Re: Single transaction in the tablesync worker?

2021-02-10 Thread Peter Smith
nd Regards, Peter Smith. Fujitsu Australia

Re: 64-bit XIDs in deleted nbtree pages

2021-02-10 Thread Peter Geoghegan
e into, but the assumption/design has many more advantages than disadvantages. I have tried to capture this in v3 of the patch. Can you take a look? See the new comments inside _bt_vacuum_needs_cleanup(). Plus the comments when we call it inside btvacuumscan(). Do you think that those new comments are helpful? Does this address your concern? Thanks -- Peter Geoghegan v3-0001-Use-full-64-bit-XID-for-nbtree-page-deletion.patch Description: Binary data v3-0002-Add-pages_newly_deleted-to-VACUUM-VERBOSE.patch Description: Binary data

Re: 64-bit XIDs in deleted nbtree pages

2021-02-10 Thread Peter Geoghegan
On Wed, Feb 10, 2021 at 7:10 PM Peter Geoghegan wrote: > Attached is v3 of the index. I'll describe the changes I made in more > detail in my response to your points below. I forget to mention that v3 adds several assertions like this one: Assert(!_bt_page_recyclable(Buffer

<    2   3   4   5   6   7   8   9   10   11   >