Re: Possible corruption by CreateRestartPoint at promotion

2022-04-27 Thread Michael Paquier
On Thu, Apr 28, 2022 at 11:43:57AM +0900, Kyotaro Horiguchi wrote: > At Thu, 28 Apr 2022 09:12:13 +0900, Michael Paquier > wrote in >> On Wed, Apr 27, 2022 at 11:09:45AM -0700, Nathan Bossart wrote: >>> On Wed, Apr 27, 2022 at 02:16:01PM +0900, Michael Paquier wrote: - if (ControlFile->st

Re: Multi-Master Logical Replication

2022-04-27 Thread Laurenz Albe
On Thu, 2022-04-28 at 09:49 +1000, Peter Smith wrote: > To prevent the circular recursive problem Vignesh is developing a > patch [1] that introduces new SUBSCRIPTION options "local_only" (for > publishing only data originating at the publisher node) and > "copy_data=force". Using this patch, we ha

Re: Implementing Incremental View Maintenance

2022-04-27 Thread Yugo NAGATA
Hello Greg, On Sat, 23 Apr 2022 08:18:01 +0200 Greg Stark wrote: > I'm trying to figure out how to get this feature more attention. Everyone > agrees it would be a huge help but it's a scary patch to review. > > I wonder if it would be helpful to have a kind of "readers guide" > explanation of

Re: Defer selection of asynchronous subplans until the executor initialization stage

2022-04-27 Thread Etsuro Fujita
On Mon, Apr 25, 2022 at 1:29 PM Etsuro Fujita wrote: > I modified mark_async_capable_plan() a bit further; 1) adjusted code > in the ProjectionPath case, just for consistency with other cases, and > 2) tweaked/improved comments a bit. Attached is a new version of the > patch (“prevent-async-2.pat

Re: Add --{no-,}bypassrls flags to createuser

2022-04-27 Thread Shinya Kato
Thank you for the reviews! On 2022-04-26 05:19, Nathan Bossart wrote: - printf(_(" -g, --role=ROLE new role will be a member of this role\n")); + printf(_(" -g, --role=ROLEnew role will be a member of this role\n")); This looks lik an unexpected change. I fixed it. I'

Re: avoid multiple hard links to same WAL file after a crash

2022-04-27 Thread Michael Paquier
On Wed, Apr 27, 2022 at 11:42:04AM -0700, Nathan Bossart wrote: > Here is a new patch set with these assertions added. I think at least the > xlog.c change ought to be back-patched. The problem may be unlikely, but > AFAICT the possible consequences include WAL corruption. Okay, so I have applie

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Amit Kapila
On Thu, Apr 28, 2022 at 3:26 AM Tomas Vondra wrote: > > so I've been looking at tweaking the code so that the behavior matches > Alvaro's expectations. It passes check-world but I'm not claiming it's > nowhere near commitable - the purpose is mostly to give better idea of > how invasive the change

Re: pgsql: Add contrib/pg_walinspect.

2022-04-27 Thread Jeff Davis
On Thu, 2022-04-28 at 12:11 +1200, Thomas Munro wrote: > > Another option might be to abandon this whole no-wait concept and > revert 2258e76f's changes to xlogutils.c. pg_walinspect already does > preliminary checks that LSNs are in range, so you can't enter a value > that will wait indefinitely

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-27 Thread Kyotaro Horiguchi
At Wed, 27 Apr 2022 01:31:55 -0400, Tom Lane wrote in > Michael Paquier writes: > > On Wed, Apr 27, 2022 at 12:36:10PM +0800, Rui Zhao wrote: > >> Do you have interest in adding a test like one in my patch? > > > I have studied the test case you are proposing, and I am afraid that > > it is too

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-27 Thread Kyotaro Horiguchi
At Thu, 28 Apr 2022 09:12:13 +0900, Michael Paquier wrote in > On Wed, Apr 27, 2022 at 11:09:45AM -0700, Nathan Bossart wrote: > > On Wed, Apr 27, 2022 at 02:16:01PM +0900, Michael Paquier wrote: > >> - if (ControlFile->state == DB_IN_ARCHIVE_RECOVERY && > >> - ControlFile->checkPointCop

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-27 Thread Kyotaro Horiguchi
At Wed, 27 Apr 2022 14:16:01 +0900, Michael Paquier wrote in > On Tue, Apr 26, 2022 at 08:26:09PM -0700, Nathan Bossart wrote: > > On Wed, Apr 27, 2022 at 10:43:53AM +0900, Kyotaro Horiguchi wrote: > >> At Tue, 26 Apr 2022 11:33:49 -0700, Nathan Bossart > >> wrote in > >>> I suspect we'll sta

Re: trivial comment fix

2022-04-27 Thread John Naylor
On Thu, Apr 28, 2022 at 7:27 AM Euler Taveira wrote: > > Hi, > > While reading worker.c, I noticed that the referred SQL command was wrong. > ALTER SUBSCRIPTION ... REFRESH PUBLICATION instead of ALTER TABLE ... REFRESH > PUBLICATION. Trivial fix attached. Pushed, thanks! -- John Naylor EDB: ht

Re: Handle infinite recursion in logical replication setup

2022-04-27 Thread Peter Smith
Here are my review comments for v10-0002 (TAP tests part only) FIle: src/test/subscription/t/032_localonly.pl == 1. +# Detach node C and clean the table contents. +sub detach_node_clean_table_data +{ 1a. Maybe say "Detach node C from the node-group of (A, B, C) and clean the table contents

Re: remove redundant check of item pointer

2022-04-27 Thread Junwang Zhao
got it, thanks for the explanation. On Wed, Apr 27, 2022 at 11:34 PM Tom Lane wrote: > > Junwang Zhao writes: > > In function ItemPointerEquals, the ItemPointerGetBlockNumber > > already checked the ItemPointer if valid, there is no need > > to check it again in ItemPointerGetOffset, so use > >

Re: Unstable tests for recovery conflict handling

2022-04-27 Thread Andres Freund
Hi, On 2022-04-27 14:08:45 -0400, Tom Lane wrote: > I wrote: > > It's been kind of hidden by other buildfarm noise, but > > 031_recovery_conflict.pl is not as stable as it should be [1][2][3][4]. > > ... > > I think this is showing us a real bug, ie we sometimes fail to cancel > > the conflicting

Re: Unstable tests for recovery conflict handling

2022-04-27 Thread Andres Freund
Hi, On 2022-04-27 10:11:53 -0700, Mark Dilger wrote: > > > > On Apr 27, 2022, at 9:45 AM, Tom Lane wrote: > > > > [ starting a new thread cuz the shared-stats one is way too long ] > > > > Andres Freund writes: > >> Add minimal tests for recovery conflict handling. > > > > It's been kind of

RE: Data is copied twice when specifying both child and parent table in publication

2022-04-27 Thread shiy.f...@fujitsu.com
On Sun, Apr 24, 2022 2:16 PM Wang, Wei/王 威 wrote: > > Attach the new patches.[suggestions by Amit-San] > The patch for HEAD: > 1. Add a new function to get tables info by a publications array. > The patch for REL14: > 1. Use an alias to make the statement understandable. BTW, I adjusted the > ali

trivial comment fix

2022-04-27 Thread Euler Taveira
Hi, While reading worker.c, I noticed that the referred SQL command was wrong. ALTER SUBSCRIPTION ... REFRESH PUBLICATION instead of ALTER TABLE ... REFRESH PUBLICATION. Trivial fix attached. -- Euler Taveira EDB https://www.enterprisedb.com/ diff --git a/src/backend/replication/logical/worker

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-27 Thread Michael Paquier
On Wed, Apr 27, 2022 at 11:09:45AM -0700, Nathan Bossart wrote: > On Wed, Apr 27, 2022 at 02:16:01PM +0900, Michael Paquier wrote: >> - if (ControlFile->state == DB_IN_ARCHIVE_RECOVERY && >> - ControlFile->checkPointCopy.redo < lastCheckPoint.redo) >> - { >> 7ff23c6 has removed the last c

Re: pgsql: Add contrib/pg_walinspect.

2022-04-27 Thread Thomas Munro
On Wed, Apr 27, 2022 at 10:22 PM Bharath Rupireddy wrote: > Here's v2 patch (up on Thomas's v1 at [1]) using private_data to set > the end of the WAL flag. Please have a look at it. I don't have a strong view on whether it's better to use a NULL error for this private communication between pg_wal

Multi-Master Logical Replication

2022-04-27 Thread Peter Smith
MULTI-MASTER LOGICAL REPLICATION 1.0 BACKGROUND Let’s assume that a user wishes to set up a multi-master environment so that a set of PostgreSQL instances (nodes) use logical replication to share tables with every other node in the set. We define this as a multi-master logical replication (MMLR)

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Tomas Vondra
Hi, so I've been looking at tweaking the code so that the behavior matches Alvaro's expectations. It passes check-world but I'm not claiming it's nowhere near commitable - the purpose is mostly to give better idea of how invasive the change is etc. As described earlier, this abandons the idea of

Re: [RFC] building postgres with meson -v8

2022-04-27 Thread Peter Eisentraut
Here is a patch that adds in NLS. There are some opportunities to improve this. For example, we could move the list of languages from the meson.build files into separate LINGUAS files, which could be shared with the makefile-based build system. I need to research this a bit more. Also, thi

Re: avoid multiple hard links to same WAL file after a crash

2022-04-27 Thread Nathan Bossart
On Wed, Apr 27, 2022 at 04:09:20PM +0900, Michael Paquier wrote: > I am not sure that have any need to backpatch this change based on the > unlikeliness of the problem, TBH. One thing that is itching me a bit, > like Robert upthread, is that we don't check anymore that the newfile > does not exist

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-27 Thread Nathan Bossart
On Wed, Apr 27, 2022 at 02:16:01PM +0900, Michael Paquier wrote: > On Tue, Apr 26, 2022 at 08:26:09PM -0700, Nathan Bossart wrote: >> On Wed, Apr 27, 2022 at 10:43:53AM +0900, Kyotaro Horiguchi wrote: >>> + ControlFile->minRecoveryPoint = InvalidXLogRecPtr; >>> + ControlFile->mi

Re: Unstable tests for recovery conflict handling

2022-04-27 Thread Tom Lane
I wrote: > It's been kind of hidden by other buildfarm noise, but > 031_recovery_conflict.pl is not as stable as it should be [1][2][3][4]. > ... > I think this is showing us a real bug, ie we sometimes fail to cancel > the conflicting query. After digging around in the code, I think this is almos

Re: Unstable tests for recovery conflict handling

2022-04-27 Thread Mark Dilger
> On Apr 27, 2022, at 10:11 AM, Mark Dilger > wrote: > > I'll try again on master Still with coverage and dtrace enabled, I get the same thing, except that master formats the logs a bit differently: # Postmaster PID for node "primary" is 19797 psql::1: ERROR: prepared transaction with

Re: pgsql: Add contrib/pg_walinspect.

2022-04-27 Thread Jeff Davis
On Wed, 2022-04-27 at 13:47 +0530, Bharath Rupireddy wrote: > I found an easy way to reproduce this consistently (I think on any > server): > > I basically generated huge WAL record (I used a fun extension that I > wrote - https://github.com/BRupireddy/pg_synthesize_wal, but one can > use pg_logic

Re: Unstable tests for recovery conflict handling

2022-04-27 Thread Mark Dilger
> On Apr 27, 2022, at 9:45 AM, Tom Lane wrote: > > [ starting a new thread cuz the shared-stats one is way too long ] > > Andres Freund writes: >> Add minimal tests for recovery conflict handling. > > It's been kind of hidden by other buildfarm noise, but > 031_recovery_conflict.pl is not a

Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.

2022-04-27 Thread Julien Rouhaud
On Wed, Apr 27, 2022 at 03:04:23PM +, Wilm Hoyer wrote: > > I used the following hack to get the "real" Major and Minor Version of > Windows - it's in C# (.Net) and needs to be adjusted (you can compile as x64 > and use a long-long as return value ) to return the Service Number too and > transl

Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.

2022-04-27 Thread Julien Rouhaud
On Wed, Apr 27, 2022 at 05:13:12PM +0900, Michael Paquier wrote: > On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote: > > so I'm still on the opinion that we should > > unconditionally use the FILE_MAP_LARGE_PAGES flag if it's defined and call > > it a > > day. > > Are we sure that t

Unstable tests for recovery conflict handling

2022-04-27 Thread Tom Lane
[ starting a new thread cuz the shared-stats one is way too long ] Andres Freund writes: > Add minimal tests for recovery conflict handling. It's been kind of hidden by other buildfarm noise, but 031_recovery_conflict.pl is not as stable as it should be [1][2][3][4]. Three of those failures loo

Re: Wrong rows count in EXPLAIN

2022-04-27 Thread Пантюшин Александр Иванович
Hi, >Which Postgres version do you use? I checked this on PG 11 postgres=# select version(); version -

Re: remove redundant check of item pointer

2022-04-27 Thread Tom Lane
Junwang Zhao writes: > In function ItemPointerEquals, the ItemPointerGetBlockNumber > already checked the ItemPointer if valid, there is no need > to check it again in ItemPointerGetOffset, so use > ItemPointerGetOffsetNumberNoCheck instead. I do not think this change is worth making. The point

remove redundant check of item pointer

2022-04-27 Thread Junwang Zhao
In function ItemPointerEquals, the ItemPointerGetBlockNumber already checked the ItemPointer if valid, there is no need to check it again in ItemPointerGetOffset, so use ItemPointerGetOffsetNumberNoCheck instead. -- Regards Junwang Zhao v1-0001-remove-redundant-check-of-item-pointer.patch Descr

AW: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.

2022-04-27 Thread Wilm Hoyer
On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote: >> I searched a bit and apparently some people are using this function >> directly opening some dll, which seems wrong. > I was wondering about this whole business, and the manifest approach is a > *horrible* design for an API whe

cirrus: run macos with COPY_PARSE_PLAN_TREES etc

2022-04-27 Thread Justin Pryzby
fork: <20220325000933.vgazz7pjk2ytj...@alap3.anarazel.de> On Thu, Mar 24, 2022 at 05:09:33PM -0700, Andres Freund wrote: > On 2022-03-24 18:51:30 -0400, Andrew Dunstan wrote: > > I wonder if we should add these compile flags to the cfbot's setup? > > Yes, I think we should. There's a bit of discu

Re: Wrong rows count in EXPLAIN

2022-04-27 Thread Bruce Momjian
On Wed, Apr 27, 2022 at 09:44:21AM -0400, Tom Lane wrote: > =?koi8-r?B?8MHO1MDbyc4g4czFy9PBzsTSIOnXwc7P18ne?= > writes: > > When I create a new table, and then I evaluate the execution of the SELECT > > query, I see a strange rows count in EXPLAIN > > CREATE TABLE test1(f INTEGER PRIMARY KEY NOT

Re: Wrong rows count in EXPLAIN

2022-04-27 Thread Tom Lane
=?koi8-r?B?8MHO1MDbyc4g4czFy9PBzsTSIOnXwc7P18ne?= writes: > When I create a new table, and then I evaluate the execution of the SELECT > query, I see a strange rows count in EXPLAIN > CREATE TABLE test1(f INTEGER PRIMARY KEY NOT NULL); > ANALYZE test1; > EXPLAIN SELECT * FROM test1; >

Re: Skipping schema changes in publication

2022-04-27 Thread vignesh C
On Tue, Apr 26, 2022 at 11:32 AM osumi.takami...@fujitsu.com wrote: > > On Thursday, April 21, 2022 12:15 PM vignesh C wrote: > > Updated patch by changing the syntax to use EXCEPT instead of SKIP. > Hi > > > This is my review comments on the v2 patch. > > (1) gram.y > > I think we can make a uni

[PATCH v1] remove redundant check of item pointer

2022-04-27 Thread Junwang Zhao
In function ItemPointerEquals, the ItemPointerGetBlockNumber already checked the ItemPointer if valid, there is no need to check it again in ItemPointerGetOffset, so use ItemPointerGetOffsetNumberNoCheck instead. Signed-off-by: Junwang Zhao --- src/backend/storage/page/itemptr.c | 4 ++-- 1 file

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Amit Kapila
On Wed, Apr 27, 2022 at 4:27 PM Alvaro Herrera wrote: > > On 2022-Apr-27, Amit Kapila wrote: > > > On Wed, Apr 27, 2022 at 3:13 PM Alvaro Herrera > > wrote: > > > > > Changing this to behave the way you expect would be quite difficult, > > > > because at the moment we build a single OR expressio

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Alvaro Herrera
On 2022-Apr-27, Amit Kapila wrote: > On Wed, Apr 27, 2022 at 3:13 PM Alvaro Herrera > wrote: > > > Changing this to behave the way you expect would be quite difficult, > > > because at the moment we build a single OR expression from all the row > > > filters. We'd have to keep the individual ex

Re: Wrong rows count in EXPLAIN

2022-04-27 Thread David Rowley
On Wed, 27 Apr 2022 at 21:08, Andrey Borodin wrote: > Which Postgres version do you use? 3d351d91 changed things so we could tell the difference between a relation which was analyzed and is empty vs a relation that's never been analyzed. That's why you're not seeing the same behaviour as the OP.

Re: Wrong rows count in EXPLAIN

2022-04-27 Thread Andrey Borodin
> 27 апр. 2022 г., в 15:17, Пантюшин Александр Иванович > написал(а): > > Hi, > >Which Postgres version do you use? > I checked this on PG 11 > ... > and on PG 13 Yes, I think before 3d351d91 it was impossible to distinguish between actually empty and never analyzed table. But now it is wo

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Amit Kapila
On Wed, Apr 27, 2022 at 3:13 PM Alvaro Herrera wrote: > > On 2022-Apr-26, Tomas Vondra wrote: > > > I'm not quite sure which of the two behaviors is more "desirable". In a > > way, it's somewhat similar to publish_as_relid, which is also calculated > > not considering which of the row filters matc

Re: pgsql: Add contrib/pg_walinspect.

2022-04-27 Thread Bharath Rupireddy
On Wed, Apr 27, 2022 at 1:47 PM Bharath Rupireddy wrote: > > > > > > I've now done several runs with your patch and not seen the test failure. > > > However, I think we ought to rethink this API a bit rather than just > > > apply the patch as-is. Even if it were documented, relying on > > > error

RE: bogus: logical replication rows/cols combinations

2022-04-27 Thread houzj.f...@fujitsu.com
On Wednesday, April 27, 2022 12:56 PM From: Amit Kapila wrote: > On Tue, Apr 26, 2022 at 4:00 AM Tomas Vondra > wrote: > > > > On 4/25/22 17:48, Alvaro Herrera wrote: > > > > > The desired result on subscriber is: > > > > > > table uno; > > > a │ b │ c > > > ┼───┼─── > > > 1 │ 2 │ > > >

Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.

2022-04-27 Thread Magnus Hagander
On Tue, Apr 26, 2022, 05:55 Julien Rouhaud wrote: > Hi, > > Please keep the list in copy, especially if that's about Windows specific > as > I'm definitely not very knowledgeable about it. > > On Fri, Apr 01, 2022 at 09:18:03AM +, Wilm Hoyer wrote: > > > > If you don't wanna go the manifest w

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Alvaro Herrera
On 2022-Apr-27, Amit Kapila wrote: > On Tue, Apr 26, 2022 at 4:00 AM Tomas Vondra > wrote: > > I can take a stab at it, but it seems strange to not apply the same > > logic to evaluation of publish_as_relid. > > Yeah, the current behavior seems to be consistent with what we already > do. Sorry

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Alvaro Herrera
On 2022-Apr-26, Tomas Vondra wrote: > I'm not quite sure which of the two behaviors is more "desirable". In a > way, it's somewhat similar to publish_as_relid, which is also calculated > not considering which of the row filters match? I grepped doc/src/sgml for `publish_as_relid` and found no hit

Re: Wrong rows count in EXPLAIN

2022-04-27 Thread Andrey Borodin
Hi! > 26 апр. 2022 г., в 13:45, Пантюшин Александр Иванович > написал(а): > > When I create a new table, and then I evaluate the execution of the SELECT > query, I see a strange rows count in EXPLAIN > CREATE TABLE test1(f INTEGER PRIMARY KEY NOT NULL); > ANALYZE test1; > EXPLAIN SELECT * FROM

Re: pgsql: Add contrib/pg_walinspect.

2022-04-27 Thread Bharath Rupireddy
On Wed, Apr 27, 2022 at 8:57 AM Bharath Rupireddy wrote: > > On Wed, Apr 27, 2022 at 8:45 AM Tom Lane wrote: > > > > I wrote: > > > Thomas Munro writes: > > >> BTW If you had your local change from debug.patch (upthread), that'd > > >> defeat the patch. I mean this: > > > > >> +if(!*err

Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.

2022-04-27 Thread Michael Paquier
On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote: > I searched a bit and apparently some people are using this function directly > opening some dll, which seems wrong. I was wondering about this whole business, and the manifest approach is a *horrible* design for an API where the goa

Re: Fix primary crash continually with invalid checkpoint after promote

2022-04-27 Thread Nathan Bossart
On Tue, Apr 26, 2022 at 03:16:13PM +0800, Zhao Rui wrote: > In function CreateRestartPoint, control file is updated and old wals are > removed. But in some situations, control file is not updated, old wals are > still removed. Thus produces an invalid checkpoint with nonexistent wal. > Crucial l

Wrong rows count in EXPLAIN

2022-04-27 Thread Пантюшин Александр Иванович
When I create a new table, and then I evaluate the execution of the SELECT query, I see a strange rows count in EXPLAIN CREATE TABLE test1(f INTEGER PRIMARY KEY NOT NULL); ANALYZE test1; EXPLAIN SELECT * FROM test1; QUERY PLAN

Fix primary crash continually with invalid checkpoint after promote

2022-04-27 Thread Zhao Rui
Newly promoted primary may leave an invalid checkpoint. In function CreateRestartPoint, control file is updated and old wals are removed. But in some situations, control file is not updated, old wals are still removed. Thus produces an invalid checkpoint with nonexistent wal. Crucial log: "inva

Re: Fix NULL pointer reference in _outPathTarget()

2022-04-27 Thread Peter Eisentraut
On 22.04.22 16:18, Tom Lane wrote: Peter Eisentraut writes: On 20.04.22 18:53, Tom Lane wrote: Yeah, that's another way to do it. I think though that the unresolved question is whether or not we want the field name to appear in the output when the field is null. I believe that I intentionall

Re: avoid multiple hard links to same WAL file after a crash

2022-04-27 Thread Michael Paquier
On Tue, Apr 26, 2022 at 01:09:35PM -0700, Nathan Bossart wrote: > Here is an attempt at creating something that can be back-patched. 0001 > simply replaces calls to durable_rename_excl() with durable_rename() and is > intended to be back-patched. 0002 removes the definition of > durable_rename_ex