Re: pg_dump, pg_basebackup don't error out with wrong option for "--format"

2021-11-25 Thread Michael Paquier
On Thu, Nov 25, 2021 at 11:22:11AM +0530, Bharath Rupireddy wrote: > On Thu, Nov 25, 2021 at 11:02 AM Dilip Kumar wrote: >> I think for parsing we use getopt_long(), as per that if you use the >> prefix of the string and that is not conflicting with any other option >> then that is allowed. So --

Re: Deduplicate code updating ControleFile's DBState.

2021-11-25 Thread Michael Paquier
On Thu, Nov 25, 2021 at 04:04:23PM +0900, Michael Paquier wrote: > I have not check the performance implication of that with a micro > benchmark or the like, but I can get behind 0001 on consistency > grounds between the backend and the frontend. /* Now create pg_control */ InitControlFile

Re: row filtering for logical replication

2021-11-25 Thread Amit Kapila
On Fri, Nov 26, 2021 at 12:01 PM Peter Smith wrote: > > On Fri, Nov 26, 2021 at 4:18 PM Peter Smith wrote: > > > > > > Do you mean to say that we should give an error on Update/Delete if any > > > > of the > > > > publications contain table rowfilter that has columns that are not part > > > > o

Re: [PATCH] ALTER tab completion

2021-11-25 Thread Michael Paquier
On Fri, Nov 26, 2021 at 01:55:46PM +0900, Ken Kato wrote: > I noticed that there are some tab completions missing for the following > commands: > -ALTER DEFAULT PRIVILEGES: missing FOR USER FOR ROLE is an equivalent. That does not seem mandatory to me. > -ALTER FOREIGN DATA WRAPPER: missing NO H

Re: row filtering for logical replication

2021-11-25 Thread Peter Smith
On Fri, Nov 26, 2021 at 4:18 PM Peter Smith wrote: > > On Fri, Nov 26, 2021 at 4:05 PM houzj.f...@fujitsu.com > wrote: > > > > On Fri, Nov 26, 2021 11:32 AM Amit Kapila wrote: > > > On Thu, Nov 25, 2021 at 7:39 PM Euler Taveira wrote: > > > > > > > > On Thu, Nov 25, 2021, at 10:39 AM, houzj.f..

Re: pg_waldump stucks with options --follow or -f and --stats or -z

2021-11-25 Thread Michael Paquier
On Sat, Nov 20, 2021 at 11:46:35AM +0530, Bharath Rupireddy wrote: > Thanks. Here's the v3 patch, a much simpler one. Please review it. + pqsignal(SIGINT, SignalHandlerForTermination); + pqsignal(SIGTERM, SignalHandlerForTermination); + pqsignal(SIGQUIT, SignalHandlerForTermination); FWIW, I

Inconsistent results from seqscan and gist indexscan

2021-11-25 Thread Richard Guo
Here is how it can be reproduced. create table point_tbl (f1 point); insert into point_tbl(f1) values ('(5.1, 34.5)'); insert into point_tbl(f1) values (' ( Nan , NaN ) '); analyze; create index gpointind on point_tbl using gist (f1); set enable_seqscan to on; set enable_indexscan to off; # sel

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-25 Thread Michael Paquier
On Thu, Nov 25, 2021 at 06:19:03PM -0800, SATYANARAYANA NARLAPURAM wrote: > Is there a plan in place to remove the exclusive backup option from the > core in PG 15/16? This was discussed, but removing it could also harm users relying on it. Perhaps it could be revisited, but I am not sure if this

Re: row filtering for logical replication

2021-11-25 Thread Peter Smith
On Fri, Nov 26, 2021 at 4:05 PM houzj.f...@fujitsu.com wrote: > > On Fri, Nov 26, 2021 11:32 AM Amit Kapila wrote: > > On Thu, Nov 25, 2021 at 7:39 PM Euler Taveira wrote: > > > > > > On Thu, Nov 25, 2021, at 10:39 AM, houzj.f...@fujitsu.com wrote: > > > > > > When researching and writing a top-

RE: row filtering for logical replication

2021-11-25 Thread houzj.f...@fujitsu.com
On Fri, Nov 26, 2021 11:32 AM Amit Kapila wrote: > On Thu, Nov 25, 2021 at 7:39 PM Euler Taveira wrote: > > > > On Thu, Nov 25, 2021, at 10:39 AM, houzj.f...@fujitsu.com wrote: > > > > When researching and writing a top-up patch about this. > > I found a possible issue which I'd like to confirm f

[PATCH] ALTER tab completion

2021-11-25 Thread Ken Kato
Hi hackers, I noticed that there are some tab completions missing for the following commands: -ALTER DEFAULT PRIVILEGES: missing FOR USER -ALTER FOREIGN DATA WRAPPER: missing NO HANDLER, NO VALIDATOR -ALTER SEQUENCE: missing AS -ALTER VIEW: no completion after ALTER COLUMN column_name -ALTER TR

Re: Windows build warnings

2021-11-25 Thread Tom Lane
Greg Nancarrow writes: > AFAICS, the fundamental difference here seems to be that the GCC > compiler still regards a variable as "unused" if it is never read, > whereas if the variable is set (but not necessarily read) that's > enough for the Windows C compiler to regard it as "used". It depends.

Re: Windows build warnings

2021-11-25 Thread Greg Nancarrow
On Thu, Nov 25, 2021 at 11:03 PM Daniel Gustafsson wrote: > > To silence the warnings in the meantime (if the rework at all happens) we > should either apply the patch from Greg or add C4101 to disablewarnings in > src/tools/msvc/Project.pm as mentioned above. On top of that, we should apply > th

Re: prevent immature WAL streaming

2021-11-25 Thread Amul Sul
On Fri, Nov 26, 2021 at 1:42 AM Tom Lane wrote: > > Alvaro Herrera writes: > > On 2021-Nov-25, Tom Lane wrote: > >> Really? AFAICS the WAL record contains the correct value, or at least > >> we should define that one as being correct, for precisely this reason. > > > I don't know what is the cor

Re: row filtering for logical replication

2021-11-25 Thread Greg Nancarrow
On Fri, Nov 26, 2021 at 1:16 PM houzj.f...@fujitsu.com wrote: > > Based on this direction, I tried to write a top up POC patch(0005) which I'd > like to share. > I noticed a minor issue. In the top-up patch, the following error message detail: + errdetail("Not all row filter columns are not par

Re: RFC: Logging plan of the running query

2021-11-25 Thread torikoshia
On 2021-11-17 22:44, Ekaterina Sokolova wrote: Hi! You forgot my last fix to build correctly on Mac. I have added it. Thanks for the notification! Since the patch could not be applied to the HEAD anymore, I also updated it. About our discussion of pg_query_state: torikoshia писал 2021-11

Re: row filtering for logical replication

2021-11-25 Thread Amit Kapila
On Thu, Nov 25, 2021 at 7:39 PM Euler Taveira wrote: > > On Thu, Nov 25, 2021, at 10:39 AM, houzj.f...@fujitsu.com wrote: > > When researching and writing a top-up patch about this. > I found a possible issue which I'd like to confirm first. > > It's possible the table is published in two publicat

Re: pg_get_publication_tables() output duplicate relid

2021-11-25 Thread Amit Kapila
On Fri, Nov 26, 2021 at 7:10 AM houzj.f...@fujitsu.com wrote: > > On Thursday, November 25, 2021 4:57 PM Amit Kapila > wrote: > > On Thu, Nov 25, 2021 at 1:30 PM Amit Langote > > > > > > I agree with backpatching the doc fix. I've attached a diff against > > > master, though it also appears to

Re: pg_upgrade and publication/subscription problem

2021-11-25 Thread Amit Kapila
On Thu, Nov 25, 2021 at 8:00 PM Marcos Pegoraro wrote: >> >> Yes, the way you are doing I think it is bound to happen. There is >> some discussion about why this is happening in email [2]. AFAIK, it is >> not documented and if so, I think it will be a good idea to document >> > And my problem rema

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-25 Thread SATYANARAYANA NARLAPURAM
Thanks Michael! This is a known issue with exclusive backups, which is a reason why > non-exclusive backups have been implemented. pg_basebackup does that, > and using "false" as the third argument of pg_start_backup() would > have the same effect. So I would recommend to switch to that. > Is t

RE: Skipping logical replication transactions on subscriber side

2021-11-25 Thread tanghy.f...@fujitsu.com
On Friday, November 26, 2021 9:30 AM Masahiko Sawada wrote: > > Indeed. Attached an updated patch. Thanks! Thanks for your patch. A small comment: + OID of the relation that the worker is synchronizing; null for the + main apply worker Should we modify it to "OID of the relation t

RE: pg_get_publication_tables() output duplicate relid

2021-11-25 Thread houzj.f...@fujitsu.com
On Thursday, November 25, 2021 4:57 PM Amit Kapila wrote: > On Thu, Nov 25, 2021 at 1:30 PM Amit Langote > > > > I agree with backpatching the doc fix. I've attached a diff against > > master, though it also appears to apply to 13 and 14 branches. > > > > I think we can for publish_via_partit

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Masahiko Sawada
On Thu, Nov 25, 2021 at 10:06 PM houzj.f...@fujitsu.com wrote: > > On Thur, Nov 25, 2021 8:29 PM Masahiko Sawada wrote: > > On Thu, Nov 25, 2021 at 1:57 PM Amit Kapila wrote: > > > > > > On Wed, Nov 24, 2021 at 5:14 PM Masahiko Sawada > > wrote: > > > > > > > > Changed. I've removed first_error

Re: WIP: WAL prefetch (another approach)

2021-11-25 Thread Tomas Vondra
Hi, It's great you posted a new version of this patch, so I took a look a brief look at it. The code seems in pretty good shape, I haven't found any real issues - just two minor comments: This seems a bit strange: #define DEFAULT_DECODE_BUFFER_SIZE 0x1 Why not to define this as a simple

Re: prevent immature WAL streaming

2021-11-25 Thread Tom Lane
I wrote: > However, this seems too forgiving: ... also, I don't know if you intended this already, but the VerifyOverwriteContrecord change should only be applied in back branches. There's no need for it in HEAD. regards, tom lane

Re: prevent immature WAL streaming

2021-11-25 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Nov-25, Tom Lane wrote: >> Really? AFAICS the WAL record contains the correct value, or at least >> we should define that one as being correct, for precisely this reason. > I don't know what is the correct value for a record that comes exactly > after the page he

Re: Non-superuser subscription owners

2021-11-25 Thread Jeff Davis
On Thu, 2021-11-25 at 09:51 +0530, Amit Kapila wrote: > Won't it be better to just check if the current user is superuser > before applying each change as a matter of this first patch? Sorry, I > was under impression that first, we want to close the current gap > where we allow to proceed with repl

Re: prevent immature WAL streaming

2021-11-25 Thread Alvaro Herrera
On 2021-Nov-25, Tom Lane wrote: > Alvaro Herrera writes: > > > The problem is that the bug occurs while writing the WAL record. Fixed > > servers won't produce such records, but if you run an unpatched server > > and it happens to write one, without a mitigation you cannot get away > > from FAT

Re: prevent immature WAL streaming

2021-11-25 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Nov-25, Tom Lane wrote: >> Uh, why? The fix should remove the problem, and if it doesn't, we're >> still looking at inconsistent WAL aren't we? > The problem is that the bug occurs while writing the WAL record. Fixed > servers won't produce such records, but if

Re: prevent immature WAL streaming

2021-11-25 Thread Alvaro Herrera
On 2021-Nov-25, Tom Lane wrote: > Alvaro Herrera writes: > > Oh, but also I think I should push a mitigation in case a production > > system hits this problem: maybe reduce the message from FATAL to WARNING > > if the registered LSN is at a page boundary. > > Uh, why? The fix should remove the

Re: prevent immature WAL streaming

2021-11-25 Thread Tom Lane
Alvaro Herrera writes: > Oh, but also I think I should push a mitigation in case a production > system hits this problem: maybe reduce the message from FATAL to WARNING > if the registered LSN is at a page boundary. Uh, why? The fix should remove the problem, and if it doesn't, we're still looki

Re: prevent immature WAL streaming

2021-11-25 Thread Alvaro Herrera
Oh, but also I think I should push a mitigation in case a production system hits this problem: maybe reduce the message from FATAL to WARNING if the registered LSN is at a page boundary. -- Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/ "Entristecido, Wutra

Re: prevent immature WAL streaming

2021-11-25 Thread Alvaro Herrera
On 2021-Nov-25, Amul Sul wrote: > In XLogReadRecord(), both the variables being compared have > inconsistency in the assignment -- one gets assigned from > state->currRecPtr where other is from RecPtr. > > . > state->overwrittenRecPtr = state->currRecPtr; > . > state->abortedRecPtr = RecP

Re: Non-decimal integer literals

2021-11-25 Thread John Naylor
Hi Peter, 0001 -/* we no longer allow unary minus in numbers. - * instead we pass it separately to parser. there it gets - * coerced via doNegate() -- Leon aug 20 1999 +/* + * Numbers + * + * Unary minus is not part of a number here. Instead we pass it separately to + * parser, and there it gets

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2021-11-25 Thread Mark Dilger
> On Nov 24, 2021, at 12:53 PM, Bossart, Nathan wrote: > > Another option we might consider is only checking for the > HEAP_XMAX_LOCK_ONLY bit instead of everything in > HEAP_XMAX_IS_LOCKED_ONLY. IIUC everything else is only expected to > happen for upgrades from v9.2 or earlier, so it might

Re: Reduce function call costs on ELF platforms

2021-11-25 Thread Andrew Dunstan
On 11/24/21 22:57, Andres Freund wrote: > >> Which things does it break exactly? > -Bsymbolic causes symbols that are defined and referenced within one shared > library to use that definition. E.g. if a shared lib has a function > "do_something()" and some of its code calls do_something(), you ca

Re: Non-decimal integer literals

2021-11-25 Thread Zhihong Yu
On Thu, Nov 25, 2021 at 5:18 AM Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > On 01.11.21 07:09, Peter Eisentraut wrote: > > Here is an updated patch for this. It's the previous patch polished a > > bit more, and it contains changes so that numeric literals reject > > trailing id

Re: TOAST - why separate visibility map

2021-11-25 Thread Virender Singla
"Given the size of toasted data, the overhead is unlikely to be a significant overhead. It's much more an issue for the main table, where narrow rows are common." Completely agree, row size should not be a big concern for toast tables. However write amplification will happen with vacuum freeze wh

Re: pg_upgrade and publication/subscription problem

2021-11-25 Thread Marcos Pegoraro
> > The reason is after an upgrade, there won't be any data in > pg_subscription_rel, and only when you tried to refresh it is trying > to sync again which leads to the "duplicate key value ..." problem you > are seeing. > > So, is pg_upgrade populating pg_subscription and not pg_subscription_rel ?

Re: row filtering for logical replication

2021-11-25 Thread Euler Taveira
On Thu, Nov 25, 2021, at 10:39 AM, houzj.f...@fujitsu.com wrote: > When researching and writing a top-up patch about this. > I found a possible issue which I'd like to confirm first. > > It's possible the table is published in two publications A and B, publication > A > only publish "insert" , pu

Re: pg_upgrade and publication/subscription problem

2021-11-25 Thread Amit Kapila
On Thu, Nov 25, 2021 at 5:13 PM Marcos Pegoraro wrote: > > A publication for all tables was running fine, Master is a PostgreSQL 11.11. > Replica was running version 13 (don´t remember minor version). > > Then we tried to update only subscriber server, nothing was done on master > side. > > Then

RE: row filtering for logical replication

2021-11-25 Thread houzj.f...@fujitsu.com
On Wed, Nov 24, 2021 1:46 PM Amit Kapila wrote: > On Wed, Nov 24, 2021 at 6:51 AM houzj.f...@fujitsu.com > wrote: > > > > On Tues, Nov 23, 2021 6:16 PM Amit Kapila wrote: > > > On Tue, Nov 23, 2021 at 1:29 PM houzj.f...@fujitsu.com > > > wrote: > > > > > > > > On Tues, Nov 23, 2021 2:27 PM vign

Re: Support for NSS as a libpq TLS backend

2021-11-25 Thread Joshua Brindle
On Wed, Nov 24, 2021 at 8:49 AM Joshua Brindle wrote: > > On Wed, Nov 24, 2021 at 8:46 AM Joshua Brindle > wrote: > > > > On Wed, Nov 24, 2021 at 6:59 AM Daniel Gustafsson wrote: > > > > > > > On 23 Nov 2021, at 23:39, Joshua Brindle > > > > wrote: > > > > > > > It no longer happens with v49,

Re: Yet another fast GiST build

2021-11-25 Thread Andrey Borodin
> 17 нояб. 2021 г., в 16:33, Daniel Gustafsson написал(а): > >> On 5 Jul 2021, at 08:27, Emre Hasegeli wrote: > >> ... >> >> I couldn't understand patch number 2 "Remove DEBUG1 verification". It >> seems like something rather useful. Emre, thanks for the review! And sorry for this delay.

Re: Non-decimal integer literals

2021-11-25 Thread Peter Eisentraut
On 01.11.21 07:09, Peter Eisentraut wrote: Here is an updated patch for this.  It's the previous patch polished a bit more, and it contains changes so that numeric literals reject trailing identifier parts without whitespace in between, as discussed. Maybe I should split that into incremental p

RE: Skipping logical replication transactions on subscriber side

2021-11-25 Thread houzj.f...@fujitsu.com
On Thur, Nov 25, 2021 8:29 PM Masahiko Sawada wrote: > On Thu, Nov 25, 2021 at 1:57 PM Amit Kapila wrote: > > > > On Wed, Nov 24, 2021 at 5:14 PM Masahiko Sawada > wrote: > > > > > > Changed. I've removed first_error_time as per discussion on the > > > thread for adding xact stats. > > > > > > >

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Masahiko Sawada
On Thu, Nov 25, 2021 at 9:08 PM Greg Nancarrow wrote: > > On Wed, Nov 24, 2021 at 10:44 PM Masahiko Sawada > wrote: > > > > I've attached an updated version patch. Unless I miss something, all > > comments I got so far have been incorporated into this patch. Please > > review it. > > > > Only a

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Masahiko Sawada
On Thu, Nov 25, 2021 at 7:36 PM vignesh C wrote: > > On Wed, Nov 24, 2021 at 5:14 PM Masahiko Sawada wrote: > > > > On Wed, Nov 17, 2021 at 8:14 PM Amit Kapila wrote: > > > > > > On Tue, Nov 16, 2021 at 12:01 PM Masahiko Sawada > > > wrote: > > > > > > > > Right. I've fixed this issue and atta

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Masahiko Sawada
On Thu, Nov 25, 2021 at 1:57 PM Amit Kapila wrote: > > On Wed, Nov 24, 2021 at 5:14 PM Masahiko Sawada wrote: > > > > Changed. I've removed first_error_time as per discussion on the thread > > for adding xact stats. > > > > We also agreed to change the column names to start with last_error_* > [1

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2021-11-25 Thread Bharath Rupireddy
On Thu, Nov 25, 2021 at 3:49 PM Bharath Rupireddy wrote: > > > > Thanks all. Here's the v1 patch set for the new extension pg_walinspect. > > Note that I didn't include the documentation part now, I will be doing it a > > bit later. > > > > Please feel free to review and provide your thoughts. >

Re: pg_get_publication_tables() output duplicate relid

2021-11-25 Thread Amit Langote
On Thu, Nov 25, 2021 at 5:57 PM Amit Kapila wrote: > On Thu, Nov 25, 2021 at 1:30 PM Amit Langote wrote: > > I agree with backpatching the doc fix. I've attached a diff against > > master, though it also appears to apply to 13 and 14 branches. > > I think we can for publish_via_partition_root,

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Greg Nancarrow
On Wed, Nov 24, 2021 at 10:44 PM Masahiko Sawada wrote: > > I've attached an updated version patch. Unless I miss something, all > comments I got so far have been incorporated into this patch. Please > review it. > Only a couple of minor points: src/backend/postmaster/pgstat.c (1) pgstat_get_sub

Re: Windows build warnings

2021-11-25 Thread Daniel Gustafsson
> On 22 Nov 2021, at 16:06, Tom Lane wrote: > > Alvaro Herrera writes: >> .. but see >> https://postgr.es/m/cah2-wznwwu+9on9nzcnztk7ua238mctgpxyr1ty7u_msn5z...@mail.gmail.com >> where this was already discussed. I think if we're going to workaround >> PG_USED_FOR_ASSERTS_ONLY not actually worki

pg_upgrade and publication/subscription problem

2021-11-25 Thread Marcos Pegoraro
A publication for all tables was running fine, Master is a PostgreSQL 11.11. Replica was running version 13 (don´t remember minor version). Then we tried to update only subscriber server, nothing was done on master side. Then we did ... - installed postgresql-14. - configured postgresql.conf to b

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-11-25 Thread Dilip Kumar
On Thu, Nov 25, 2021 at 1:07 PM Greg Nancarrow wrote: > > On Tue, Oct 5, 2021 at 7:07 PM Dilip Kumar wrote: > > > > Patch details: > > 0001 to 0006 implements an approach1 > > 0007 removes the code of pg_class scanning and adds the directory scan. > > > > I had a scan through the patches, though

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread vignesh C
On Wed, Nov 24, 2021 at 5:14 PM Masahiko Sawada wrote: > > On Wed, Nov 17, 2021 at 8:14 PM Amit Kapila wrote: > > > > On Tue, Nov 16, 2021 at 12:01 PM Masahiko Sawada > > wrote: > > > > > > Right. I've fixed this issue and attached an updated patch. One very minor comment: conflict can be move

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2021-11-25 Thread Bharath Rupireddy
On Thu, Nov 18, 2021 at 6:43 PM Bharath Rupireddy wrote: > > On Thu, Oct 7, 2021 at 10:43 AM Bharath Rupireddy > wrote: > > > Looking at the proposed API from the initial email, I like that there's > > > both stats functionality and WAL record inspection functionality > > > (similar to pg_waldum

Re: pg_get_publication_tables() output duplicate relid

2021-11-25 Thread Amit Kapila
On Thu, Nov 25, 2021 at 1:30 PM Amit Langote wrote: > > On Wed, Nov 24, 2021 at 5:44 PM Amit Kapila wrote: > > On Wed, Nov 24, 2021 at 12:02 PM Amit Langote > > wrote: > > > So yeah, documenting the ATTACH issue as a limitation sounds like the > > > best course for now. I might word it as foll

Re: pg_get_publication_tables() output duplicate relid

2021-11-25 Thread Amit Langote
On Wed, Nov 24, 2021 at 5:44 PM Amit Kapila wrote: > On Wed, Nov 24, 2021 at 12:02 PM Amit Langote wrote: > > So yeah, documenting the ATTACH issue as a limitation sounds like the > > best course for now. I might word it as follows and add it under > > Notes at https://www.postgresql.org/docs/cu