RE: row filtering for logical replication

2022-01-11 Thread tanghy.f...@fujitsu.com
On Tuesday, January 11, 2022 10:16 AM houzj.f...@fujitsu.com wrote: > > Attach the v62 patch set which address the above comments and slightly > adjust the commit message in 0002 patch. > I saw a possible problem about Row-Filter tablesync SQL, which is related to partition table. If a parent

Re: Improve error handling of HMAC computations and SCRAM

2022-01-11 Thread Sergey Shinderuk
On 11.01.2022 10:57, Michael Paquier wrote: On Tue, Jan 11, 2022 at 10:50:50AM +0300, Sergey Shinderuk wrote: + * Returns a static string providing errors about an error that happened "errors about an error" looks odd. Sure, that could be reworded. What about "providing details about an erro

Re: a misbehavior of partition row movement (?)

2022-01-11 Thread Amit Langote
On Thu, Jan 6, 2022 at 9:36 PM Alvaro Herrera wrote: > On 2022-Jan-06, Amit Langote wrote: > > On Thu, Jan 6, 2022 at 7:27 AM Alvaro Herrera > > wrote: > > > > I have pushed it thinking that we would not backpatch any of this fix. > > > However, after running the tests and realizing that I didn'

Re: Skipping logical replication transactions on subscriber side

2022-01-11 Thread Masahiko Sawada
On Tue, Jan 11, 2022 at 3:12 PM Amit Kapila wrote: > > On Tue, Jan 11, 2022 at 8:52 AM Masahiko Sawada wrote: > > > > On Mon, Jan 10, 2022 at 8:50 PM Amit Kapila wrote: > > > > > > I was thinking what if we don't advance origin explicitly in this > > > case? Actually, that will be no different t

Re: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers

2022-01-11 Thread Kyotaro Horiguchi
At Fri, 7 Jan 2022 09:44:15 -0800, SATYANARAYANA NARLAPURAM wrote in > On Fri, Jan 7, 2022 at 12:27 AM Kyotaro Horiguchi > wrote: > > One is to serialize WAL sending (of course it is unacceptable at all) > > or aotehr is to send WAL to all standbys at once then make the > > decision after makin

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2022-01-11 Thread Konstantin Knizhnik
On 11.01.2022 03:06, Bossart, Nathan wrote: I noticed this thread and thought I'd share my experiences building something similar for Multi-AZ DB clusters [0]. It's not a strict RPO mechanism, but it does throttle backends in an effort to keep the replay lag below a configured maximum. I can

Re: ICU for global collation

2022-01-11 Thread Peter Eisentraut
On 10.01.22 07:00, Julien Rouhaud wrote: And then I changed in varstr_cmp(): if (collid != DEFAULT_COLLATION_OID) mylocale = pg_newlocale_from_collation(collid); to just mylocale = pg_newlocale_from_collation(collid); I find that the \timing results are indist

Re: In-placre persistance change of a relation

2022-01-11 Thread Jakub Wartak
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested I've retested v15 of the patch with everything that came to my mi

Re: logical replication empty transactions

2022-01-11 Thread Ajin Cherian
On Wed, Sep 1, 2021 at 8:57 PM Ajin Cherian wrote: > > Thanks for the comments. Addressed them in the attached patch. > > regards, > Ajin Cherian > Fujitsu Australia Minor update to rebase the patch so that it applies clean on HEAD. regards, Ajin Cherian regards, Ajin Cherian v16-0001-Skip-em

Re: row filtering for logical replication

2022-01-11 Thread Amit Kapila
On Tue, Jan 11, 2022 at 1:32 PM tanghy.f...@fujitsu.com wrote: > > On Tuesday, January 11, 2022 10:16 AM houzj.f...@fujitsu.com > wrote: > > > > Attach the v62 patch set which address the above comments and slightly > > adjust the commit message in 0002 patch. > > > > I saw a possible problem ab

Re: Add client connection check during the execution of the query

2022-01-11 Thread Thomas Munro
On Tue, Dec 14, 2021 at 11:50 PM Thomas Munro wrote: > On Tue, Dec 14, 2021 at 11:18 AM Thomas Munro wrote: > > Well, I was trying to avoid bikeshedding an API change just for a > > hypothetical problem we could solve when the time comes (say, after > > fixing the more egregious problems with App

Re: Proposal: More structured logging

2022-01-11 Thread Ronan Dunklau
Le mercredi 29 décembre 2021, 14:59:16 CET Justin Pryzby a écrit : > > Subject: [PATCH v3 2/3] Add test module for the new tag functionality. > > ... > > > +test_logging(PG_FUNCTION_ARGS) > > +{ > > ... > > > +(errmsg("%s", message), > > + ({ > > + forboth(lk, keys, lv, values

Re: Skipping logical replication transactions on subscriber side

2022-01-11 Thread Amit Kapila
On Tue, Jan 11, 2022 at 1:51 PM Masahiko Sawada wrote: > > On Tue, Jan 11, 2022 at 3:12 PM Amit Kapila wrote: > > > > On Tue, Jan 11, 2022 at 8:52 AM Masahiko Sawada > > wrote: > > > > > > On Mon, Jan 10, 2022 at 8:50 PM Amit Kapila > > > wrote: > > > > > > > > I was thinking what if we don't

Re: Skipping logical replication transactions on subscriber side

2022-01-11 Thread Amit Kapila
On Tue, Jan 11, 2022 at 8:52 AM Masahiko Sawada wrote: > > On Mon, Jan 10, 2022 at 8:50 PM Amit Kapila wrote: > > > > > > Few other comments on the latest patch: > > = > > 1. > > A conflict will produce an error and will stop the replication; it must be > > res

Re: ICU for global collation

2022-01-11 Thread Julien Rouhaud
On Tue, Jan 11, 2022 at 10:10:25AM +0100, Peter Eisentraut wrote: > > On 10.01.22 07:00, Julien Rouhaud wrote: > > > > So I tried to run Noah's benchmark to see if I could reproduce the slowdown. > > Unfortunately the results I'm getting don't really make sense as removing > > the > > optimisati

Re: ICU for global collation

2022-01-11 Thread Peter Eisentraut
On 07.01.22 10:03, Julien Rouhaud wrote: I changed the datcollate, datctype, and the new daticucoll fields to type text (from name). That way, the daticucoll field can be set to null if it's not applicable. Also, the limit of 63 characters can actually be a problem if you want to use some com

Re: ICU for global collation

2022-01-11 Thread Peter Eisentraut
On 10.01.22 12:49, Daniel Verite wrote: With the current patch, it's not possible, AFAICS, because the user can't tell that the collation is non-deterministic. Presumably this would require another option to CREATE DATABASE and another column to store that bit of information. Adding this would

Query regarding replication slots

2022-01-11 Thread RKN Sai Krishna
Hi All, I have a very basic question related to replication slots. Why should the master/primary server maintain the replication slot info like lsn corresponding to each standby server etc. Instead, why can't each standby server send the lsn that it needs, and master/primary server maintain the mi

Re: a misbehavior of partition row movement (?)

2022-01-11 Thread Alvaro Herrera
On 2022-Jan-11, Amit Langote wrote: > As for the fix to make cross-partition updates work correctly with > foreign keys, I just realized it won't work for the users' existing > foreign keys, because the parent table's triggers that are needed for > the fix to work would not be present. Were you t

Re: [PATCH] Allow multiple recursive self-references

2022-01-11 Thread Denis Hirn
> I have been studying this a bit more. I don't understand your argument here. > Why would this query have different semantics than, say > > WITH RECURSIVE t(n) AS ( > VALUES (1) > UNION ALL > VALUES (2) > UNION ALL > SELECT n+1 FROM t WHERE n < 100 > ) SELECT * FROM t LIMI

Re: ICU for global collation

2022-01-11 Thread Daniel Verite
Julien Rouhaud wrote: > > I guess there's still the possibility of requiring that the ICU db-wide > > collation of the new database does exist in the template database, > > and then the CREATE DATABASE would refer to that collation instead of > > an independent locale string. > > That cou

Re: Query regarding replication slots

2022-01-11 Thread Julien Rouhaud
On Tue, Jan 11, 2022 at 04:48:59PM +0530, RKN Sai Krishna wrote: > Hi All, > > I have a very basic question related to replication slots. Why should > the master/primary server maintain the replication slot info like lsn > corresponding to each standby server etc. Instead, why can't each > standby

Re: ICU for global collation

2022-01-11 Thread Julien Rouhaud
On Tue, Jan 11, 2022 at 12:36:46PM +0100, Daniel Verite wrote: > > If CREATE DATABASE referred to a collation in the template db, > either that collation already exists, or the user would have to add it > to the template db with CREATE COLLATION. > initdb already populates the template databases w

[PATCH]Add tab completion for foreigh table

2022-01-11 Thread tanghy.f...@fujitsu.com
Hi Attached a patch to improve the tab completion for foreigh table. Also modified some DOC description of ALTER TABLE at [1] in according with CREATE INDEX at [2]. In [1], we use "ALTER INDEX ATTACH PARTITION" In [2], we use "ALTER INDEX ... ATTACH PARTITION" I think the format in [2] is bett

Re: Time to drop plpython2?

2022-01-11 Thread Peter Eisentraut
On 15.11.21 19:52, Peter Eisentraut wrote: I think we should just write to the build farm owners, we plan to drop python2 support in, say, 60 days, please update your setup to use python3 or disable python support. This discussion stalled. I think we should do *something* for PostgreSQL 15.

Re: [Ext:] Re: Stream Replication not working

2022-01-11 Thread Amit Kapila
On Tue, Jan 11, 2022 at 2:12 AM Allie Crawford wrote: > > Thank you so much for your help on this Satya. I have detailed right below > the output of the query you asked me to run. > > > > Master > > postgresql@ ~>psql > > psql (13.5) > > Type "help" for help. > > > > postgresql=# select * from pg

Boyer-More-Horspool searching LIKE queries

2022-01-11 Thread Atsushi Ogawa
I have created a patch to enable the Boyer-More-Horspool search algorithm (B-M-H) for LIKE queries. B-M-H needs to initialize the skip table and keep it during SQL execution. In this patch, flinfo->fn_extra is used to keep the skip table. The conditions under which B-M-H can be used are as follow

Re: enhance pg_log_backend_memory_contexts() to log memory contexts of auxiliary processes

2022-01-11 Thread Fujii Masao
On 2022/01/08 1:50, Bharath Rupireddy wrote: PSA v7 patch. Thanks for updating the patch! I applied some cosmetic changes and pushed the patch. Thanks! Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION

Re: Logical replication timeout problem

2022-01-11 Thread Fabrice Chapuis
Can you explain why you think this will help in solving your current problem? Indeed your are right this function won't help, we have to look elsewhere. It is still not clear to me why the problem happened? IIUC, after restoring 4096 changes from snap files, we send them to the subscriber, and th

Re: [Ext:] Re: Stream Replication not working

2022-01-11 Thread Allie Crawford
Satya, I am a newbie on postgresql, I have no previous experience with postgresql and I need to get this replication working. In looking at the data that the pg_lock is showing, I do not know how to interpret it. I will really appreciate any help you can give me in resolving this issue. Regards,

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-01-11 Thread Alexander Lakhin
10.01.2022 23:52, Thomas Munro wrote: >> I'm yet to find out whether the other >> WaitLatchOrSocket' users (e. g. postgres_fdw) can suffer from the >> disconnected socket state, but this approach definitely works for >> walreceiver. > I see where you're going: there might be safe call sequences an

Re: [Ext:] Re: Stream Replication not working

2022-01-11 Thread Allie Crawford
Amit, Thank you for your help in trying to understand the information that the pg_locks table is showing. Regarding your question, I am not sure who to answer it. How do I figure out which database and relation is db:16384 and relation:12141.? Thanks, Allie From: Amit Kapila Date: Tuesday, Jan

Re: make tuplestore helper function

2022-01-11 Thread Melanie Plageman
On Wed, Jan 5, 2022 at 7:57 PM Justin Pryzby wrote: > > On Wed, Jan 05, 2022 at 12:09:16PM -0500, Melanie Plageman wrote: > > On Fri, Dec 17, 2021 at 3:04 PM Justin Pryzby wrote: > > > There's a couples places that you're checking expectedDesc where it wasn't > > > being checked before. Is that

Re: sepgsql logging

2022-01-11 Thread Dave Page
Hi On Tue, Jan 11, 2022 at 12:04 AM Jacob Champion wrote: > On Wed, Apr 14, 2021 at 8:42 AM Dave Page wrote: > > Attached is a patch to clean this up. It will log denials as such > > regardless of whether or not either selinux or sepgsql is in > > permissive mode. When either is in permissive m

Re: Time to drop plpython2?

2022-01-11 Thread Tom Lane
Peter Eisentraut writes: > On 15.11.21 19:52, Peter Eisentraut wrote: >> I think we should just write to the build farm owners, we plan to drop >> python2 support in, say, 60 days, please update your setup to use >> python3 or disable python support. > This discussion stalled. I think we shoul

Re: sequences vs. synchronous replication

2022-01-11 Thread Tomas Vondra
On 12/22/21 18:50, Fujii Masao wrote: > > > On 2021/12/22 21:11, Tomas Vondra wrote: >> Interesting idea, but I think it has a couple of issues :-( > > Thanks for the review! > >> 1) We'd need to know the LSN of the last WAL record for any given >> sequence, and we'd need to communicate that

Re: generic plans and "initial" pruning

2022-01-11 Thread Robert Haas
On Fri, Dec 24, 2021 at 10:36 PM Amit Langote wrote: > However, using an idea that Robert suggested to me off-list a little > while back, it seems possible to determine the set of partitions that > we can safely skip locking. The idea is to look at the "initial" or > "pre-execution" pruning instr

Re: sepgsql logging

2022-01-11 Thread Andrew Dunstan
On 1/11/22 10:40, Dave Page wrote: > > > On Wed, 2021-04-14 at 09:49 -0400, Robert Haas wrote: > > Looks superficially reasonable on first glance, but I think we > should > > try to get an opinion from someone who knows more about SELinux. > > I am not that someone, but this l

Re: Windows crash / abort handling

2022-01-11 Thread Andrew Dunstan
On 1/11/22 02:51, Andres Freund wrote: > Hi, > > On 2022-01-10 10:57:00 -0500, Andrew Dunstan wrote: >> On 10/5/21 15:30, Andres Freund wrote >>> The above ends up dumping all crashes into a single file, but that can >>> probably be improved. But cdb is so gnarly that I wanted to stop looking >>

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-11 Thread Melanie Plageman
On Mon, Jan 10, 2022 at 5:50 PM Melanie Plageman wrote: > > I have attached a v3 which includes two commits -- one of which > implements the directmgr API and uses it and the other which adds > functionality to use either directmgr or bufmgr API during index build. > > Also registering for march c

Re: sepgsql logging

2022-01-11 Thread Tom Lane
Andrew Dunstan writes: > I am not that person either. I agree this looks reasonable, but I also > would like the opinion of an expert, if we have one. I'm not sure we do anymore. Anyway, I tried this on Fedora 35 and confirmed that it compiles and the (very tedious) test process described in the

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2022-01-11 Thread John Naylor
On Tue, Dec 7, 2021 at 10:51 PM Bossart, Nathan wrote: > > On 12/7/21, 5:21 PM, "Bharath Rupireddy" > wrote: > > On Wed, Dec 8, 2021 at 4:17 AM Bossart, Nathan wrote: > >> I agree with Tom. I would just s/server/backend/ (as per the > >> attached) and call it a day. > > > > Thanks. v5 patch lo

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 10:06 AM, "John Naylor" wrote: > I pushed this with one small change -- I felt the comment didn't need > to explain the warning message, since it now simply matches the coding > more exactly. Also, v5 was a big enough change from v4 that I put > Nathan as the first author. Thanks! Na

Re: fix crash with Python 3.11

2022-01-11 Thread Jacob Champion
On Wed, 2021-12-22 at 09:24 +0100, Peter Eisentraut wrote: > The fix is that we need to catch the PostgreSQL error and turn it into a > Python exception, like we do for other places where plpy.* methods call > into PostgreSQL internals. Tested on Ubuntu 20.04, with 3.11.0a3 (built by hand) and 3

More data files / forks

2022-01-11 Thread Chris Cleveland
I'm working on a table access method that stores indexes in a structure that looks like an LSM tree. Changes get written to small segment files, which then get merged into larger segment files. It's really tough to manage these files using existing fork/buffer/page files, because when you delete a

Re: sequences vs. synchronous replication

2022-01-11 Thread Tomas Vondra
On 12/24/21 11:40, Tomas Vondra wrote: > > ... > > FWIW I plan to explore the idea of looking at sequence page LSN, and > flushing up to that position. > So, I explored the page LSN idea, and it seems to be working pretty nicely. There still is some impact on the workload doing just nextval cal

Re: Column Filtering in Logical Replication

2022-01-11 Thread Alvaro Herrera
On 2022-Jan-10, Alvaro Herrera wrote: > Hmm. So you're saying that we should only raise errors about the column > list if we are publishing UPDATE or DELETE, but otherwise let the > replica identity be anything. OK, I'll see if I can come up with a > reasonable set of rules ... This is an attem

Re: [PATCH] Add reloption for views to enable RLS

2022-01-11 Thread Laurenz Albe
On Fri, 2021-12-17 at 18:31 +0100, Christoph Heiss wrote: > As part of a customer project we are looking to implement an reloption > for views which when set, runs the subquery as invoked by the user > rather than the view owner, as is currently the case. > The rewrite rule's table references are

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-11 Thread Bossart, Nathan
On 1/10/22, 5:01 PM, "Imseih (AWS), Sami" wrote: > I have attached the 3rd revision of the patch which also includes the > documentation changes. Also attached is a rendered html of the docs for > review. > > "max_index_vacuum_cycle_time" has been removed. > "index_rows_vacuumed" renamed to "ind

Re: improve CREATE EXTENSION error message

2022-01-11 Thread Tom Lane
"Bossart, Nathan" writes: > Okay, the message looks like this in v5: > postgres=# CREATE EXTENSION does_not_exist; > ERROR: extension "does_not_exist" is not available > DETAIL: Could not open extension control file > "/usr/local/pgsql/share/extension/does_not_exist.con

Re: improve CREATE EXTENSION error message

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 11:23 AM, "Tom Lane" wrote: > "Bossart, Nathan" writes: >> Okay, the message looks like this in v5: > >> postgres=# CREATE EXTENSION does_not_exist; >> ERROR: extension "does_not_exist" is not available >> DETAIL: Could not open extension control file >> "/u

Re: pg_upgrade should truncate/remove its logs before running

2022-01-11 Thread Justin Pryzby
On Tue, Jan 11, 2022 at 04:41:58PM +0900, Michael Paquier wrote: > On Sat, Jan 08, 2022 at 12:48:57PM -0600, Justin Pryzby wrote: > > I fixed it by calling get_restricted_token() before parseCommandLine(). > > There's precedent for that in pg_regress (but the 3 other callers do it > > differently).

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-01-11 Thread Thomas Munro
On Wed, Jan 12, 2022 at 4:00 AM Alexander Lakhin wrote: > So here we get similar hanging on WaitLatchOrSocket(). > Just to make sure that it's indeed the same issue, I've removed socket > shutdown&close and the test executed to the end (several times). Argh. Ouch. I think our options at this poi

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-01-11 Thread Tom Lane
Thomas Munro writes: > Ouch. I think our options at this point are: > 1. Revert 6051857fc (and put it back when we have a working > long-lived WES as I showed). This is not very satisfying, now that we > understand the bug, because even without that change I guess you must > be able to reach th

Re: Boyer-More-Horspool searching LIKE queries

2022-01-11 Thread Heikki Linnakangas
On 11/01/2022 15:55, Atsushi Ogawa wrote: I have created a patch to enable the Boyer-More-Horspool search algorithm (B-M-H) for LIKE queries. Cool! The conditions under which B-M-H can be used are as follows. (1) B-M-H in LIKE search supports only single-byte character sets and UTF8. Multiby

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-11 Thread Imseih (AWS), Sami
On 1/11/22, 1:01 PM, "Bossart, Nathan" wrote: On 1/10/22, 5:01 PM, "Imseih (AWS), Sami" wrote: > I have attached the 3rd revision of the patch which also includes the documentation changes. Also attached is a rendered html of the docs for review. > > "max_index_vacuum_cycle_time

Re: Add jsonlog log_destination for JSON server logs

2022-01-11 Thread Bossart, Nathan
On 1/10/22, 4:51 AM, "Michael Paquier" wrote: > The issue comes from an incorrect change in syslogger_parseArgs() > where I missed that the incrementation of argv by 3 has no need to be > changed. A build with -DEXEC_BACKEND is enough to show the failure, > which caused a crash when starting up t

Re: Use -fvisibility=hidden for shared libraries

2022-01-11 Thread Tom Lane
Andres Freund writes: > What is bugging me is that I am fairly sure that my local compilers at some > point complained about such mismatches on linux as well. But I can't reproduce > that right now :/ > Now I wonder if I just saw it when cross compiling locally... I still don't understand what a

Re: Column Filtering in Logical Replication

2022-01-11 Thread Justin Pryzby
Is there any coordination between the "column filter" patch and the "row filter" patch ? Are they both on track for PG15 ? Has anybody run them together ? Whichever patch is merged 2nd should include tests involving a subset of columns along with a WHERE clause. I have a suggestion: for the fun

Re: Windows crash / abort handling

2022-01-11 Thread Andres Freund
Hi, On 2022-01-11 12:01:42 -0500, Andrew Dunstan wrote: > On 1/11/22 02:51, Andres Freund wrote: > > It'd be a bit of a fight with cdb's awfully documented and quirky > > scripting [1], but the best solution would probably be to just use an > > environment variable from the target process to deter

Re: row filtering for logical replication

2022-01-11 Thread Alvaro Herrera
I just looked at 0002 because of Justin Pryzby's comment in the column filtering thread, and realized that the pgoutput row filtering has a very strange API, which receives both heap tuples and slots; and we seem to convert to and from slots in seemingly unprincipled ways. I don't think this is go

Re: Parallel Full Hash Join

2022-01-11 Thread Melanie Plageman
On Fri, Nov 26, 2021 at 3:11 PM Thomas Munro wrote: > > On Sun, Nov 21, 2021 at 4:48 PM Justin Pryzby wrote: > > On Wed, Nov 17, 2021 at 01:45:06PM -0500, Melanie Plageman wrote: > > > Yes, this looks like that issue. > > > > > > I've attached a v8 set with the fix I suggested in [1] included. >

Re: Windows crash / abort handling

2022-01-11 Thread Andrew Dunstan
On 1/11/22 16:13, Andres Freund wrote: > Hi, > > On 2022-01-11 12:01:42 -0500, Andrew Dunstan wrote: >> On 1/11/22 02:51, Andres Freund wrote: >>> It'd be a bit of a fight with cdb's awfully documented and quirky >>> scripting [1], but the best solution would probably be to just use an >>> enviro

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 12:33 PM, "Imseih (AWS), Sami" wrote: > What about something like "The number of indexes that are eligible for > vacuuming". > This covers the cases where either an individual index is skipped or the > entire "index vacuuming" phase is skipped. Hm. I don't know if "eligible" is th

Re: [EXTERNAL] Re: PQcancel does not use tcp_user_timeout, connect_timeout and keepalive settings

2022-01-11 Thread Tom Lane
Jelte Fennema writes: > Attached are 3 patches that address the feedback from Andres about code > duplication > and splitting up commits. I completely removed internal_cancel now, since it > only had > one caller at this point. Here's some cleaned-up versions of 0001 and 0002. I have not bot

Re: Skipping logical replication transactions on subscriber side

2022-01-11 Thread Masahiko Sawada
On Tue, Jan 11, 2022 at 7:08 PM Amit Kapila wrote: > > On Tue, Jan 11, 2022 at 1:51 PM Masahiko Sawada wrote: > > > > On Tue, Jan 11, 2022 at 3:12 PM Amit Kapila wrote: > > > > > > On Tue, Jan 11, 2022 at 8:52 AM Masahiko Sawada > > > wrote: > > > > > > > > On Mon, Jan 10, 2022 at 8:50 PM Amit

Re: Skipping logical replication transactions on subscriber side

2022-01-11 Thread Masahiko Sawada
On Tue, Jan 11, 2022 at 7:11 PM Amit Kapila wrote: > > On Tue, Jan 11, 2022 at 8:52 AM Masahiko Sawada wrote: > > > > On Mon, Jan 10, 2022 at 8:50 PM Amit Kapila wrote: > > > > > > > > > Few other comments on the latest patch: > > > = > > > 1. > > > A conflict wil

Re: More data files / forks

2022-01-11 Thread Tomas Vondra
On 1/11/22 19:39, Chris Cleveland wrote: > I'm working on a table access method that stores indexes in a structure > that looks like an LSM tree. Changes get written to small segment files, > which then get merged into larger segment files. > > It's really tough to manage these files using exist

Re: [EXTERNAL] Re: PQcancel does not use tcp_user_timeout, connect_timeout and keepalive settings

2022-01-11 Thread Tom Lane
I wrote: > (The fact that we now have test coverage for PQcancel makes me a lot > more willing to try this than I might otherwise be. Will be > interested to see the cfbot's results.) On closer look, I'm not sure that psql/t/020_cancel.pl is actually doing anything on Windows; the cfbot's test tr

Re: Column Filtering in Logical Replication

2022-01-11 Thread Alvaro Herrera
On 2022-Jan-11, Alvaro Herrera wrote: > On 2022-Jan-10, Alvaro Herrera wrote: > > > Hmm. So you're saying that we should only raise errors about the column > > list if we are publishing UPDATE or DELETE, but otherwise let the > > replica identity be anything. OK, I'll see if I can come up with

Re: do only critical work during single-user vacuum?

2022-01-11 Thread John Naylor
On Tue, Dec 21, 2021 at 4:56 PM Peter Geoghegan wrote: > But if we're going to add a new option to the VACUUM command (or > something of similar scope), then we might as well add a new behavior > that is reasonably exact -- something that (say) only *starts* a > VACUUM for those tables whose relf

Re: [Ext:] Re: Stream Replication not working

2022-01-11 Thread Kyotaro Horiguchi
Hi. At Tue, 11 Jan 2022 15:05:55 +, Allie Crawford wrote in > er it. How do I figure out which database and relation is db:16384 > and relation:12141.? On any database, select datname from pg_database where oid = 16384; Then on the shown database, select relname from pg_class where oid

Re: Can there ever be out of sequence WAL files?

2022-01-11 Thread Bharath Rupireddy
On Tue, Dec 28, 2021 at 7:45 AM Bharath Rupireddy wrote: > > Hi, > > Can the postgres server ever have/generate out of sequence WAL files? > For instance, 0001020C00A2, 0001020C00A3, > 0001020C00A5 and so on, missing 0001020C00A4. > Manual/Accidental

Re: do only critical work during single-user vacuum?

2022-01-11 Thread Peter Geoghegan
On Tue, Jan 11, 2022 at 4:59 PM John Naylor wrote: > I've attached a PoC *untested* patch to show what it would look like > as a top-level statement. If the "shape" is uncontroversial, I'll put > work into testing it and fleshing it out. Great! > For the PoC I wanted to try re-using existing key

Re: Can there ever be out of sequence WAL files?

2022-01-11 Thread Julien Rouhaud
On Wed, Jan 12, 2022 at 07:19:48AM +0530, Bharath Rupireddy wrote: > > > > Can the postgres server ever have/generate out of sequence WAL files? > > For instance, 0001020C00A2, 0001020C00A3, > > 0001020C00A5 and so on, missing 0001020C00A4. > > Manual

Re: do only critical work during single-user vacuum?

2022-01-11 Thread Justin Pryzby
On Tue, Jan 11, 2022 at 07:58:56PM -0500, John Naylor wrote: > + // FIXME: also check reloption > + // WIP: 95% is a starting point for discussion > + if ((table_xid_age < autovacuum_freeze_max_age * 0.95) || > + (table_mxid_age < autovacuum_m

Re: Skipping logical replication transactions on subscriber side

2022-01-11 Thread Amit Kapila
On Wed, Jan 12, 2022 at 5:49 AM Masahiko Sawada wrote: > > On Tue, Jan 11, 2022 at 7:08 PM Amit Kapila wrote: > > > > On Tue, Jan 11, 2022 at 1:51 PM Masahiko Sawada > > wrote: > > > > > > On second thought, the same is true for other cases, for example, > > > preparing the transaction and clea

Re: Improve error handling of HMAC computations and SCRAM

2022-01-11 Thread Michael Paquier
On Tue, Jan 11, 2022 at 11:08:59AM +0300, Sergey Shinderuk wrote: > Yeah, that's better. I thought "providing errors about an error" was a > typo, but now I see the same comment was committed in b69aba745. Is it just > me? :) It is not only you :) I have applied a fix to fix the comments on HEA

Re: pg_upgrade should truncate/remove its logs before running

2022-01-11 Thread Michael Paquier
On Tue, Jan 11, 2022 at 02:03:07PM -0600, Justin Pryzby wrote: > I added mkdir() before the other stuff that messes with logfiles, because it > needs to happen before that. > > Are you suggesting to change the pre-existing behavior of when logfiles are > created, like 0002 ? Yes, something like t

Re: pg_upgrade should truncate/remove its logs before running

2022-01-11 Thread Justin Pryzby
On Wed, Jan 12, 2022 at 12:59:54PM +0900, Michael Paquier wrote: > On Tue, Jan 11, 2022 at 02:03:07PM -0600, Justin Pryzby wrote: > > There's no reason not to. We created the dir, and the user didn't specify > > to > > preserve it. It'd be their fault if they put something valuable there after >

Re: pg_dump/restore --no-tableam

2022-01-11 Thread Justin Pryzby
On Tue, Jan 11, 2022 at 04:50:23PM +0900, Michael Paquier wrote: > On Mon, Jan 03, 2022 at 03:44:24PM -0600, Justin Pryzby wrote: > > + > > + > > + --no-table-am > > + > > + > > +Do not output commands to select table access methods. > > +With this option

Re: Can there ever be out of sequence WAL files?

2022-01-11 Thread Michael Paquier
On Wed, Jan 12, 2022 at 10:18:11AM +0800, Julien Rouhaud wrote: > On Wed, Jan 12, 2022 at 07:19:48AM +0530, Bharath Rupireddy wrote: >>> Can the postgres server ever have/generate out of sequence WAL files? >>> For instance, 0001020C00A2, 0001020C00A3, >>> 0001020C00

Re: [EXTERNAL] Re: PQcancel does not use tcp_user_timeout, connect_timeout and keepalive settings

2022-01-11 Thread Tom Lane
... btw, speaking of signal-safe functions: I am dismayed to notice that strerror (and strerror_r) are *not* in POSIX's list of async-signal-safe functions. This is really quite unsurprising, considering that they are chartered to return locale-dependent strings. Unless the data has already been

Re: Can there ever be out of sequence WAL files?

2022-01-11 Thread Julien Rouhaud
On Wed, Jan 12, 2022 at 01:10:25PM +0900, Michael Paquier wrote: > > xlog.c can be a good read to check the assumptions WAL replay relies > on, with things like CheckRecoveryConsistency() or > reachedConsistency. That should only stand for a WAL expected to be missing right? For something unexpe

Re: pg_upgrade parallelism

2022-01-11 Thread Jaime Casanova
On Wed, Nov 17, 2021 at 08:04:41PM +, Jacob Champion wrote: > On Wed, 2021-11-17 at 14:44 -0500, Jaime Casanova wrote: > > I'm trying to add more parallelism by copying individual segments > > of a relfilenode in different processes. Does anyone one see a big > > problem in trying to do that? I

Re: row filtering for logical replication

2022-01-11 Thread Peter Smith
Here are my review comments for v62-0001 ~~~ 1. src/backend/catalog/pg_publication.c - GetTopMostAncestorInPublication @@ -276,17 +276,46 @@ GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt, } /* + * Check if any of the ancestors are published in the publication. If

2022-01 Commitfest

2022-01-11 Thread Julien Rouhaud
Hi, The January commitfest should have started almost two weeks ago, but given that nothing happened until now I think that it's safe to assume that either everyone forgot or no one wanted to volunteer. I'm therfore volunteering to manage this commitfest, although since it's already quite late it

Re: [PATCH] Full support for index LP_DEAD hint bits on standby

2022-01-11 Thread Julien Rouhaud
Hi, On Wed, Nov 10, 2021 at 3:06 AM Michail Nikolaev wrote: > > > Attached is a proposal for a minor addition that would make sense to me, add > > it if you think it's appropriate. > > Added. Also, I updated the documentation a little. > > > I have changed approach, so it is better to start from

Re: ResourceOwner refactoring

2022-01-11 Thread Julien Rouhaud
Hi, On Fri, Nov 26, 2021 at 8:41 PM Aleksander Alekseev wrote: > > > I will submit the actual code review in the follow-up email > > The patchset is in a good shape. I'm changing the status to "Ready for > Committer". The 2nd patch doesn't apply anymore due to a conflict on resowner_private.h: h

Re: Skipping logical replication transactions on subscriber side

2022-01-11 Thread Masahiko Sawada
On Wed, Jan 12, 2022 at 12:21 PM Amit Kapila wrote: > > On Wed, Jan 12, 2022 at 5:49 AM Masahiko Sawada wrote: > > > > On Tue, Jan 11, 2022 at 7:08 PM Amit Kapila wrote: > > > > > > On Tue, Jan 11, 2022 at 1:51 PM Masahiko Sawada > > > wrote: > > > > > > > > On second thought, the same is true

Re: Skipping logical replication transactions on subscriber side

2022-01-11 Thread Masahiko Sawada
On Mon, Jan 10, 2022 at 6:27 PM vignesh C wrote: > > On Fri, Jan 7, 2022 at 11:23 AM Masahiko Sawada wrote: > > > > On Fri, Jan 7, 2022 at 10:04 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Jan 5, 2022 at 12:31 PM Amit Kapila > > > wrote: > > > > > > > > On Mon, Dec 27, 2021 at 9:54 AM M

Re: Add checkpoint and redo LSN to LogCheckpointEnd log message

2022-01-11 Thread Julien Rouhaud
Hi, On Tue, Dec 28, 2021 at 10:56 AM Bharath Rupireddy wrote: > > attaching v1-0001-XXX from the initial mail again just for the sake of > completion: Unfortunately this breaks the cfbot as it tries to apply this patch too: http://cfbot.cputube.org/patch_36_3474.log. For this kind of situation

Re: support for MERGE

2022-01-11 Thread Jaime Casanova
On Wed, Dec 22, 2021 at 11:35:56AM +, Simon Riggs wrote: > On Mon, 15 Nov 2021 at 22:45, Alvaro Herrera wrote: > > > > On 2021-Nov-15, Alvaro Herrera wrote: > > > > > Thanks everyone for the feedback. I attach a version with the fixes > > > that were submitted, as well as some additional chan

Re: Add connection active, idle time to pg_stat_activity

2022-01-11 Thread Julien Rouhaud
Hi, On Mon, Nov 29, 2021 at 11:04 PM Kuntal Ghosh wrote: > > You also need to update the documentation. You also need to update rules.sql: https://cirrus-ci.com/task/6145265819189248

cfbot wrangling (was Re: Add checkpoint and redo LSN to LogCheckpointEnd log message)

2022-01-11 Thread Tom Lane
Julien Rouhaud writes: > For this kind of situation I think that the usual solution is to use a > .txt extension to make sure that the cfbot won't try to apply it. Yeah ... this has come up before. Is there a documented way to attach files that the cfbot will ignore? Two specific scenarios seem

Re: Add jsonlog log_destination for JSON server logs

2022-01-11 Thread Michael Paquier
On Tue, Jan 11, 2022 at 08:34:26PM +, Bossart, Nathan wrote: > I've been looking at the latest patch set intermittently and playing > around with jsonlog a little. It seems to work well, and I don't have > any significant comments about the code. 0001 and 0002 seem > straightforward and uncon

Re: cfbot wrangling (was Re: Add checkpoint and redo LSN to LogCheckpointEnd log message)

2022-01-11 Thread Julien Rouhaud
On Wed, Jan 12, 2022 at 01:19:22AM -0500, Tom Lane wrote: > Julien Rouhaud writes: > > For this kind of situation I think that the usual solution is to use a > > .txt extension to make sure that the cfbot won't try to apply it. > > Yeah ... this has come up before. Is there a documented way to >

Re: cfbot wrangling (was Re: Add checkpoint and redo LSN to LogCheckpointEnd log message)

2022-01-11 Thread Tom Lane
Julien Rouhaud writes: > On Wed, Jan 12, 2022 at 01:19:22AM -0500, Tom Lane wrote: >> 2. You are attaching some random files, and would like to not >> displace the cfbot's idea of the latest patchset. > I'm assuming that someone wanting to send an additional patch to be applied on > top of the OP

Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

2022-01-11 Thread Julien Rouhaud
Hi, On Fri, Dec 24, 2021 at 2:06 AM Максим Орлов wrote: > > Thanks for your review! Fixed all these remaining things from patch v6. > PFA v7 patch. The cfbot reports that you have mixed declarations and code (https://cirrus-ci.com/task/6407449413419008): [17:21:26.926] pg_amcheck.c: In function

Re: do only critical work during single-user vacuum?

2022-01-11 Thread Masahiko Sawada
On Wed, Jan 12, 2022 at 10:57 AM Peter Geoghegan wrote: > > On Tue, Jan 11, 2022 at 4:59 PM John Naylor > wrote: > > I've attached a PoC *untested* patch to show what it would look like > > as a top-level statement. If the "shape" is uncontroversial, I'll put > > work into testing it and fleshing

  1   2   >