Re: WIP: WAL prefetch (another approach)

2022-03-14 Thread Julien Rouhaud
On Mon, Mar 14, 2022 at 06:15:59PM +1300, Thomas Munro wrote: > On Fri, Mar 11, 2022 at 9:27 PM Julien Rouhaud wrote: > > > > Also, is it worth an assert (likely at the top of the function) for > > > > that? > > > > > > How could I assert that EndRecPtr has the right value? > > > > Sorry, I meant

Re: Issue with pg_stat_subscription_stats

2022-03-14 Thread Masahiko Sawada
On Mon, Mar 14, 2022 at 2:05 AM Melanie Plageman wrote: > > On Sat, Mar 12, 2022 at 3:15 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-03-12 08:28:35 +0530, Amit Kapila wrote: > > > On Sat, Mar 12, 2022 at 2:14 AM Melanie Plageman > > > wrote: > > > > > > > > So, I noticed that pg_stat_res

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-14 Thread Michael Paquier
On Mon, Mar 14, 2022 at 10:54:56AM +0530, Bharath Rupireddy wrote: > Yes, this is a concern. Also, when there were no logical replication > slots on a plain server or the server removed or cleaned up all the > snapshot/mappings files, why would anyone want to have these messages > with all 0s in th

Re: BufferAlloc: don't take two simultaneous locks

2022-03-14 Thread Kyotaro Horiguchi
At Mon, 14 Mar 2022 09:15:11 +0300, Yura Sokolov wrote in > В Пн, 14/03/2022 в 14:31 +0900, Kyotaro Horiguchi пишет: > > I'd like to ask you to remove nalloced from partitions then add a > > global atomic for the same use? > > I really believe it should be global. I made it per-partition to > n

Re: simplifying foreign key/RI checks

2022-03-14 Thread Amit Langote
On Tue, Jan 18, 2022 at 3:30 PM Amit Langote wrote: > v13 is attached. I noticed that the recent 641f3dffcdf's changes to get_constraint_index() made it basically unusable for this patch's purposes. Reading in the thread that led to 641f3dffcdf why get_constraint_index() was changed the way it w

Re: BufferAlloc: don't take two simultaneous locks

2022-03-14 Thread Kyotaro Horiguchi
At Mon, 14 Mar 2022 17:12:48 +0900 (JST), Kyotaro Horiguchi wrote in > Then, I tried the same with the patch, and I am surprized to see that > the rise of the number of newly allocated elements didn't stop and > went up to 511 elements after the 100 seconds run. So I found that my > concern was

Re: On login trigger: take three

2022-03-14 Thread Daniel Gustafsson
> On 14 Mar 2022, at 00:33, Andres Freund wrote: > We already have GUCs like row_security, so it doesn't seem insane to add one > that disables login event triggers. What is the danger that you see? My fear is that GUCs like that end up as permanent fixtures in scripts etc after having been used

Re: simplifying foreign key/RI checks

2022-03-14 Thread Zhihong Yu
On Mon, Mar 14, 2022 at 1:33 AM Amit Langote wrote: > On Tue, Jan 18, 2022 at 3:30 PM Amit Langote > wrote: > > v13 is attached. > > I noticed that the recent 641f3dffcdf's changes to > get_constraint_index() made it basically unusable for this patch's > purposes. > > Reading in the thread that

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-03-14 Thread Nitin Jadhav
> > I am not suggesting > > removing the existing 'flags' field of pg_stat_progress_checkpoint > > view and adding a new field 'throttled'. The content of the 'flags' > > field remains the same. I was suggesting replacing the 'next_flags' > > field with 'throttled' field since the new request with

RE: Skipping logical replication transactions on subscriber side

2022-03-14 Thread shiy.f...@fujitsu.com
On Fri, Mar 11, 2022 4:20 PM Masahiko Sawada wrote: > > I've attached an updated version patch. This patch can be applied on > top of the latest disable_on_error patch[1]. > Thanks for your patch. Here are some comments for the v13 patch. 1. doc/src/sgml/ref/alter_subscription.sgml +

Re: Column Filtering in Logical Replication

2022-03-14 Thread Amit Kapila
On Mon, Mar 14, 2022 at 2:37 AM Tomas Vondra wrote: > > On 3/12/22 05:30, Amit Kapila wrote: > >> ... > > > > Okay, please find attached. I have done basic testing of this, if we > > agree with this approach then this will require some more testing. > > > > Thanks, the proposed changes seem like a

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-14 Thread Bharath Rupireddy
On Mon, Mar 14, 2022 at 1:33 PM Michael Paquier wrote: > > On Mon, Mar 14, 2022 at 10:54:56AM +0530, Bharath Rupireddy wrote: > > Yes, this is a concern. Also, when there were no logical replication > > slots on a plain server or the server removed or cleaned up all the > > snapshot/mappings files

Re: Implementing Incremental View Maintenance

2022-03-14 Thread Yugo NAGATA
Hello Zhihong Yu, I already replied to your comments before, but I forgot to include the list to CC, so I resend the same again. Sorry for the duplicate emails. On Thu, 3 Feb 2022 09:51:52 -0800 Zhihong Yu wrote: > For CreateIndexOnIMMV(): > > + ereport(NOTICE, > +

Re: Add 64-bit XIDs into PostgreSQL 15

2022-03-14 Thread Aleksander Alekseev
Hi hackers, > > Here is a new version of the patchset. SLRU refactoring was moved to a > > separate patch. Both v14-0003 (XID_FMT macro) and v14-0004 (SLRU > > refactoring) can be delivered in PG15. > > Here is a new version of the patchset. The changes compared to v14 are > minimal. Most importan

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-14 Thread Amit Kapila
On Thu, Mar 10, 2022 at 12:04 PM Amit Kapila wrote: > > On Wed, Mar 9, 2022 at 7:57 PM osumi.takami...@fujitsu.com > wrote: > > > > Hi, attached v32 removed my additional code for maybe_reread_subscription. > > > > Thanks, the patch looks good to me. I have made minor edits in the > attached. I a

RE: Column Filtering in Logical Replication

2022-03-14 Thread houzj.f...@fujitsu.com
On Monday, March 14, 2022 5:08 AM Tomas Vondra wrote: > > On 3/12/22 05:30, Amit Kapila wrote: > >> ... > > > > Okay, please find attached. I have done basic testing of this, if we > > agree with this approach then this will require some more testing. > > > > Thanks, the proposed changes seem l

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-14 Thread Bharath Rupireddy
On Mon, Mar 14, 2022 at 1:33 PM Michael Paquier wrote: > > On Mon, Mar 14, 2022 at 10:54:56AM +0530, Bharath Rupireddy wrote: > > Yes, this is a concern. Also, when there were no logical replication > > slots on a plain server or the server removed or cleaned up all the > > snapshot/mappings files

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-03-14 Thread Julien Rouhaud
On Mon, Mar 14, 2022 at 03:16:50PM +0530, Nitin Jadhav wrote: > > > I am not suggesting > > > removing the existing 'flags' field of pg_stat_progress_checkpoint > > > view and adding a new field 'throttled'. The content of the 'flags' > > > field remains the same. I was suggesting replacing the 'ne

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-14 Thread Dilip Kumar
On Sat, Mar 12, 2022 at 11:06 AM Dilip Kumar wrote: > > In fact while we are rewriting the relation during vacuum full that > time also we are calling log_newpage() under RelationCopyStorage() and > during standby if it gets promoted we will be having some buffers in > the buffer pool with the new

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-14 Thread osumi.takami...@fujitsu.com
On Monday, March 14, 2022 7:49 PM Amit Kapila wrote: > On Thu, Mar 10, 2022 at 12:04 PM Amit Kapila > wrote: > > > > On Wed, Mar 9, 2022 at 7:57 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > Hi, attached v32 removed my additional code for > maybe_reread_subscription. > > > > > > > Thanks

Re: BufferAlloc: don't take two simultaneous locks

2022-03-14 Thread Yura Sokolov
В Пн, 14/03/2022 в 17:12 +0900, Kyotaro Horiguchi пишет: > At Mon, 14 Mar 2022 09:15:11 +0300, Yura Sokolov > wrote in > > В Пн, 14/03/2022 в 14:31 +0900, Kyotaro Horiguchi пишет: > > > I'd like to ask you to remove nalloced from partitions then add a > > > global atomic for the same use? > > >

Re: Column Filtering in Logical Replication

2022-03-14 Thread Tomas Vondra
On 3/14/22 10:53, Amit Kapila wrote: > On Mon, Mar 14, 2022 at 2:37 AM Tomas Vondra > wrote: >> >> On 3/12/22 05:30, Amit Kapila wrote: ... >>> >>> Okay, please find attached. I have done basic testing of this, if we >>> agree with this approach then this will require some more testing. >>

Re: Column Filtering in Logical Replication

2022-03-14 Thread Tomas Vondra
On 3/14/22 12:12, houzj.f...@fujitsu.com wrote: > On Monday, March 14, 2022 5:08 AM Tomas Vondra > wrote: >> >> On 3/12/22 05:30, Amit Kapila wrote: ... >>> >>> Okay, please find attached. I have done basic testing of this, if we >>> agree with this approach then this will require some mo

Re: Tab completion not listing schema list for create/alter publication for all tables in schema

2022-03-14 Thread vignesh C
On Mon, Mar 14, 2022 at 5:23 AM Tom Lane wrote: > > vignesh C writes: > > Here "pg_%" should be "pg_%%". > > Right you are. Patch pushed, thanks! Thanks for pushing the patch. Regards, Vignesh

RE: Skipping logical replication transactions on subscriber side

2022-03-14 Thread osumi.takami...@fujitsu.com
On Friday, March 11, 2022 5:20 PM Masahiko Sawada wrote: > I've attached an updated version patch. This patch can be applied on top of > the > latest disable_on_error patch[1]. Hi, few extra comments on v13. (1) src/backend/replication/logical/worker.c With regard to clear_subscription_skip_l

Re: [PATCH] Add reloption for views to enable RLS

2022-03-14 Thread Christoph Heiss
On 3/9/22 16:06, Laurenz Albe wrote: This paragraph contains a couple of grammatical errors. How about Note that the user performing the insert, update or delete on the view must have the corresponding insert, update or delete privilege on the view. Unless security_invoker is se

Re: Column Filtering in Logical Replication

2022-03-14 Thread Amit Kapila
On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra wrote: > > On 3/14/22 12:12, houzj.f...@fujitsu.com wrote: > > On Monday, March 14, 2022 5:08 AM Tomas Vondra > > wrote: > > Anyway, the fix does not address tablesync, as explained in [1]. I'm not > sure what to do about it - in principle, we could c

Re: ICU for global collation

2022-03-14 Thread Peter Eisentraut
On 05.03.22 09:38, Julien Rouhaud wrote: I say it works because I did manually check, as far as I can see there isn't any test that ensures it. I'm using this naive scenario: DROP DATABASE IF EXISTS dbicu; CREATE DATABASE dbicu LOCALE_PROVIDER icu LOCALE 'en_US' ICU_LOCALE 'en-u-kf-upper' temp

Re: On login trigger: take three

2022-03-14 Thread Robert Haas
On Sun, Mar 13, 2022 at 7:34 PM Andres Freund wrote: > IMO the other types of event triggers make it a heck of a lot harder to get > yourself into a situation that you can't get out of... In particular, unless something has changed since I committed this stuff originally, there's no existing type

Re: refactoring basebackup.c

2022-03-14 Thread Robert Haas
On Fri, Mar 11, 2022 at 8:52 PM Andres Freund wrote: > You could also just append a manifest as a compresed tar to the compressed tar > stream. Unfortunately GNU tar requires -i to read concated compressed > archives, so perhaps that's not quite an alternative. s/Unfortunately/Fortunately/ :-p I

Re: Column Filtering in Logical Replication

2022-03-14 Thread Tomas Vondra
On 3/14/22 13:47, Amit Kapila wrote: > On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra > wrote: >> >> On 3/14/22 12:12, houzj.f...@fujitsu.com wrote: >>> On Monday, March 14, 2022 5:08 AM Tomas Vondra >>> wrote: >> >> Anyway, the fix does not address tablesync, as explained in [1]. I'm not >> su

Re: On login trigger: take three

2022-03-14 Thread Daniel Gustafsson
> On 14 Mar 2022, at 01:08, Andres Freund wrote: > I was thinking that the way to use it would be to specify it as a client > option. Like PGOPTIONS='-c ignore_event_trigger=login' psql. Attached is a quick PoC/sketch of such a patch, where 0001 adds a guc, 0002 is the previously posted v25 logi

Re: pg_stat_statements and "IN" conditions

2022-03-14 Thread Robert Haas
On Sat, Mar 12, 2022 at 9:11 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > Here is the limited version of list collapsing functionality, which > doesn't utilize eval_const_expressions and ignores most of the stuff > except ArrayExprs. Any thoughts/more suggestions? The proposed commit message

Re: support for MERGE

2022-03-14 Thread Amit Langote
On Sun, Mar 13, 2022 at 12:36 AM Alvaro Herrera wrote: > FYI I intend to get the ModifyTable split patch (0001+0003) pushed > hopefully on Tuesday or Wednesday next week, unless something really > ugly is found on it. I looked at 0001 and found a few things that could perhaps be improved. + /*

Re: role self-revocation

2022-03-14 Thread Stephen Frost
Greetings, * Mark Dilger (mark.dil...@enterprisedb.com) wrote: > > On Mar 11, 2022, at 4:56 PM, Stephen Frost wrote: > > First … I outlined a fair bit of further description in the message you > > just responded to but neglected to include in your response, which strikes > > me as odd that you’

Re: Add 64-bit XIDs into PostgreSQL 15

2022-03-14 Thread Pavel Borisov
Hi, Hackers! > Hi! Here is updated version of the patch, based on Alexander's ver16. > I'd like to add a few quick notes on what's been done in v17. Patches 0001 and 0002 that are planned to be committed to PG15 are almost unchanged with the exception of one unnecessary cast in 0002 removed. We'

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-14 Thread Stephen Frost
Greetings, * Jeff Davis (pg...@j-davis.com) wrote: > On Thu, 2022-03-10 at 15:54 -0500, Stephen Frost wrote: > > The standard is basically that all of the functions it brings are > > written to enforce the PG privilege system and you aren't able to use > > the extension to bypass those privileges.

Re: pg_stat_statements and "IN" conditions

2022-03-14 Thread Dmitry Dolgov
> On Mon, Mar 14, 2022 at 10:17:57AM -0400, Robert Haas wrote: > On Sat, Mar 12, 2022 at 9:11 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > Here is the limited version of list collapsing functionality, which > > doesn't utilize eval_const_expressions and ignores most of the stuff > > except A

Re: pg_stat_statements and "IN" conditions

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 10:57 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > Well, yeah, the commit message is somewhat clumsy in this regard. It > works almost in the way you've described, except if the list is all > constants and long enough to satisfy the threshold then *first N > elements (w

Re: pg_stat_statements and "IN" conditions

2022-03-14 Thread Dmitry Dolgov
> On Mon, Mar 14, 2022 at 11:02:16AM -0400, Robert Haas wrote: > On Mon, Mar 14, 2022 at 10:57 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > Well, yeah, the commit message is somewhat clumsy in this regard. It > > works almost in the way you've described, except if the list is all > > constan

Re: support for MERGE

2022-03-14 Thread Japin Li
On Sat, 12 Mar 2022 at 11:53, Alvaro Herrera wrote: > On 2022-Mar-09, Zhihong Yu wrote: > >> Hi, >> For v14-0002-Split-ExecUpdate-and-ExecDelete-in-reusable-piec.patch : >> >> + TupleTableSlot *insertProjectedTuple; >> >> With `insert` at the beginning of the variable name, someone may think i

Re: pg_stat_statements and "IN" conditions

2022-03-14 Thread Tom Lane
Robert Haas writes: > On Mon, Mar 14, 2022 at 10:57 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: >> I'm not sure if I follow the last point. WHERE x in (1,3) and x = >> any(array[1,3]) are two different things for sure, but in which way are >> they going to be mixed together because of this cha

Re: Estimating HugePages Requirements?

2022-03-14 Thread Magnus Hagander
On Thu, Sep 9, 2021 at 11:53 PM Bossart, Nathan wrote: > > On 9/8/21, 9:19 PM, "Michael Paquier" wrote: > > FWIW, I don't have an environment at hand these days to test properly > > 0001, so this will have to wait a bit. I really like the approach > > taken by 0002, and it is independent of the

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-14 Thread Robert Haas
On Sat, Mar 12, 2022 at 12:36 AM Dilip Kumar wrote: > So here my stand is that we need to drop database buffers and remove > pending sync requests because we are deleting underlying files and if > we do not do that in some extreme cases then there is no need to drop > the buffers or remove the pen

Re: pg_stat_statements and "IN" conditions

2022-03-14 Thread Dmitry Dolgov
> On Mon, Mar 14, 2022 at 11:23:17AM -0400, Tom Lane wrote: > Robert Haas writes: > > I do find it odd that the proposed patch doesn't cause the *entire* > list to be skipped over. That seems like extra complexity and confusion > to no benefit. That's a bit surprising for me, I haven't even thou

Re: pg_stat_statements and "IN" conditions

2022-03-14 Thread Tom Lane
Dmitry Dolgov <9erthali...@gmail.com> writes: > On Mon, Mar 14, 2022 at 11:23:17AM -0400, Tom Lane wrote: >> I do find it odd that the proposed patch doesn't cause the *entire* >> list to be skipped over. That seems like extra complexity and confusion >> to no benefit. > That's a bit surprising f

Re: Add 64-bit XIDs into PostgreSQL 15

2022-03-14 Thread Pavel Borisov
> > Hi, Hackers! > >> Hi! Here is updated version of the patch, based on Alexander's ver16. >> > I'd like to add a few quick notes on what's been done in v17. > > Patches 0001 and 0002 that are planned to be committed to PG15 are almost > unchanged with the exception of one unnecessary cast in 0002

Re: role self-revocation

2022-03-14 Thread Mark Dilger
> On Mar 14, 2022, at 7:38 AM, Stephen Frost wrote: > > So ... do you feel like that's now the case? Or were you looking for > more? I don't have any more questions at the moment. Thanks! — Mark Dilger EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: pg_stat_statements and "IN" conditions

2022-03-14 Thread Dmitry Dolgov
> On Mon, Mar 14, 2022 at 11:38:23AM -0400, Tom Lane wrote: > Dmitry Dolgov <9erthali...@gmail.com> writes: > > On Mon, Mar 14, 2022 at 11:23:17AM -0400, Tom Lane wrote: > >> I do find it odd that the proposed patch doesn't cause the *entire* > >> list to be skipped over. That seems like extra com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 7:51 AM Dilip Kumar wrote: > Other than that, I have fixed some mistakes in comments and supported > tab completion for the new options. I was looking at 0001 and 0002 again and realized that I swapped the names load_relmap_file() and read_relmap_file() from what I should

Re: On login trigger: take three

2022-03-14 Thread Andres Freund
Hi, On 2022-03-14 14:47:09 +0100, Daniel Gustafsson wrote: > > On 14 Mar 2022, at 01:08, Andres Freund wrote: > > > I was thinking that the way to use it would be to specify it as a client > > option. Like PGOPTIONS='-c ignore_event_trigger=login' psql. > > Attached is a quick PoC/sketch of suc

Re: refactoring basebackup.c

2022-03-14 Thread Dipesh Pandit
Hi, I tried to implement support for parallel ZSTD compression. The library provides an option (ZSTD_c_nbWorkers) to specify the number of compression workers. The number of parallel workers can be set as part of compression parameter and if this option is specified then the library performs paral

Re: [PATCH] Add reloption for views to enable RLS

2022-03-14 Thread Laurenz Albe
On Mon, 2022-03-14 at 13:40 +0100, Christoph Heiss wrote: > On 3/9/22 16:06, Laurenz Albe wrote: > > This paragraph contains a couple of grammatical errors. > > Replaced the two paragraphs with your suggestion, it is indeed easier to > read. > > > Also, this: > > could be written like this (intro

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-14 Thread Imseih (AWS), Sami
>I'm still unsure the current design of 0001 patch is better than other >approaches we’ve discussed. Even users who don't use parallel vacuum >are forced to allocate shared memory for index vacuum progress, with >GetMaxBackends() entries from the beginning. Also, it’s likely to >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 12:04 PM Robert Haas wrote: > On Mon, Mar 14, 2022 at 7:51 AM Dilip Kumar wrote: > > Other than that, I have fixed some mistakes in comments and supported > > tab completion for the new options. > > I was looking at 0001 and 0002 again and realized that I swapped the > nam

Re: refactoring basebackup.c (zstd workers)

2022-03-14 Thread Justin Pryzby
On Mon, Mar 14, 2022 at 09:41:35PM +0530, Dipesh Pandit wrote: > I tried to implement support for parallel ZSTD compression. The > library provides an option (ZSTD_c_nbWorkers) to specify the > number of compression workers. The number of parallel > workers can be set as part of compression paramet

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-14 Thread Dilip Kumar
On Mon, Mar 14, 2022 at 10:04 PM Robert Haas wrote: > > Regarding 0004, I can't really see a reason for this function to take > a LockRelId as a parameter rather than two separate OIDs. I also can't > entirely see why it should be called LockRelationId. Maybe > LockRelationInDatabaseById(Oid dbid,

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 12:44 PM Dilip Kumar wrote: > On Mon, Mar 14, 2022 at 10:04 PM Robert Haas wrote: > > Regarding 0004, I can't really see a reason for this function to take > > a LockRelId as a parameter rather than two separate OIDs. I also can't > > entirely see why it should be called L

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-14 Thread Matthias van de Meent
Thank you all for the feedback. Please find attached v2 of the patchset, which contains updated comments and applies the suggested changes. On Sat, 12 Mar 2022 at 02:03, Andres Freund wrote: > > Hi, > > On 2022-03-11 22:42:42 +0200, Heikki Linnakangas wrote: > > Have you been able to create a tes

Re: Allow async standbys wait for sync replication

2022-03-14 Thread Andres Freund
Hi, On 2022-03-14 11:30:02 +0900, Kyotaro Horiguchi wrote: > That is, (as my understanding) async standbys are required to allow > overwriting existing unreplayed records after reconnection. But, > putting aside how to remember that LSN, if that happens at a segment > boundary, the async replica

Re: refactoring basebackup.c (zstd workers)

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 12:35 PM Justin Pryzby wrote: > I suggest to use a syntax that's more general than that, maybe something like > > :[level=]N,parallel=N,flag,flag,... > > For example, someone may want to use zstd "long" mode or (when it's released) > rsyncable mode, or specify fine-grained

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-14 Thread Nathan Bossart
On Mon, Mar 14, 2022 at 05:03:59PM +0530, Bharath Rupireddy wrote: > On Mon, Mar 14, 2022 at 1:33 PM Michael Paquier wrote: >> On Mon, Mar 14, 2022 at 10:54:56AM +0530, Bharath Rupireddy wrote: >> > Yes, this is a concern. Also, when there were no logical replication >> > slots on a plain server o

Re: refactoring basebackup.c (zstd workers)

2022-03-14 Thread Justin Pryzby
On Mon, Mar 14, 2022 at 01:02:20PM -0400, Robert Haas wrote: > On Mon, Mar 14, 2022 at 12:35 PM Justin Pryzby wrote: > > I suggest to use a syntax that's more general than that, maybe something > > like > > > > :[level=]N,parallel=N,flag,flag,... > > > > For example, someone may want to use zstd

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-14 Thread Andres Freund
Hi A random thought I had while thinking about the size limits: We could use the low bits of the length and xl_prev to store XLR_SPECIAL_REL_UPDATE | XLR_CHECK_CONSISTENCY and give rmgrs the full 8 bit of xl_info. Which would allow us to e.g. get away from needing Heap2. Which would aestethically

Re: refactoring basebackup.c (zstd workers)

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 1:11 PM Justin Pryzby wrote: > Internally, I was thinking they'd all be handled as first-class options, with > separate struct fields and separate replication protocol options. If an > option > isn't known, it'd be rejected on the client side, rather than causing an error

Re: obsolete reference to a SubPlan field

2022-03-14 Thread Robert Haas
On Wed, Feb 2, 2022 at 3:08 AM Amit Langote wrote: > I noticed $subject while looking at something involving SubLinks and > SubPlans. It seems eab6b8b27eb removed the "plan" field from the > SubPlan node struct definition, but the following line from > expression_tree_mutator(): > >

Re: Estimating HugePages Requirements?

2022-03-14 Thread Nathan Bossart
On Mon, Mar 14, 2022 at 04:26:43PM +0100, Magnus Hagander wrote: > Nothing fixing this ended up actually getting committed, right? That > is, we still get the extra log output? Correct. > And in fact, the command documented on > https://www.postgresql.org/docs/devel/kernel-resources.html doesn't

Re: obsolete reference to a SubPlan field

2022-03-14 Thread Tom Lane
Robert Haas writes: > On Wed, Feb 2, 2022 at 3:08 AM Amit Langote wrote: >> Attached patch removes those. > Looks right to me. Tom, any comments? I'm pretty sure I left those comments alone on purpose back in 2007, and I don't find simply removing them to be an improvement. In principle, reade

Re: obsolete reference to a SubPlan field

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 1:45 PM Tom Lane wrote: > Robert Haas writes: > > On Wed, Feb 2, 2022 at 3:08 AM Amit Langote wrote: > >> Attached patch removes those. > > > Looks right to me. Tom, any comments? > > I'm pretty sure I left those comments alone on purpose back in 2007, > and I don't find

Re: Lowering the ever-growing heap->pd_lower

2022-03-14 Thread Peter Geoghegan
On Thu, Mar 10, 2022 at 5:49 AM Matthias van de Meent wrote: > I double-checked the changes, and to me it seems like that was the > only place in the code where PageGetMaxOffsetNumber was not handled > correctly. This was fixed in the latest patch (v8). > > Peter, would you have time to further re

Re: Issue with pg_stat_subscription_stats

2022-03-14 Thread Melanie Plageman
On Mon, Mar 14, 2022 at 4:02 AM Masahiko Sawada wrote: > > On Mon, Mar 14, 2022 at 2:05 AM Melanie Plageman > wrote: > > > > On Sat, Mar 12, 2022 at 3:15 PM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2022-03-12 08:28:35 +0530, Amit Kapila wrote: > > > > On Sat, Mar 12, 2022 at 2:14 AM M

Re: generic plans and "initial" pruning

2022-03-14 Thread Robert Haas
On Fri, Mar 11, 2022 at 9:35 AM Amit Langote wrote: > Attached is v5, now broken into 3 patches: > > 0001: Some refactoring of runtime pruning code > 0002: Add a plan_tree_walker > 0003: Teach AcquireExecutorLocks to skip locking pruned relations So is any other committer planning to look at this

Re: ICU for global collation

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 8:51 AM Peter Eisentraut wrote: > [ new patches ] I'm not very knowledgeable about this topic, but to me, it seems confusing to think of having both a libc collation and an ICU collation associated with a database. I have two main questions: 1. What will happen if I set t

Re: Fix uninitialized variable access (src/backend/utils/mmgr/freepage.c)

2022-03-14 Thread Robert Haas
On Thu, Jan 27, 2022 at 7:33 AM Greg Nancarrow wrote: > Why not, at least, just add "Assert(result.page != NULL);" after the > "Assert(!result.found);" in FreePageManagerPutInternal()? > The following code block in FreePageBtreeSearch() - which lacks those > initializations - should never be invok

Re: Fix uninitialized variable access (src/backend/utils/mmgr/freepage.c)

2022-03-14 Thread Peter Geoghegan
On Mon, Mar 14, 2022 at 12:15 PM Robert Haas wrote: > If surgery does happen in the future, I would argue that this > change could easily *mask* bugs, because if somebody tries to apply > valgrind to this code the useless initializations will just cover up > what valgrind would otherwise detect as

Re: generic plans and "initial" pruning

2022-03-14 Thread Tom Lane
Robert Haas writes: > In my opinion, the most important theoretical issue here is around > reuse of plans that are no longer entirely valid, but the parts that > are no longer valid are certain to be pruned. If, because we know that > some parameter has some particular value, we skip locking a bun

Re: generic plans and "initial" pruning

2022-03-14 Thread Robert Haas
On Mon, Mar 14, 2022 at 3:38 PM Tom Lane wrote: > ... like EXPLAIN, for example? Exactly! I think that's the foremost example, but extension modules like auto_explain or even third-party extensions are also a risk. I think there was some discussion of this previously. > If "pruning" means physic

Re: Query about time zone patterns in to_char

2022-03-14 Thread Robert Haas
On Fri, Jul 9, 2021 at 10:44 AM Tomas Vondra wrote: > Yeah, does not seem to be worth it, as there seem to be no actual > reports of issues in the field. > > FWIW there seem to be quite a bit of other to_char differences compared > to Oracle (judging by docs and playing with sqlfiddle). But the pa

Re: [PATCH] Disable bgworkers during servers start in pg_upgrade

2022-03-14 Thread Robert Haas
On Fri, Jan 28, 2022 at 9:51 PM Julien Rouhaud wrote: > On Fri, Jan 28, 2022 at 10:20:07AM -0800, Andres Freund wrote: > > On 2022-01-28 21:56:36 +0800, Julien Rouhaud wrote: > > > I think having a new option for vacuumdb is the right move. > > > > Can't we pass the option via the connection strin

Re: document the need to analyze partitioned tables

2022-03-14 Thread Robert Haas
On Fri, Jan 21, 2022 at 1:31 PM Tomas Vondra wrote: > [ new patch ] This patch is originally by Justin. The latest version is by Tomas. I think the next step is for Justin to say whether he's OK with the latest version that Tomas posted. If he is, then I suggest that he also mark it Ready for Com

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-14 Thread Robert Haas
On Mon, Mar 7, 2022 at 3:39 AM Kyotaro Horiguchi wrote: > So the new framework has been dropped in this version. > The second test is removed as it is irrelevant to this bug. > > In this version the patch is a single file that contains the test. The status of this patch in the CommitFest was set

Re: Logical replication timeout problem

2022-03-14 Thread Björn Harrtell
Hi, I have been following this discussion for a while because I believe we are hit by this pretty hard. This sounds very reasonable to me: "Why don't we check both the count and the time? That is, I think we can send a keep-alive either if we skipped 1 changes or if we didn't sent anything fo

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-14 Thread Nathan Bossart
My compiler is worried that syncslotname may be used uninitialized in start_table_sync(). The attached patch seems to silence this warning. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 05e4e03af5afa1658ede8d78b31c1c999b5c7deb Mon Sep 17 00:00:00 2001 From: Nathan Bossart

Re: [PATCH] Fix various spelling errors

2022-03-14 Thread Justin Pryzby
On Mon, Mar 14, 2022 at 11:03:50PM +, Kekalainen, Otto wrote: > Hello! > > I propose the attached patch to be applied on the 'master' branch of > PostgreSQL > to fix various spelling errors. > > Most fixes are in comments and have no effect on functionality. Some fixes are > also in variabl

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-14 Thread Thomas Munro
On Tue, Mar 8, 2022 at 4:01 PM Kyotaro Horiguchi wrote: > At Tue, 8 Mar 2022 10:28:46 +0900, Michael Paquier > wrote in > > On Tue, Mar 08, 2022 at 10:06:50AM +0900, Kyotaro Horiguchi wrote: > > > At Tue, 8 Mar 2022 10:39:06 +1300, Thomas Munro > > > wrote in > > >> Thanks, you're right. Test

RE: Handle infinite recursion in logical replication setup

2022-03-14 Thread kuroda.hay...@fujitsu.com
Dear Vignesh, Thank you for updating your patch! > Let's consider an existing Multi master logical replication setup > between Node1 and Node2 that is created using the following steps: > a) Node1 - Publication publishing employee table - pub1 > b) Node2 - Subscription subscribing from publicatio

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-14 Thread Thomas Munro
On Tue, Mar 15, 2022 at 2:33 PM Thomas Munro wrote: > As for the complaint about pg_tablespace_location() failing, would it > be better to return an empty string? That's what was passed in as > LOCATION. Something like the attached. (Hrrmm, the contract for pgwin32_is_junction() is a little wei

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-14 Thread Michael Paquier
On Tue, Mar 15, 2022 at 02:33:17PM +1300, Thomas Munro wrote: > Ok, I pushed the fix for pg_basebackup. > > As for the complaint about pg_tablespace_location() failing, would it > be better to return an empty string? That's what was passed in as > LOCATION. Something like the attached. Hmm, I d

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-14 Thread Bharath Rupireddy
On Mon, Mar 14, 2022 at 8:25 PM Stephen Frost wrote: > > > As this patch is currently written, pg_monitor has access these > > functions, though I don't think that's the right privilege level at > > least for pg_get_raw_wal_record(). > > Yeah, I agree that pg_monitor isn't the right thing for such

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2022-03-14 Thread Michael Paquier
On Mon, Mar 14, 2022 at 01:53:54PM +0900, Michael Paquier wrote: > +select * from pg_ls_logicalmapdir() limit 0; > +select * from pg_ls_logicalsnapdir() limit 0; > +select * from pg_ls_replslotdir('') limit 0; > +select * from pg_ls_tmpdir() limit 0; > +select * from pg_ls_waldir() limit 0; > +sele

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-14 Thread osumi.takami...@fujitsu.com
On Tuesday, March 15, 2022 8:04 AM Nathan Bossart wrote: > My compiler is worried that syncslotname may be used uninitialized in > start_table_sync(). The attached patch seems to silence this warning. Thank you for your reporting ! Your fix looks good to me. Best Regards, Takamichi Os

Re: Add 64-bit XIDs into PostgreSQL 15

2022-03-14 Thread Kyotaro Horiguchi
At Mon, 14 Mar 2022 19:43:40 +0400, Pavel Borisov wrote in > > I'd like to add a few quick notes on what's been done in v17. I have some commens by a quick look-through. Apologize in advance for wrong comments from the lack of the knowledge of the whole patch-set. > > Patches 0001 and 0002 th

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-14 Thread Michael Paquier
On Mon, Mar 14, 2022 at 03:54:19PM +0530, Bharath Rupireddy wrote: > At times, the snapshot or mapping files can be large in number and one > some platforms it takes time for checkpoint to process all of them. > Having the stats about them in server logs can help us better analyze > why checkpoint

RE: Column Filtering in Logical Replication

2022-03-14 Thread shiy.f...@fujitsu.com
On Mon, Mar 14, 2022 5:08 AM Tomas Vondra wrote: > > On 3/12/22 05:30, Amit Kapila wrote: > >> ... > > > > Okay, please find attached. I have done basic testing of this, if we > > agree with this approach then this will require some more testing. > > > > Thanks, the proposed changes seem like a

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2022-03-14 Thread Justin Pryzby
On Mon, Mar 14, 2022 at 01:53:54PM +0900, Michael Paquier wrote: > On Wed, Mar 09, 2022 at 10:50:45AM -0600, Justin Pryzby wrote: > > I also changed pg_ls_dir_recurse() to handle concurrent removal of a dir, > > which > > I noticed caused an infrequent failure on CI. However I'm not including >

Re: Estimating HugePages Requirements?

2022-03-14 Thread Michael Paquier
On Mon, Mar 14, 2022 at 10:34:17AM -0700, Nathan Bossart wrote: > On Mon, Mar 14, 2022 at 04:26:43PM +0100, Magnus Hagander wrote: >> And in fact, the command documented on >> https://www.postgresql.org/docs/devel/kernel-resources.html doesn't >> actually produce the output that the docs show, it a

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2022-03-14 Thread Michael Paquier
On Mon, Mar 14, 2022 at 09:37:25PM -0500, Justin Pryzby wrote: > One could argue that most of the pg_ls_* functions aren't needed (including > 1922d7c6e), since the same things are possible with pg_ls_dir() and > pg_stat_file(). > |1922d7c6e Add SQL functions to monitor the directory contents of re

Re: [PATCH] Fix various spelling errors

2022-03-14 Thread Michael Paquier
On Mon, Mar 14, 2022 at 06:49:07PM -0500, Justin Pryzby wrote: > On Mon, Mar 14, 2022 at 11:03:50PM +, Kekalainen, Otto wrote: >> I propose the attached patch to be applied on the 'master' branch of >> PostgreSQL >> to fix various spelling errors. >> >> Most fixes are in comments and have no

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2022-03-14 Thread David Rowley
On Sat, 1 Jan 2022 at 15:40, Zhihong Yu wrote: > + locallock->nLocks -= locallockowner->nLocks; > + Assert(locallock->nLocks >= 0); > > I think the assertion is not needed since the above code is in if block : > > + if (locallockowner->nLocks < locallock->nLocks) > > the

Re: Skipping logical replication transactions on subscriber side

2022-03-14 Thread Masahiko Sawada
On Mon, Mar 14, 2022 at 6:50 PM shiy.f...@fujitsu.com wrote: > > On Fri, Mar 11, 2022 4:20 PM Masahiko Sawada wrote: > > > > I've attached an updated version patch. This patch can be applied on > > top of the latest disable_on_error patch[1]. > > > > Thanks for your patch. Here are some comments

  1   2   >