Re: Lowering the ever-growing heap->pd_lower

2021-08-02 Thread Simon Riggs
On Tue, 18 May 2021 at 20:33, Peter Geoghegan wrote: > > On Tue, May 18, 2021 at 12:29 PM Matthias van de Meent > wrote: > > PFA the updated version of this patch. Apart from adding line pointer > > truncation in PageRepairFragmentation (as in the earlier patches), I > > also altered PageTruncate

Re: when the startup process doesn't (logging startup delays)

2021-08-02 Thread Nitin Jadhav
> Thanks. Can you please have a look at what I suggested down toward the > bottom of > http://postgr.es/m/ca+tgmoap2wefsktmcgwt9lxuz7y99hnduyshpk7qnfuqb98...@mail.gmail.com ? Implemented the above approach and verified the patch. Kindly have a look and share your thoughts. Thanks & Regards, Niti

Re: Changes to recovery_min_apply_delay are ignored while waiting for delay

2021-08-02 Thread Kyotaro Horiguchi
At Mon, 2 Aug 2021 22:21:56 -0700, Soumyadeep Chakraborty wrote in > Hello, > > I came across this issue while I was tweaking a TAP test with Ashwin for > this thread [1]. > > We noticed that while the startup process waits for a recovery delay, it does > not respect changes to the recovery_mi

RE: Added schema level support for publication.

2021-08-02 Thread tanghy.f...@fujitsu.com
On Monday, August 2, 2021 11:40 PM vignesh C wrote: > > Thanks for the comments, attached v17 patches has the fixes for the same. Thanks for your new patch. I saw the following warning when compiling. It seems we don't need this variable any more. publicationcmds.c: In function ‘AlterPublicati

Re: Remove unused 'len' from pg_stat_recv_* functions

2021-08-02 Thread Kyotaro Horiguchi
At Tue, 3 Aug 2021 12:40:23 +0900, Masahiko Sawada wrote in > Hi all, > > While working on a patch adding new stats, houzj pointed out that > 'len' function arguments of all pgstat_recv_* functions are not used > at all. Looking at git history, pgstat_recv_* functions have been > having ‘len’ s

RE: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread osumi.takami...@fujitsu.com
On Tuesday, August 3, 2021 2:29 PM Masahiko Sawada wrote: > On Tue, Aug 3, 2021 at 11:47 AM Amit Kapila > wrote: > > > > On Mon, Aug 2, 2021 at 1:13 PM Masahiko Sawada > wrote: > > > > > > On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com > > > wrote: > > > > > > > > > > > > Accordin

Re: Slow standby snapshot

2021-08-02 Thread Andrey Borodin
Hi, > 3 авг. 2021 г., в 03:01, Andres Freund написал(а): > > Hi, > > On 2021-08-03 00:07:23 +0300, Michail Nikolaev wrote: >> The main idea is simple optimistic optimization - store offset to next >> valid entry. So, in most cases, we could just skip all the gaps. >> Of course, it adds some add

Re: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread vignesh C
On Mon, Aug 2, 2021 at 1:13 PM Masahiko Sawada wrote: > > On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com > wrote: > > > > On Thursday, July 29, 2021 10:50 AM Masahiko Sawada > > wrote: > > > On Thu, Jul 8, 2021 at 3:55 PM osumi.takami...@fujitsu.com > > > wrote: > > > > When the cu

Re: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread Masahiko Sawada
On Tue, Aug 3, 2021 at 11:47 AM Amit Kapila wrote: > > On Mon, Aug 2, 2021 at 1:13 PM Masahiko Sawada wrote: > > > > On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > > > > Accordingly, I'm thinking to have unsuccessful and successful stats on > > > the sub side.

Changes to recovery_min_apply_delay are ignored while waiting for delay

2021-08-02 Thread Soumyadeep Chakraborty
Hello, I came across this issue while I was tweaking a TAP test with Ashwin for this thread [1]. We noticed that while the startup process waits for a recovery delay, it does not respect changes to the recovery_min_apply_delay GUC. So: // Standby is waiting on recoveryWakeupLatch with a large re

Sort keys are omitted incorrectly after grouping sets

2021-08-02 Thread Richard Guo
Hi hackers, In each PathKey we have an EC representing the value being sorted on. This works correctly and efficiently in scan/join planning since the members in an EC are always equal to each other. However, after applying GROUP BY grouping sets, the members in an EC may not be that equivalent a

Remove unused 'len' from pg_stat_recv_* functions

2021-08-02 Thread Masahiko Sawada
Hi all, While working on a patch adding new stats, houzj pointed out that 'len' function arguments of all pgstat_recv_* functions are not used at all. Looking at git history, pgstat_recv_* functions have been having ‘len’ since the stats collector was introduced by commit 140ddb78fe 20 years ago b

Re: row filtering for logical replication

2021-08-02 Thread Peter Smith
FYI - v20 --> v21 (Only very minimal changes) * I noticed that the v20 TAP test (023_row_filter.pl) began failing due to a recent commit [1], so I have rebased it to keep the cfbot happy. -- [1] https://github.com/postgres/postgres/commit/201a76183e2056c2217129e12d68c25ec9c559c8 Kind Regar

RE: [PATCH]Comment improvement in publication.sql

2021-08-02 Thread tanghy.f...@fujitsu.com
On Monday, August 2, 2021 11:56 PM vignesh C wrote: > > Few minor suggestions: > 1) Should we change below to "fail - tables can't be added, dropped or > set to "FOR ALL TABLES" publications" > --- fail - can't add to for all tables publication > +-- fail - tables can't be added to or dropped fro

Re: A micro-optimisation for ProcSendSignal()

2021-08-02 Thread Andres Freund
Hi, On 2021-08-03 13:44:58 +1200, Thomas Munro wrote: > New idea. Instead of adding pgprocno to SERIALIZABLEXACT (which we > should really be trying to shrink, not grow), let's look it up by > vxid->backendId. I didn't consider that before, because I was trying > not to get tangled up with Back

Re: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread Amit Kapila
On Mon, Aug 2, 2021 at 1:13 PM Masahiko Sawada wrote: > > On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com > wrote: > > > > > > Accordingly, I'm thinking to have unsuccessful and successful stats on the > > sub side. > > Sawada-san is now implementing a new view in [1]. > > Do you thin

Re: A micro-optimisation for ProcSendSignal()

2021-08-02 Thread Soumyadeep Chakraborty
Hey Thomas, On Mon, Aug 2, 2021 at 6:45 PM Thomas Munro wrote: > > Hi Soumyadeep, > > On Sat, Jul 24, 2021 at 5:26 PM Soumyadeep Chakraborty > wrote: > > On Tue, Jul 20, 2021 at 10:40 PM Thomas Munro > > wrote: > > > I wonder why we need this member anyway, when you can compute it from > > > t

Re: archive status ".ready" files may be created too early

2021-08-02 Thread Kyotaro Horiguchi
At Mon, 2 Aug 2021 23:28:19 +, "Bossart, Nathan" wrote in > On 8/2/21, 2:42 PM, "Alvaro Herrera" wrote: > > I think it's okay to make lastNotifiedSeg protected by just info_lck, > > and RecordBoundaryMap protected by just ArchNotifyLock. It's okay to > > acquire the spinlock inside the lwl

Dividing line between before_shmem_exit and on_shmem_exit?

2021-08-02 Thread Andres Freund
Hi, I don't think I really understand what decides what should use before_shmem_exit() and what into on_shmem_exit(). The existing comments say: /* * before_shmem_exit * * Register early callback to perf

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Thomas Munro
On Tue, Aug 3, 2021 at 9:52 AM Thomas Munro wrote: > On Tue, Aug 3, 2021 at 1:17 AM Robert Haas wrote: > > That's great. I just realized that this leaves us with identical > > RequestCheckpoint() calls in two nearby places. Is there any reason > > not to further simplify as in the attached? > > L

Re: 2021-07 CF now in progress

2021-08-02 Thread Masahiko Sawada
On Tue, Aug 3, 2021 at 10:50 AM Masahiko Sawada wrote: > > On Tue, Aug 3, 2021 at 10:43 AM Tom Lane wrote: > > > > Masahiko Sawada writes: > > > I can close this commitfest. But should we mark uncommitted patches as > > > "Moved to next CF" or "Returned with Feedback" beforehand? > > > > Should

fix DECLARE tab completion

2021-08-02 Thread Shinya11.Kato
Hi! In the discussion of [1], the option ASENSITIVE was added to DECLARE. I have improved its tab completion and attached a patch. Do you think? [1] https://www.postgresql.org/message-id/flat/96ee8b30-9889-9e1b-b053-90e10c050e85%40enterprisedb.com Regards, Shinya Kato fix_declare_tab_complet

Re: 2021-07 CF now in progress

2021-08-02 Thread Masahiko Sawada
On Tue, Aug 3, 2021 at 10:43 AM Tom Lane wrote: > > Masahiko Sawada writes: > > I can close this commitfest. But should we mark uncommitted patches as > > "Moved to next CF" or "Returned with Feedback" beforehand? > > Should be moved to next CF, or at least most of them should be. > (But I doubt

Re: A micro-optimisation for ProcSendSignal()

2021-08-02 Thread Thomas Munro
Hi Soumyadeep, On Sat, Jul 24, 2021 at 5:26 PM Soumyadeep Chakraborty wrote: > On Tue, Jul 20, 2021 at 10:40 PM Thomas Munro wrote: > > I wonder why we need this member anyway, when you can compute it from > > the address... #define GetPGProcNumber(p) ((p) - ProcGlobal->allProcs) > > or somethin

Re: 2021-07 CF now in progress

2021-08-02 Thread Tom Lane
Masahiko Sawada writes: > I can close this commitfest. But should we mark uncommitted patches as > "Moved to next CF" or "Returned with Feedback" beforehand? Should be moved to next CF, or at least most of them should be. (But I doubt anyone wants to try to kill off patches that aren't going anyw

Re: 2021-07 CF now in progress

2021-08-02 Thread Masahiko Sawada
On Tue, Aug 3, 2021 at 12:34 AM Ibrar Ahmed wrote: > > > > On Mon, Jul 26, 2021 at 5:52 PM Ibrar Ahmed wrote: >> >> >> >> On Mon, Jul 19, 2021 at 4:37 PM Ibrar Ahmed wrote: >>> >>> >>> >>> On Mon, Jul 12, 2021 at 4:59 PM Ibrar Ahmed wrote: Hackers, The Commitfest 2021-0

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Mon, May 10, 2021 at 1:31 PM vignesh C wrote: > ... > > 2) I felt we can change lsn data type from Int64 to XLogRecPtr > + > +Int64 > + > +The LSN of the prepare. > + > + > + > + > +Int64 > + > +The end LSN of the transaction. > + > + > > 3) I felt we can change

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
Please find attached the latest patch set v102* Differences: * Rebased to HEAD @ today. * This is a documentation change only. A recent commit [1] has changed the documentation style for the message formats slightly to annotate the data types. For consistency, the same style change needs to be a

Re: Record a Bitmapset of non-pruned partitions

2021-08-02 Thread David Rowley
On Mon, 2 Aug 2021 at 20:16, David Rowley wrote: > If anyone has any objections to both the v4 0001 and 0002 patch, can > they let me know soon. I'm currently seeing no reason that they can't > go in. I've now pushed both of these. Thanks for the reviews. David

RE: make MaxBackends available in _PG_init

2021-08-02 Thread wangsh.f...@fujitsu.com
Hi, Bossart, Nathan wrote: > I just encountered the same thing, so I am bumping this thread. Thank you for bumping this thread. > > I've wondered, independent of this thread, about not making MaxBackends > > externally visible, and requiring a function call to access it. It should be > > easier

Re: archive status ".ready" files may be created too early

2021-08-02 Thread Bossart, Nathan
On 8/2/21, 2:42 PM, "Alvaro Herrera" wrote: > I think it's okay to make lastNotifiedSeg protected by just info_lck, > and RecordBoundaryMap protected by just ArchNotifyLock. It's okay to > acquire the spinlock inside the lwlock-protected area, as long as we > make sure never to do the opposite.

Re: disfavoring unparameterized nested loops

2021-08-02 Thread Mike Klaas
I think that it is worth paying more than nothing to avoid plans that are so far from optimal that they might as well take forever to execute I recently came across this article from 2016 that expounded upon a bad plan of the sort discussed in this thread: https://heap.io/blog/when-to-avoid-jsonb

Re: Use generation context to speed up tuplesorts

2021-08-02 Thread David Rowley
On Sat, 31 Jul 2021 at 08:38, Andres Freund wrote: > There is at least one path using tuplecontext that reaches code that > could end up freeing memory to a significant enough degree to care about > generation.c effectively not using that memory: > tuplesort_putdatum()->datumCopy()->EOH_flatten_in

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Bossart, Nathan
On 8/2/21, 3:42 PM, "Andres Freund" wrote: > I've wondered, independent of this thread, about not making MaxBackends > externally visible, and requiring a function call to access it. It should be > easier to find cases of misuse if we errored out when accessed at the wrong > time. And we could use

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 22:35:13 +, Bossart, Nathan wrote: > On 8/2/21, 3:12 PM, "Andres Freund" wrote: > > I think this is overblown. We already size resources *after* > > shared_preload_libraries' _PG_init() runs, because that's the whole point of > > shared_preload_libraries. What's proposed in

Re: Make relfile tombstone files conditional on WAL level

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 16:03:31 -0400, Robert Haas wrote: > The two most principled solutions to this problem that I can see are > (1) remove wal_level=minimal and I'm personally not opposed to this. It's not practically relevant and makes a lot of stuff more complicated. We imo should rather focus on

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Bossart, Nathan
On 8/2/21, 3:12 PM, "Andres Freund" wrote: > I think this is overblown. We already size resources *after* > shared_preload_libraries' _PG_init() runs, because that's the whole point of > shared_preload_libraries. What's proposed in this thread is to *disallow* > increasing resource usage in s_p_l

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-08-02 Thread Melanie Plageman
On Fri, Jun 4, 2021 at 5:52 PM Alvaro Herrera wrote: > > On 2021-Apr-12, Melanie Plageman wrote: > > > As for the way I have recorded strategy writes -- it is quite inelegant, > > but, I wanted to make sure that I only counted a strategy write as one > > in which the backend wrote out the dirty bu

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 21:57:18 +, Bossart, Nathan wrote: > On 8/2/21, 1:37 PM, "Andres Freund" wrote: > > On 2021-08-02 16:06:15 -0400, Robert Haas wrote: > >> I think this is a good idea. Anyone object? > > > > I'm not so sure it's a good idea. I've seen several shared_preload_library > > using

Re: Slow standby snapshot

2021-08-02 Thread Andres Freund
Hi, On 2021-08-03 00:07:23 +0300, Michail Nikolaev wrote: > The main idea is simple optimistic optimization - store offset to next > valid entry. So, in most cases, we could just skip all the gaps. > Of course, it adds some additional impact for workloads without long > (few seconds) transactions

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Bossart, Nathan
On 8/2/21, 1:37 PM, "Andres Freund" wrote: > On 2021-08-02 16:06:15 -0400, Robert Haas wrote: >> I think this is a good idea. Anyone object? > > I'm not so sure it's a good idea. I've seen several shared_preload_library > using extensions that adjust some GUCs (e.g. max_prepared_transactions) > be

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Thomas Munro
On Tue, Aug 3, 2021 at 1:17 AM Robert Haas wrote: > That's great. I just realized that this leaves us with identical > RequestCheckpoint() calls in two nearby places. Is there any reason > not to further simplify as in the attached? LGTM.

Re: archive status ".ready" files may be created too early

2021-08-02 Thread Alvaro Herrera
On 2021-Jul-31, Bossart, Nathan wrote: > This is why I was trying to get away with just using info_lck for > reading lastNotifiedSeg. ArchNotifyLock is mostly intended to protect > RecordBoundaryMap. However, since lastNotifiedSeg is used in > functions like GetLatestRecordBoundarySegment() that

Re: Implementing Incremental View Maintenance

2021-08-02 Thread Zhihong Yu
On Sun, Aug 1, 2021 at 11:30 PM Yugo NAGATA wrote: > Hi hackers, > > On Mon, 19 Jul 2021 09:24:30 +0900 > Yugo NAGATA wrote: > > > On Wed, 14 Jul 2021 21:22:37 +0530 > > vignesh C wrote: > > > > The patch does not apply on Head anymore, could you rebase and post a > > > patch. I'm changing the

Re: [PATCH] proposal for regexp_count, regexp_instr, regexp_substr and regexp_replace

2021-08-02 Thread Gilles Darold
Le 02/08/2021 à 01:21, Tom Lane a écrit : > Gilles Darold writes: >> [ v5-0001-regexp-foo-functions.patch ] > I've gone through this whole patch now, and found quite a lot that I did > not like. In no particular order: > > * Wrapping parentheses around the user's regexp doesn't work. It can > tu

Re: Slow standby snapshot

2021-08-02 Thread Michail Nikolaev
Hello. > I have tried such an approach but looks like it is not effective, > probably because of CPU caching issues. It was a mistake by me. I have repeated the approach and got good results with small and a non-invasive patch. The main idea is simple optimistic optimization - store offset to ne

Release 13 of the PostgreSQL BuildFarm client

2021-08-02 Thread Andrew Dunstan
I have pushed Release 13 of the PostgreSQL BuildFarm client. Change highlights: * add tests for a few cases that were previously missing * more fine-grained control over which TAP test sets run * --no-keepall can be specified on the command line * repair a repo if it crashed during a c

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 16:06:15 -0400, Robert Haas wrote: > On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan wrote: > > I just encountered the same thing, so I am bumping this thread. I was > > trying to use MaxBackends in a call to RequestAddinShmemSpace() in a > > _PG_init() function for a module, bu

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Alvaro Herrera
On 2021-Aug-02, Robert Haas wrote: > On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan wrote: > > > > I think the comments about needing to register background workers > > before initializing MaxBackends have been incorrect since the addition > > of max_worker_processes in v9.4 (6bc8ef0b). > I thin

Re: make MaxBackends available in _PG_init

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan wrote: > I just encountered the same thing, so I am bumping this thread. I was > trying to use MaxBackends in a call to RequestAddinShmemSpace() in a > _PG_init() function for a module, but since MaxBackends is not yet > initialized, you essentially

Re: Make relfile tombstone files conditional on WAL level

2021-08-02 Thread Robert Haas
On Thu, Jun 10, 2021 at 6:47 AM Heikki Linnakangas wrote: > It would indeed be nice to have some other mechanism to prevent the > issue with wal_level=minimal, the tombstone files feel hacky and > complicated. Maybe a new shared memory hash table to track the > relfilenodes of dropped tables. Jus

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Alvaro Herrera
On 2021-Aug-02, Andres Freund wrote: > On Mon, Aug 2, 2021, at 12:12, Alvaro Herrera wrote: > > Hmm, I don't remember that an shmem-unconnected bgworker first connected > > to it and then let go. It seems weird to do it that way. My intention, > > as far as I recall, is that they would just nev

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, On Mon, Aug 2, 2021, at 12:12, Alvaro Herrera wrote: > On 2021-Aug-02, Andres Freund wrote: > > I do think there's some potential gains in simplicity and robustness > > that are made mildly harder by a subprocess that first attaches and > > detaches from shm (it's the only case where we can't

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Alvaro Herrera
On 2021-Aug-02, Andres Freund wrote: > > When I included this case I was thinking in tasks which would just run > > stuff not directly connected to data. Something like a sub-daemon: say > > a connection pooler, which is a bgworker just so that it starts and > > stops together with postmaster, an

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, On Mon, Aug 2, 2021, at 11:08, Alvaro Herrera wrote: > On 2021-Aug-02, Tom Lane wrote: > > > Robert Haas writes: > > > If you're saying that this code has been 100% broken for 7 years and > > > nobody's noticed until now, then that suggests that nobody actually > > > uses non-shmem-connected

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Alvaro Herrera
On 2021-Aug-02, Tom Lane wrote: > Robert Haas writes: > > If you're saying that this code has been 100% broken for 7 years and > > nobody's noticed until now, then that suggests that nobody actually > > uses non-shmem-connected bgworkers. I sort of hate to give up on that > > concept but if we've

Re: straightening out backend process startup

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 12:57:36 -0400, Tom Lane wrote: > Andres Freund writes: > > I think there's quite a few different issues around this - here I'm just > > trying to tackle a few of the most glaring (to me): > > No opinion yet about most of this, but I did want to react to > > > In fact, I think

Re: straightening out backend process startup

2021-08-02 Thread Tom Lane
Andres Freund writes: > I think there's quite a few different issues around this - here I'm just > trying to tackle a few of the most glaring (to me): No opinion yet about most of this, but I did want to react to > In fact, I think there's a good argument to be made that we should > entirely rem

Passing data out of indexam or tableam

2021-08-02 Thread Chris Cleveland
Hi, I'm attempting to develop a new type of full-text index. I've started building a new index access method. Is it possible to calculate values during the index scan and then make them available for display in a select clause? For example, SELECT title, score, total_hits FROM mytable WHERE (som

straightening out backend process startup

2021-08-02 Thread Andres Freund
Hi, I've previously complained ([1]) that process initialization has gotten very complicated. I hit this once more last week when trying to commit one of the shared memory stats pieces... I think there's quite a few different issues around this - here I'm just trying to tackle a few of the most g

Re: Minimal logical decoding on standbys

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 16:45:23 +0200, Drouvot, Bertrand wrote: > On 7/27/21 7:22 PM, Andres Freund wrote: > > On 2021-07-27 09:23:48 +0200, Drouvot, Bertrand wrote: > > > Thanks for the warning, rebase done and new v21 version attached. > > Did you have a go at fixing the walsender race conditions I >

Re: [PATCH]Comment improvement in publication.sql

2021-08-02 Thread vignesh C
On Mon, Aug 2, 2021 at 3:31 PM tanghy.f...@fujitsu.com wrote: > > Hi Hackers > > When review and test another patch at [1], I found some comments in existing > test code of " src/test/regress/sql/publication.sql " is a little bit > confused. > Attached a patch to fix them, please take a check. >

Re: Corrected documentation of data type for the logical replication message formats.

2021-08-02 Thread vignesh C
On Mon, Aug 2, 2021 at 9:10 PM Tom Lane wrote: > > Peter Smith writes: > > I agree. The specified value looks better when it comes first, as you did > > it. > > Actually, it looks to me like we don't have to resolve the question of > which should come first, because I don't see any cases where i

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, On 2021-08-02 11:00:49 -0400, Tom Lane wrote: > Robert Haas writes: > > If you're saying that this code has been 100% broken for 7 years and > > nobody's noticed until now, then that suggests that nobody actually > > uses non-shmem-connected bgworkers. I sort of hate to give up on that > > co

Re: Minimal logical decoding on standbys

2021-08-02 Thread Ronan Dunklau
Le lundi 2 août 2021, 17:31:46 CEST Drouvot, Bertrand a écrit : > > I think the beahviour when dropping a database on the primary should be > > documented, and proper procedures for handling it correctly should be > > suggested. > > > > Something along the lines of: > > > > "If a database is drop

Re: Corrected documentation of data type for the logical replication message formats.

2021-08-02 Thread Tom Lane
Peter Smith writes: > I agree. The specified value looks better when it comes first, as you did it. Actually, it looks to me like we don't have to resolve the question of which should come first, because I don't see any cases where it's useful to have both. I don't agree with appending "uint8" t

Re: 2021-07 CF now in progress

2021-08-02 Thread Ibrar Ahmed
On Mon, Jul 26, 2021 at 5:52 PM Ibrar Ahmed wrote: > > > On Mon, Jul 19, 2021 at 4:37 PM Ibrar Ahmed wrote: > >> >> >> On Mon, Jul 12, 2021 at 4:59 PM Ibrar Ahmed >> wrote: >> >>> >>> Hackers, >>> >>> The Commitfest 2021-07 is now in progress. It is one of the biggest one. >>> Total number of p

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Tom Lane
Robert Haas writes: > If you're saying that this code has been 100% broken for 7 years and > nobody's noticed until now, then that suggests that nobody actually > uses non-shmem-connected bgworkers. I sort of hate to give up on that > concept but if we've really gone that many years without anyone

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 2:51 AM Andres Freund wrote: > which presents a problem: We've initialized all kind of references to shared > memory, own a PGPROC, but have detached from shared memory. > > In practice this will lead pretty quickly to a segfault, because process exit > will run proc_exit ca

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 9:51 AM Jakub Wartak wrote: > BTW, if you now there's this big push for refactoring StartupXLOG() then what > frustrating^H^H^H^H^H could be done better - at least from end-user point of > view - > is that there is lack of near real time cyclic messages (every 1min?) about

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 9:40 AM Amul Sul wrote: > > That's great. I just realized that this leaves us with identical > > RequestCheckpoint() calls in two nearby places. Is there any reason > > not to further simplify as in the attached? > > > +1, also, can we just get rid off "promoted" flag? The o

[BUG] wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION

2021-08-02 Thread houzj.f...@fujitsu.com
Hi hackers, When testing some other logical replication related patches, I found two unexpected behaviours about ALTER SUBSCRIPTION ADD/DROP PUBLICATION. (1) when I execute the following sqls[1], the data of tables that registered to 'pub' wasn't copied to the subscriber side which means tablesy

RE: Background writer and checkpointer in crash recovery

2021-08-02 Thread Jakub Wartak
> On Fri, Jul 30, 2021 at 4:00 PM Andres Freund wrote: > > I don't agree with that? If (user+system) << wall then it is very > > likely that recovery is IO bound. If system is a large percentage of > > wall, then shared buffers is likely too small (or we're replacing the > > wrong > > buffers) bec

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Amul Sul
On Mon, Aug 2, 2021 at 6:47 PM Robert Haas wrote: > > On Mon, Aug 2, 2021 at 1:37 AM Thomas Munro wrote: > > I pushed 0001. > > That's great. I just realized that this leaves us with identical > RequestCheckpoint() calls in two nearby places. Is there any reason > not to further simplify as in th

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Robert Haas
On Mon, Aug 2, 2021 at 1:37 AM Thomas Munro wrote: > I pushed 0001. That's great. I just realized that this leaves us with identical RequestCheckpoint() calls in two nearby places. Is there any reason not to further simplify as in the attached? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Background writer and checkpointer in crash recovery

2021-08-02 Thread Robert Haas
On Fri, Jul 30, 2021 at 4:00 PM Andres Freund wrote: > I don't agree with that? If (user+system) << wall then it is very likely > that recovery is IO bound. If system is a large percentage of wall, then > shared buffers is likely too small (or we're replacing the wrong > buffers) because you spend

Re: .ready and .done files considered harmful

2021-08-02 Thread Dipesh Pandit
Hi, > I think what you are saying is true before v14, but not in v14 and master. Yes, we can use archiver specific shared memory. Thanks. > I don't think it's great that we're using up SIGINT for this purpose. > There aren't that many signals available at the O/S level that we can > use for our p

Update of partition key on foreign server

2021-08-02 Thread Илья Гладышев
Hi, I am currently looking into a partition constraint violation that occurs on update of a partition key on a foreign server. To reproduce it you can run: On server 1 using port 5432: create extension postgres_fdw; create table players (id integer, name text) partition by list(name); create ta

Re: Minimal logical decoding on standbys

2021-08-02 Thread Ronan Dunklau
Le mardi 27 juillet 2021, 09:23:48 CEST Drouvot, Bertrand a écrit : > Thanks for the warning, rebase done and new v21 version attached. > > Bertrand Hello, I've taken a look at this patch, and it looks like you adressed every prior remark, including the race condition Andres was worried about.

Re: Use generation context to speed up tuplesorts

2021-08-02 Thread David Rowley
On Sat, 31 Jul 2021 at 14:34, Tomas Vondra wrote: > I spent a bit of time hacking on the Generation context, adding the two > improvements discussed in this thread: > > 1) internal handling of block sizes, similar to what AllocSet does (it > pretty much just copies parts of it) > > 2) keeper block

Re: Speeding up GIST index creation for tsvectors

2021-08-02 Thread John Naylor
On Sun, Aug 1, 2021 at 11:41 PM Amit Khandekar wrote: > > > FWIW, I anticipate some push back from the community because of the fact that the optimization relies on statistical phenomena. > > I dug into this issue for tsvector type. Found out that it's the way > in which the sign array elements ar

[PATCH]Comment improvement in publication.sql

2021-08-02 Thread tanghy.f...@fujitsu.com
Hi Hackers When review and test another patch at [1], I found some comments in existing test code of " src/test/regress/sql/publication.sql " is a little bit confused. Attached a patch to fix them, please take a check. Here is the detail: Existing code: CREATE TABLE testpub_tbl2 (id serial prim

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Amit Kapila
On Sun, Aug 1, 2021 at 3:51 PM Peter Smith wrote: > > On Sun, Aug 1, 2021 at 3:05 AM vignesh C wrote: > > > > On Sat, Jul 31, 2021 at 11:12 AM Ajin Cherian wrote: > > > > > > On Sat, Jul 31, 2021 at 2:39 PM Amit Kapila > > > wrote: > > > > > > > Here, the test is expecting 2 prepared transacti

Re: logical replication empty transactions

2021-08-02 Thread Amit Kapila
On Fri, Jul 23, 2021 at 3:39 PM Ajin Cherian wrote: > Let's first split the patch for prepared and non-prepared cases as that will help to focus on each of them separately. BTW, why haven't you considered implementing point 1b as explained by Andres in his email [1]? I think we can send a keepali

Re: Record a Bitmapset of non-pruned partitions

2021-08-02 Thread David Rowley
On Mon, 2 Aug 2021 at 00:31, David Rowley wrote: > I had another self review of these and I'm pretty happy with them. I'm > quite glad to see the performance of querying a single partition of a > table with large numbers of partitions no longer tails off as much as > it used to. I did some profil

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 4:33 PM Greg Nancarrow wrote: > > On Fri, Jul 30, 2021 at 2:02 PM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > A few minor comments: > > (1) doc/src/sgml/protocol.sgml > > In the following description, is

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 3:18 PM vignesh C wrote: > > On Fri, Jul 30, 2021 at 9:32 AM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > Differences: > > > > * Rebased to HEAD @ today (needed because some recent commits [1][2] broke >

Re: Failed transaction statistics to measure the logical replication progress

2021-08-02 Thread Masahiko Sawada
On Mon, Aug 2, 2021 at 2:52 PM osumi.takami...@fujitsu.com wrote: > > On Thursday, July 29, 2021 10:50 AM Masahiko Sawada > wrote: > > On Thu, Jul 8, 2021 at 3:55 PM osumi.takami...@fujitsu.com > > wrote: > > > When the current HEAD fails during logical decoding, the failure > > > increments tx

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 6:25 PM tanghy.f...@fujitsu.com wrote: > > On Friday, July 30, 2021 12:02 PM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > Thanks for your patch. A few comments on the test file: > > 1. src/test/subscriptio

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Sat, Jul 31, 2021 at 9:36 PM Amit Kapila wrote: > > On Fri, Jul 30, 2021 at 9:32 AM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > Few minor comments: > 1. > CREATE SUBSCRIPTION regress_testsub CONNECTION > 'dbname=regress_doesnotexist' PUBLICATION testpub W

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
Please find attached the latest patch v101 Differences: * Rebased to HEAD @ today. * Addresses all v100 review comments from Vignesh [1], Greg [2], Tang [3], and Amit [2]. [1] https://www.postgresql.org/message-id/CALDaNm2N3qgSv3XyHW%2Bop_SJcLmz1s%3D0jJc-taxUmeEBXW5EPw%40mail.gmail.com [2

Re: EXEC_BACKEND vs bgworkers without BGWORKER_SHMEM_ACCESS

2021-08-02 Thread Andres Freund
Hi, On 2021-08-01 23:51:16 -0700, Andres Freund wrote: > In practice this will lead pretty quickly to a segfault, because process exit > will run proc_exit callbacks, which in turn will try to do a ProcKill(). Or > logging dereferences MyProc, or ... > > It seems the above code block would need t