Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-21 Thread Michael Paquier
On Thu, Oct 22, 2020 at 02:27:34PM +0800, Julien Rouhaud wrote: > I'm a bit worried about this approach, as if I understand correctly > this can lead to false positive reports. I've certainly seen systems > with IO stalled for more than 500ms, so while this is not frequent > this could still happe

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread Kyotaro Horiguchi
At Thu, 22 Oct 2020 18:54:43 +1300, Thomas Munro wrote in > On Thu, Oct 22, 2020 at 5:52 PM Tom Lane wrote: > > Per the referenced bug-reporting thread, it was ReiserFS and/or NFS on > > SLES 9.3; so, dubious storage choices on an ancient-even-then Linux > > kernel. > > O. I can reproduce

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread Thomas Munro
On Thu, Oct 22, 2020 at 7:33 PM Kyotaro Horiguchi wrote: > At Thu, 22 Oct 2020 14:16:37 +0900 (JST), Kyotaro Horiguchi > wrote in > > smgrtruncate and msgrextend modifies that cache from their parameter, > > not from lseek(). At the very first the value in the cache comes from > > lseek() but i

Re: Enumize logical replication message actions

2020-10-21 Thread Ashutosh Bapat
Thanks Andres for your review. Thanks Li, Horiguchi-san and Amit for your comments. On Tue, 20 Oct 2020 at 04:57, Andres Freund wrote: > Hi, > > On 2020-10-16 12:55:26 +0530, Ashutosh Bapat wrote: > > Here's a patch simplifying that for top level logical replication > > messages. > > I think tha

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread Kyotaro Horiguchi
At Thu, 22 Oct 2020 14:16:37 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 22 Oct 2020 16:35:27 +1300, Thomas Munro > wrote in > > On Thu, Oct 22, 2020 at 3:07 PM k.jami...@fujitsu.com > > wrote: > > But... does the proposed caching behaviour and "accurate" flag really > > help with any

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-21 Thread Julien Rouhaud
On Thu, Oct 22, 2020 at 9:25 AM Michael Paquier wrote: > > On Thu, Oct 22, 2020 at 12:47:03AM +0300, Anastasia Lubennikova wrote: > > Thank you. I always forget about this. Do we have any checklist for such > > changes, that patch authors and reviewers can use? > > Not really. That's more a habit

Re: Track statistics for streaming of in-progress transactions

2020-10-21 Thread Dilip Kumar
On Wed, Oct 14, 2020 at 9:09 AM Amit Kapila wrote: > > Commit 464824323e has added the support of the streaming of > in-progress transactions into the built-in logical replication. The > attached patch adds the statistics about transactions streamed to the > decoding output plugin from ReorderBuff

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread Kyotaro Horiguchi
At Thu, 22 Oct 2020 01:33:31 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Jamison, Kirk/ジャミソン カーク > > The table below shows the vacuum execution time for non-recovery case. > > I've also subtracted the execution time when VACUUM (truncate off) is set. > > > > [NON-RECOVERY CASE - VACU

Re: Track statistics for streaming of in-progress transactions

2020-10-21 Thread Ajin Cherian
On Tue, Oct 20, 2020 at 4:29 PM Amit Kapila wrote: > Thanks. One thing I have considered while updating this patch was to > write a test case similar to what we have for spilled stats in > test_decoding/sql/stats.sql but I decided not to do it as that doesn't > seem to add much value for the stre

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread Thomas Munro
On Thu, Oct 22, 2020 at 5:52 PM Tom Lane wrote: > Per the referenced bug-reporting thread, it was ReiserFS and/or NFS on > SLES 9.3; so, dubious storage choices on an ancient-even-then Linux > kernel. O. I can reproduce that on a modern Linux box by forcing writeback to a full NFS filesystem

Re: Mop-up around psql's \connect behavior

2020-10-21 Thread Kyotaro Horiguchi
At Thu, 22 Oct 2020 00:34:20 -0400, Tom Lane wrote in > Kyotaro Horiguchi writes: > > At Wed, 21 Oct 2020 18:59:04 -0400, Tom Lane wrote in > > But once it got on my mind, it might be strange that just \c or \c > > -reuse-previous=y doesn't reconnect a broken session. It might be > > better we

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread Kyotaro Horiguchi
At Thu, 22 Oct 2020 16:35:27 +1300, Thomas Munro wrote in > On Thu, Oct 22, 2020 at 3:07 PM k.jami...@fujitsu.com > wrote: > +/* > + * Get the total number of to-be-invalidated blocks of a relation as well > + * as the total blocks for a given fork. The cached value returned by > +

Re: [HACKERS] Custom compression methods

2020-10-21 Thread Dilip Kumar
On Thu, Oct 22, 2020 at 2:11 AM Tomas Vondra wrote: > > On Wed, Oct 21, 2020 at 01:59:50PM +0530, Dilip Kumar wrote: > >On Sat, Oct 17, 2020 at 11:34 AM Dilip Kumar wrote: > >> > >> On Tue, Oct 13, 2020 at 10:30 AM Dilip Kumar wrote: > >> > > >> > On Mon, Oct 12, 2020 at 7:32 PM Tomas Vondra > >

Re: Add statistics to pg_stat_wal view for wal related parameter tuning

2020-10-21 Thread Kyotaro Horiguchi
At Thu, 22 Oct 2020 10:44:53 +0900, Masahiro Ikeda wrote in > On 2020-10-21 18:03, Kyotaro Horiguchi wrote: > > At Tue, 20 Oct 2020 16:11:29 +0900, Masahiro Ikeda > > wrote in > >> On 2020-10-20 12:46, Amit Kapila wrote: > >> > I see that we also need to add extra code to capture these stats (s

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread Tom Lane
Thomas Munro writes: > Hmmm. The Linux comment led me to commit ffae5cc and a 2006 thread[1] > showing a buggy sequence of system calls. Hah, blast from the past ... > AFAICS it was not even an > SMP/race problem of the type you might half expect, it was a single > process not seeing its own wr

User accounts on windows

2020-10-21 Thread Joel Mariadasan (jomariad)
Background: We have an installer for our application as part of which we are planning to include archive postgresql-13.0-1-windows-x64-binaries.zip which will be extracted along with the installation of our application. When the archive is extracted the folder's permission will belong to the cu

Re: Mop-up around psql's \connect behavior

2020-10-21 Thread Tom Lane
Kyotaro Horiguchi writes: > At Wed, 21 Oct 2020 18:59:04 -0400, Tom Lane wrote in >> ... What I propose >> is to complain if we have no o_conn *and* we are asked to re-use >> parameters from it. Otherwise, it's fine. > The reason I haven't complain about this is I don't reconnect by \c > afte

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-21 Thread Greg Nancarrow
On Fri, Oct 16, 2020 at 9:26 PM Amit Kapila wrote: > > > Cool, let me try to explain my thoughts a bit more. The idea is first > (in standard_planner) we check if there is any 'parallel_unsafe' > function/expression (via max_parallel_hazard) in the query tree. If we > don't find anything 'parallel

Re: A new function to wait for the backend exit after termination

2020-10-21 Thread Bharath Rupireddy
On Thu, Oct 22, 2020 at 8:39 AM David G. Johnston wrote: > >> If the backend is terminated within the user specified timeout then >> the function returns true, otherwise false. > > I’m suggesting an option for the second case to fail instead of returning > false. > That seems fine. > >> > >> >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread Thomas Munro
On Thu, Oct 22, 2020 at 3:07 PM k.jami...@fujitsu.com wrote: +/* + * Get the total number of to-be-invalidated blocks of a relation as well + * as the total blocks for a given fork. The cached value returned by + * smgrnblocks could be smaller than the actual number of existing bu

Re: Implementing Incremental View Maintenance

2020-10-21 Thread Yugo NAGATA
Hi Adam Brusselback, On Mon, 31 Dec 2018 11:20:11 -0500 Adam Brusselback wrote: > Hi all, just wanted to say I am very happy to see progress made on this, > my codebase has multiple "materialized tables" which are maintained with > statement triggers (transition tables) and custom functions. Th

Re: A new function to wait for the backend exit after termination

2020-10-21 Thread David G. Johnston
On Wednesday, October 21, 2020, Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > Thanks for the feedback. > > On Wed, Oct 21, 2020 at 8:01 PM David G. Johnston > wrote: > > > > On Wed, Oct 21, 2020 at 6:13 AM Magnus Hagander > wrote: > >> > >> I think it would be nicer to hav

Re: Mop-up around psql's \connect behavior

2020-10-21 Thread Kyotaro Horiguchi
At Wed, 21 Oct 2020 18:59:04 -0400, Tom Lane wrote in > While working on commit 85c54287a, I noticed a few things I did not > much care for in do_connect(). These don't quite seem to rise to > the level of back-patchable bugs, but they're still not great: > > * The initial stanza that complains

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-21 Thread Amit Kapila
On Thu, Oct 22, 2020 at 4:58 AM Peter Smith wrote: > > On Wed, Oct 21, 2020 at 7:42 PM Amit Kapila wrote: > > > > On Wed, Oct 21, 2020 at 1:38 PM Peter Smith wrote: > > > > > > The PG docs for PREPARE TRANSACTION [1] don't say anything about an > > > empty (zero length) transaction-id. > > > e.g

Re: A new function to wait for the backend exit after termination

2020-10-21 Thread Bharath Rupireddy
Thanks for the feedback. On Wed, Oct 21, 2020 at 8:01 PM David G. Johnston wrote: > > On Wed, Oct 21, 2020 at 6:13 AM Magnus Hagander wrote: >> >> I think it would be nicer to have a pg_terminate_backend(pid, wait=false), >> so a function with a second parameter which defaults to the current >

Re: A new function to wait for the backend exit after termination

2020-10-21 Thread Andres Freund
On 2020-10-21 15:13:36 +0200, Magnus Hagander wrote: > It seems this one also very much would need a timeout value. I'm not really against that, but I wonder if we just end up reimplementing statement timeout...

Re: A new function to wait for the backend exit after termination

2020-10-21 Thread Bharath Rupireddy
Thanks for the feedback. On Wed, Oct 21, 2020 at 6:43 PM Magnus Hagander wrote: > >> Currently pg_terminate_backend(), sends SIGTERM to the backend process but >> doesn't ensure it's exit. There are chances that backends still are >> running(even after pg_terminate_backend() is called) until th

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread k.jami...@fujitsu.com
On Thursday, October 22, 2020 10:34 AM, Tsunakwa-san wrote: > > I have confirmed that the above comment (commenting out the lines in > > RelationTruncate) solves the issue for non-recovery case. > > The attached 0004 patch is just for non-recovery testing and is not > > included in the final set of

Re: Allow some recovery parameters to be changed with reload

2020-10-21 Thread Kyotaro Horiguchi
At Thu, 22 Oct 2020 01:59:07 +0300, Anastasia Lubennikova wrote in > On 10.08.2020 23:20, Robert Haas wrote: > > On Sun, Aug 9, 2020 at 1:21 AM Michael Paquier > > wrote: > >> Sorry for the late reply. I have been looking at that stuff again, > >> and restore_command can be called in the conte

Re: PostgresNode::backup uses spread checkpoint?

2020-10-21 Thread Michael Paquier
On Wed, Oct 21, 2020 at 07:55:18AM +0900, Michael Paquier wrote: > Some nits: I would recommend to use the long option name, and list > the option name and its value as two separate arguments of the > command. For the archives: this got applied as of 831611b. -- Michael signature.asc Description

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread tsunakawa.ta...@fujitsu.com
The patch looks good except for the minor one: (1) +* as the total nblocks for a given fork. The cached value returned by nblocks -> blocks Regards Takayuki Tsunakawa

Re: Add statistics to pg_stat_wal view for wal related parameter tuning

2020-10-21 Thread Masahiro Ikeda
On 2020-10-21 18:03, Kyotaro Horiguchi wrote: At Tue, 20 Oct 2020 16:11:29 +0900, Masahiro Ikeda wrote in On 2020-10-20 12:46, Amit Kapila wrote: > I see that we also need to add extra code to capture these stats (some > of which is in performance-critical path especially in > XLogInsertRecord)

Re: Online verification of checksums

2020-10-21 Thread Michael Paquier
On Wed, Oct 21, 2020 at 07:10:34PM +0900, Michael Paquier wrote: > My guess is that we should be able to make use of that for base > backups as well, but I also think that I'd rather let v13 go with more > retries without depending on a new API layer, removing of the LSN > check altogether. Thinki

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-21 Thread Masahiko Sawada
On Wed, 21 Oct 2020 at 18:33, tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > So what's your opinion? > > My opinion is simple and has not changed. Let's clarify and refine the > design first in the following areas (others may have pointed out something > else too, but I don't

Re: Use list_delete_xxxcell O(1) instead of list_delete_ptr O(N) in some places

2020-10-21 Thread David Rowley
On Fri, 16 Oct 2020 at 16:42, Hou, Zhijie wrote: > And after checking the code again and I found two more places which can be > improved. > > 1. > --- a/src/backend/parser/parse_expr.c > +++ b/src/backend/parser/parse_expr.c > @@ -1702,7 +1702,7 @@ transformMultiAssignRef(ParseState *pstate, > M

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread tsunakawa.ta...@fujitsu.com
> As for 0004: > When testing TRUNCATE, remove the change to storage.c because it was > intended to troubleshoot the VACUUM test. I meant vacuum.c. Sorry. Regards Takayuki Tsunakawa

Re: Add statistics to pg_stat_wal view for wal related parameter tuning

2020-10-21 Thread Masahiro Ikeda
On 2020-10-21 15:54, lchch1...@sina.cn wrote: I think it's really a more convenient way to collect wal usage information, with it we can query when I want. Several points on my side: Thanks for your comments. 1. It will be nice If you provide a chance to reset the information in WalStats, so

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > I have confirmed that the above comment (commenting out the lines in > RelationTruncate) solves the issue for non-recovery case. > The attached 0004 patch is just for non-recovery testing and is not included > in > the final set of patches to be committed for vacuu

Re: Is Recovery actually paused?

2020-10-21 Thread Kyotaro Horiguchi
At Wed, 21 Oct 2020 11:14:24 -0400, Robert Haas wrote in > On Wed, Oct 21, 2020 at 7:16 AM Dilip Kumar wrote: > > One idea could be, if the recovery process is waiting for WAL and a > > recovery pause is requested then we can assume that the recovery is > > paused because before processing the

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-21 Thread Michael Paquier
On Thu, Oct 22, 2020 at 12:47:03AM +0300, Anastasia Lubennikova wrote: > Thank you. I always forget about this. Do we have any checklist for such > changes, that patch authors and reviewers can use? Not really. That's more a habit than anything else where any non-static routine that we publish co

Re: Add statistics to pg_stat_wal view for wal related parameter tuning

2020-10-21 Thread Masahiro Ikeda
On 2020-10-21 13:41, Amit Kapila wrote: On Tue, Oct 20, 2020 at 12:41 PM Masahiro Ikeda wrote: On 2020-10-20 12:46, Amit Kapila wrote: > On Tue, Oct 20, 2020 at 8:01 AM Masahiro Ikeda >> 1. Basic statistics of WAL activity >> >> - wal_records: Total number of WAL records generated >> - wal_fpi

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-21 Thread k.jami...@fujitsu.com
On Wednesday, October 21, 2020 4:37 PM, Tsunakawa-san wrote: > RelationTruncate() invalidates the cached fork sizes as follows. This causes > smgrnblocks() return accurate=false, resulting in not running optimization. > Try commenting out for non-recovery case. > > /* > * Make sure smgr_

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-21 Thread Peter Smith
On Wed, Oct 21, 2020 at 7:42 PM Amit Kapila wrote: > > On Wed, Oct 21, 2020 at 1:38 PM Peter Smith wrote: > > > > The PG docs for PREPARE TRANSACTION [1] don't say anything about an > > empty (zero length) transaction-id. > > e.g. PREPARE TRANSACTION ''; > > [1] https://www.postgresql.org/docs/cu

Re: Mop-up around psql's \connect behavior

2020-10-21 Thread Chapman Flack
On 10/21/20 18:59, Tom Lane wrote: > I think the reason we've not had complaints about this is that the > situation normally doesn't arise in interactive sessions (since we > won't release the old connection voluntarily), while scripts are > likely not designed to cope with connection losses anywa

Re: Allow some recovery parameters to be changed with reload

2020-10-21 Thread Anastasia Lubennikova
On 10.08.2020 23:20, Robert Haas wrote: On Sun, Aug 9, 2020 at 1:21 AM Michael Paquier wrote: Sorry for the late reply. I have been looking at that stuff again, and restore_command can be called in the context of a WAL sender process within the page_read callback of logical decoding via XLogRe

Mop-up around psql's \connect behavior

2020-10-21 Thread Tom Lane
While working on commit 85c54287a, I noticed a few things I did not much care for in do_connect(). These don't quite seem to rise to the level of back-patchable bugs, but they're still not great: * The initial stanza that complains about if (!o_conn && (!dbname || !user || !host || !port

Re: [DOC] Document concurrent index builds waiting on each other

2020-10-21 Thread David G. Johnston
On Wed, Oct 21, 2020 at 3:25 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > > v3-0002 needs a rebase over the create_index.sgml page due to the change > of the nearby xref to link. Attached as v4-0002 along with the original > v3-0001. > > attached... Reading the commit message on 0

Re: [DOC] Document concurrent index builds waiting on each other

2020-10-21 Thread David G. Johnston
On Wed, Sep 30, 2020 at 2:10 AM Michael Paquier wrote: > On Tue, Sep 08, 2020 at 01:25:21PM -0400, James Coleman wrote: > > Álvaro's patch confused the current state of this thread, so I'm > > reattaching (rebased) v2 as v3. > > + > + CREATE INDEX (including the > CONCURRENTLY > + option) c

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-21 Thread Anastasia Lubennikova
On 20.10.2020 09:24, Michael Paquier wrote: On Fri, Oct 16, 2020 at 06:02:52PM +0300, Anastasia Lubennikova wrote: In the current patch, PageIsVerifed called from pgbasebackup simply doesn't Should we change this too? I don't see any difference. I considered that, but now that does not seem wor

Re: language cleanups in code and docs

2020-10-21 Thread Thomas Munro
On Wed, Jun 17, 2020 at 10:32 PM Magnus Hagander wrote: > In looking at this I realize we also have exactly one thing referred to as > "blacklist" in our codebase, which is the "enum blacklist" (and then a small > internal variable in pgindent). AFAICT, it's not actually exposed to > userspace

Re: [PATCH] Add section headings to index types doc

2020-10-21 Thread David G. Johnston
On Wed, Sep 30, 2020 at 4:25 AM Dagfinn Ilmari Mannsåker wrote: > Michael Paquier writes: > > > On Mon, Aug 10, 2020 at 12:52:17PM +, Jürgen Purtz wrote: > >> The new status of this patch is: Waiting on Author > > > > This has not been answered yet, so I have marked the patch as returned > >

Re: Use of "long" in incremental sort code

2020-10-21 Thread Tomas Vondra
On Wed, Oct 21, 2020 at 06:06:52AM +, Tang, Haiying wrote: Hi Found one more place needed to be changed(long -> int64). Also changed the output for int64 data(Debug mode on & define EXEC_SORTDEBUG ) And, maybe there's a typo in " src\backend\executor\nodeIncrementalSort.c" as below. Obvi

Re: new heapcheck contrib module

2020-10-21 Thread Mark Dilger
> On Oct 21, 2020, at 1:51 PM, Tom Lane wrote: > > Mark Dilger writes: >> There is still something screwy here, though, as this compiles, links and >> runs fine for me on mac and linux, but not for Robert. > > Are you using --enable-nls at all on your Mac build? Because for sure it > shoul

Re: new heapcheck contrib module

2020-10-21 Thread Tom Lane
Mark Dilger writes: > There is still something screwy here, though, as this compiles, links and > runs fine for me on mac and linux, but not for Robert. Are you using --enable-nls at all on your Mac build? Because for sure it should not work there, given the failure to include -lintl in amcheck

Re: new heapcheck contrib module

2020-10-21 Thread Tom Lane
Robert Haas writes: > I was about to commit 0001, after making some cosmetic changes, when I > discovered that it won't link for me. I think there must be something > wrong with the NLS stuff. My version of 0001 is attached. The error I > got is: Well, the short answer would be "you need to add

Re: new heapcheck contrib module

2020-10-21 Thread Mark Dilger
> On Oct 21, 2020, at 1:13 PM, Alvaro Herrera wrote: > > On 2020-Oct-21, Robert Haas wrote: > >> On Wed, Oct 7, 2020 at 9:01 PM Mark Dilger >> wrote: >>> This next version, attached, has the acl checking and associated >>> documentation changes split out into patch 0005, making it easier t

Re: [HACKERS] Custom compression methods

2020-10-21 Thread Tomas Vondra
On Wed, Oct 21, 2020 at 01:59:50PM +0530, Dilip Kumar wrote: On Sat, Oct 17, 2020 at 11:34 AM Dilip Kumar wrote: On Tue, Oct 13, 2020 at 10:30 AM Dilip Kumar wrote: > > On Mon, Oct 12, 2020 at 7:32 PM Tomas Vondra > wrote: > > > > On Mon, Oct 12, 2020 at 02:28:43PM +0530, Dilip Kumar wrote:

Re: new heapcheck contrib module

2020-10-21 Thread Alvaro Herrera
On 2020-Oct-21, Robert Haas wrote: > On Wed, Oct 7, 2020 at 9:01 PM Mark Dilger > wrote: > > This next version, attached, has the acl checking and associated > > documentation changes split out into patch 0005, making it easier to review > > in isolation from the rest of the patch series. > >

Re: new heapcheck contrib module

2020-10-21 Thread Robert Haas
On Wed, Oct 7, 2020 at 9:01 PM Mark Dilger wrote: > This next version, attached, has the acl checking and associated > documentation changes split out into patch 0005, making it easier to review > in isolation from the rest of the patch series. > > Independently of acl considerations, this versi

Re: Deleting older versions in unique indexes to avoid page splits

2020-10-21 Thread Peter Geoghegan
On Wed, Oct 21, 2020 at 8:25 AM Robert Haas wrote: > That certainly isn't great. I mean, it might be not be too terrible, > because it's a leaf index page isn't nearly as potentially hot as a VM > page or a clog page, but it hurts interruptibility and risks hurting > concurrency, but if it were po

Re: dynamic result sets support in extended query protocol

2020-10-21 Thread Andres Freund
Hi, On 2020-10-20 20:17:45 -0400, Dave Cramer wrote: > You are correct we are not talking about a whole new protocol, but why not ? > Seems to me we would have a lot more latitude to get it right if we didn't > have this limitation. A new protocol will face a much bigger adoption hurdle, and ther

Re: Add header support to text format and matching feature

2020-10-21 Thread Daniel Verite
Rémi Lapeyre wrote: > It looks like this is not in the current commitfest and that Cabot does not > find it. I’m not yet accustomed to the PostgreSQL workflow, should I just > create a new entry in the current commitfest? Yes. Because in the last CommitFest it was marked as "Returned with

Re: Yet another fast GiST build

2020-10-21 Thread Pavel Borisov
> > > >> +return (ia->lower > ib->lower) ? 1 : -1; > >> +} > > > > We're only dealing with leaf items during index build, so the 'upper' > and 'lower' should always be equal here, right? Maybe add a comment and an > assertion on that. > > > > (It's pretty sad that the on-disk representation is

Re: Add header support to text format and matching feature

2020-10-21 Thread Rémi Lapeyre
It looks like this is not in the current commitfest and that Cabot does not find it. I’m not yet accustomed to the PostgreSQL workflow, should I just create a new entry in the current commitfest? Regards, Rémi > Le 13 oct. 2020 à 14:49, Rémi Lapeyre a écrit : > > Thanks Michael for taking c

Re: Yet another fast GiST build

2020-10-21 Thread Andrey Borodin
> 7 окт. 2020 г., в 17:38, Heikki Linnakangas написал(а): > > On 07/10/2020 15:27, Andrey Borodin wrote: >> Here's draft patch with implementation of sortsupport for ints and floats. > >> +static int >> +gbt_int4_cmp(Datum a, Datum b, SortSupport ssup) >> +{ >> +int32KEY *ia = (int32KEY

Re: CREATE TABLE .. PARTITION OF fails to preserve tgenabled for inherited row triggers

2020-10-21 Thread Alvaro Herrera
On 2020-Oct-21, Justin Pryzby wrote: > I came up with this, which probably needs more than a little finesse. Hmm, there are two important changes needed on this: 1) it must not emit CREATE lines for the child triggers; only the ALTER TABLE ONLY lines to set disable state on the partition are nee

Re: [PATCH] Add features to pg_stat_statements

2020-10-21 Thread Fujii Masao
On 2020/10/12 21:18, Yuki Seino wrote: The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed The patch applies

Re: refactoring basebackup.c

2020-10-21 Thread Mark Dilger
> On Jul 29, 2020, at 8:31 AM, Robert Haas wrote: > > On Fri, May 8, 2020 at 4:55 PM Robert Haas wrote: >> So it might be good if I'd remembered to attach the patches. Let's try >> that again. > > Here's an updated patch set. Hi Robert, v2-0001 through v2-0009 still apply cleanly, but v2-0

Re: Is Recovery actually paused?

2020-10-21 Thread Simon Riggs
On Wed, 21 Oct 2020 at 12:16, Dilip Kumar wrote: > > On Tue, Oct 20, 2020 at 5:59 PM Dilip Kumar wrote: > > > > On Tue, Oct 20, 2020 at 3:00 PM Simon Riggs wrote: > > > > > > On Tue, 20 Oct 2020 at 09:50, Dilip Kumar wrote: > > > > > > > > > Why would we want this? What problem are you trying t

Re: Timing of relcache inval at parallel worker init

2020-10-21 Thread Robert Haas
On Sat, Oct 17, 2020 at 7:53 AM Noah Misch wrote: > Let's move InvalidateSystemCaches() later. > Invalidation should follow any worker initialization step that changes the > results of relcache validation; otherwise, we'd need to ensure the > InvalidateSystemCaches() will not validate any relcache

[patch] [doc] Clarify temporary table name shadowing in CREATE TABLE

2020-10-21 Thread David G. Johnston
Hackers, Moving this over from -general [1] David J. [1] https://www.postgresql.org/message-id/CAKFQuwaM1K%3DprJNwKnoaC2AyDFn-7OvtCpmQ23bcVe5Z%3DLKA3Q%40mail.gmail.com diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 087cad184c..a400334092 100644 --- a/

Re: Deleting older versions in unique indexes to avoid page splits

2020-10-21 Thread Robert Haas
On Wed, Oct 7, 2020 at 7:48 PM Peter Geoghegan wrote: > To be blunt: It may be controversial that we're accessing multiple > heap pages while holding an exclusive lock on a leaf page, in the > hopes that we can avoid a page split, but without any certainty that > it'll work out. That certainly is

[patch] [doc] Minor variable related cleanup and rewording of plpgsql docs

2020-10-21 Thread David G. Johnston
Hackers, Bug # 16519 [1] is another report of confusion regarding trying to use parameters in improper locations - specifically the SET ROLE command within pl/pgsql. I'm re-attaching the doc patch and am adding it to the commitfest. David J. [1] https://www.postgresql.org/message-id/16519-9ef04

Re: [HACKERS] Custom compression methods

2020-10-21 Thread Robert Haas
On Thu, Oct 8, 2020 at 5:54 PM Tomas Vondra wrote: > And is the oidvector actually needed? If we have the extra catalog, > can't we track this simply using the regular dependencies? So we'd have > the attcompression OID of the current compression method, and the > preserved values would be tracked

[patch] [doc] Introduce view updating options more succinctly

2020-10-21 Thread David G. Johnston
Hackers, Over in -docs [1], where I attached the wrong patch anyway, the poster needed some clarity regarding view updating. A minor documentation patch is attached providing just that. David J. [1] https://www.postgresql.org/message-id/20200303174248.GB5019%40panix.com v1-doc-rules-view-upda

Re: Is Recovery actually paused?

2020-10-21 Thread Robert Haas
On Wed, Oct 21, 2020 at 7:16 AM Dilip Kumar wrote: > I have noticed one more issue, the problem is that if the recovery > process is currently not processing any WAL and just waiting for the > WAL to become available then the pg_is_wal_replay_paused will be stuck > forever. Having said that there

Re: [patch] [doc] Add SELECT clause literals to queries section headers

2020-10-21 Thread David G. Johnston
> > Hackers, > > Re-sending from -docs [1] with attachment in order to add to commitfest. > > David J. > > [1] > https://www.postgresql.org/message-id/flat/159981394174.31338.7014519396749859167%40wrigleys.postgresql.org > edit: attaching the patch v1-doc-add-select-literals-to-queries-section-h

[patch] [doc] Add SELECT clause literals to queries section headers

2020-10-21 Thread David G. Johnston
Hackers, Re-sending from -docs [1] with attachment in order to add to commitfest. David J. [1] https://www.postgresql.org/message-id/flat/159981394174.31338.7014519396749859167%40wrigleys.postgresql.org

Re: A new function to wait for the backend exit after termination

2020-10-21 Thread David G. Johnston
On Wed, Oct 21, 2020 at 6:13 AM Magnus Hagander wrote: > On Wed, Oct 21, 2020 at 3:02 PM Bharath Rupireddy < > bharath.rupireddyforpostg...@gmail.com> wrote: > >> Hi, >> >> Currently pg_terminate_backend(), sends SIGTERM to the backend process >> but doesn't ensure it's exit. There are chances th

Re: Combination of geqo and enable_partitionwise_join leads to crashes in the regression tests

2020-10-21 Thread Onder Kalaci
Hi, I think this is already discussed here: https://www.postgresql.org/message-id/flat/CAExHW5tgiLsYC_CLcaKHFFc8H56C0s9mCu_0OpahGxn%3DhUi_Pg%40mail.gmail.com#db54218ab7bb9e1484cdcc52abf2d324 Sorry for missing that thread before sending the mail. From: Onder Kalaci Date: Wednesday, 21 October

Combination of geqo and enable_partitionwise_join leads to crashes in the regression tests

2020-10-21 Thread Onder Kalaci
Hi, I was running “make installcheck” with the following settings: SET geqo_threshold=2; SET geqo_generations=1000; SETT geqo_pool_size=1000; SET enable_partitionwise_join to true; And, realized that “partition_join” test crashed. It is reproducible for both 12.3 and 13.0 (I’ve not tested furt

Re: libpq compression

2020-10-21 Thread Konstantin Knizhnik
On 06.10.2020 9:34, Andrey M. Borodin wrote: 26 марта 2019 г., в 19:46, Konstantin Knizhnik написал(а): Version of the patch correctly working when no compression algorithm are avaiable. Thanks for this work, Konstantin. PFA rebased version of this patch. This compression seems very impo

Re: A new function to wait for the backend exit after termination

2020-10-21 Thread Magnus Hagander
On Wed, Oct 21, 2020 at 3:02 PM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > Hi, > > Currently pg_terminate_backend(), sends SIGTERM to the backend process but > doesn't ensure it's exit. There are chances that backends still are > running(even after pg_terminate_backend()

A new function to wait for the backend exit after termination

2020-10-21 Thread Bharath Rupireddy
Hi, Currently pg_terminate_backend(), sends SIGTERM to the backend process but doesn't ensure it's exit. There are chances that backends still are running(even after pg_terminate_backend() is called) until the interrupts are processed(using ProcessInterrupts()). This could cause problems especiall

Re: CREATE TABLE .. PARTITION OF fails to preserve tgenabled for inherited row triggers

2020-10-21 Thread Justin Pryzby
On Tue, Oct 20, 2020 at 09:54:53PM -0300, Alvaro Herrera wrote: > On 2020-Oct-20, Justin Pryzby wrote: > > On Tue, Oct 20, 2020 at 03:56:30PM -0400, Tom Lane wrote: > > > Alvaro Herrera writes: > > > > Hmm, next question: should we backpatch a fix for this? (This applies > > > > all the way back

Re: pg_upgrade: fail early if a tablespace dir already exists for new cluster version

2020-10-21 Thread Justin Pryzby
On Tue, Oct 20, 2020 at 09:17:22PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > I wonder if pg_upgrade should try to rmdir() the tablespace dirs before > > restoring global objects, allowing it to succeed, rather than just "failing > > early". > > I'm a little confused about that. If the d

Re: Is Recovery actually paused?

2020-10-21 Thread Dilip Kumar
On Tue, Oct 20, 2020 at 5:59 PM Dilip Kumar wrote: > > On Tue, Oct 20, 2020 at 3:00 PM Simon Riggs wrote: > > > > On Tue, 20 Oct 2020 at 09:50, Dilip Kumar wrote: > > > > > > > Why would we want this? What problem are you trying to solve? > > > > > > > > The requirement is to know the last repla

Re: Parallel copy

2020-10-21 Thread Bharath Rupireddy
On Wed, Oct 21, 2020 at 3:18 PM Bharath Rupireddy wrote: > > 17. Remove extra lines after #define IsHeaderLine() > (cstate->header_line && cstate->cur_lineno == 1) in copy.h > I missed one comment: 18. I think we need to treat the number of parallel workers as an integer similar to the paralle

Re: Parallel copy

2020-10-21 Thread Amit Kapila
On Wed, Oct 21, 2020 at 3:19 PM Bharath Rupireddy wrote: > > > 9. Instead of calling CopyStringToSharedMemory() for each string > variable, can't we just create a linked list of all the strings that > need to be copied into shm and call CopyStringToSharedMemory() only > once? We could avoid 5 func

Re: Online verification of checksums

2020-10-21 Thread Michael Paquier
On Wed, Oct 21, 2020 at 12:00:23PM +0200, Michael Banck wrote: > The check was ported (or the concept of it adapted) from pgBackRest if I > remember correctly. Okay, I did not know that. >> As things stand, I'd like to think that it would be much more useful >> to remove this check and to have on

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-21 Thread Amit Kapila
On Wed, Oct 21, 2020 at 3:03 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > So what's your opinion? > > * Global visibility > This is what Amit-san suggested some times -- "design it before reviewing the > current patch." I'm a bit optimistic about this and think this FDW 2

Re: Online verification of checksums

2020-10-21 Thread Michael Banck
Hi, Am Dienstag, den 20.10.2020, 18:11 +0900 schrieb Michael Paquier: > On Mon, Apr 06, 2020 at 04:45:44PM -0400, Tom Lane wrote: > > Actually, after thinking about that a bit more: why is there an LSN-based > > special condition at all? It seems like it'd be far more useful to > > checksum every

Re: Parallel copy

2020-10-21 Thread Bharath Rupireddy
Hi Vignesh, I took a look at the v8 patch set. Here are some comments: 1. PopulateCommonCstateInfo() -- can we use PopulateCommonCStateInfo() or PopulateCopyStateInfo()? And also EstimateCstateSize() -- EstimateCStateSize(), PopulateCstateCatalogInfo() -- PopulateCStateCatalogInfo()? 2. Instead

RE: Transactions involving multiple postgres foreign servers, take 2

2020-10-21 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > So what's your opinion? My opinion is simple and has not changed. Let's clarify and refine the design first in the following areas (others may have pointed out something else too, but I don't remember), before going deeper into the code review. * FDW interface New func

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-21 Thread Amit Kapila
On Tue, Oct 20, 2020 at 9:46 AM Peter Smith wrote: > > > == > Patch v10-0002, File: src/backend/replication/logical/reorderbuffer.c > == > > COMMENT > There are some parts of the code where in my v6 review I had a doubt > about the mutually exclusive treatment of the "streaming" fl

Re: Add statistics to pg_stat_wal view for wal related parameter tuning

2020-10-21 Thread Kyotaro Horiguchi
At Tue, 20 Oct 2020 16:11:29 +0900, Masahiro Ikeda wrote in > On 2020-10-20 12:46, Amit Kapila wrote: > > I see that we also need to add extra code to capture these stats (some > > of which is in performance-critical path especially in > > XLogInsertRecord) which again makes me a bit uncomfortab

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-21 Thread Amit Kapila
On Wed, Oct 21, 2020 at 1:38 PM Peter Smith wrote: > > The PG docs for PREPARE TRANSACTION [1] don't say anything about an > empty (zero length) transaction-id. > e.g. PREPARE TRANSACTION ''; > [1] https://www.postgresql.org/docs/current/sql-prepare-transaction.html > > ~ > > Meanwhile, during tes

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-21 Thread Amit Kapila
On Tue, Oct 20, 2020 at 4:32 PM Ajin Cherian wrote: > > On Fri, Oct 16, 2020 at 5:21 PM Peter Smith wrote: > > > > Comments: > > src/backend/replication/logical/worker.c > @@ -888,6 +888,319 @@ apply_handle_prepare(StringInfo s) > + /* > + * FIXME - Following condition was in apply_handle_prepare

Re: Parallel copy

2020-10-21 Thread vignesh C
On Thu, Oct 8, 2020 at 11:15 AM vignesh C wrote: > > I'm summarizing the pending open points so that I don't miss anything: > 1) Performance test on latest patch set. It is tested and results are shared by bharath at [1] > 2) Testing points suggested. Tests are added as suggested and details sh

RE: Transactions involving multiple postgres foreign servers, take 2

2020-10-21 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > > if (PQstatus(entry->conn) != CONNECTION_OK || > > PQtransactionStatus(entry->conn) != PQTRANS_IDLE || > > entry->changing_xact_state) > > { > > elog(DEBUG3, "discarding connection %p", entry->conn); > > dis

  1   2   >