Re: Asynchronous Append on postgres_fdw nodes.

2021-02-14 Thread Etsuro Fujita
On Fri, Feb 12, 2021 at 5:30 PM Kyotaro Horiguchi wrote: > It seems too specific to async Append so I look it as a PoC of the > mechanism. Are you saying that the patch only reconsiders async ForeignScans? > It creates a hash table that keyed by connection umid to record > planids run on the con

Re: pg_cryptohash_final possible out-of-bounds access (per Coverity)

2021-02-14 Thread Michael Paquier
On Sat, Feb 13, 2021 at 09:33:48PM -0300, Ranier Vilela wrote: > Em sáb., 13 de fev. de 2021 às 20:32, Michael Paquier > escreveu: > >> You are missing the point here. What you are proposing here would not >> be backpatched. However, reusing the same words as upthread, this has >> a cost in ter

Re: How to get Relation tuples in C function

2021-02-14 Thread Michael Paquier
On Sun, Feb 14, 2021 at 09:29:08AM +0800, Andy Fan wrote: > Thank you tom for the reply. What would be the difference between the > SPI and "write a pure SQL UDF" and call it with DirectFunctionCall1? I > just ran into a similar situation some days before. Currently I think > DirectFunctionCall1

Re: Some regular-expression performance hacking

2021-02-14 Thread Joel Jacobson
On Sat, Feb 13, 2021, at 22:11, Tom Lane wrote: >0001-invent-rainbow-arcs-2.patch >0002-recognize-matchall-NFAs-2.patch I've successfully tested both patches against the 1.5M regexes-in-the-wild dataset. Out of the 1489489 (pattern, text string) pairs tested, there was only one single deviation:

Re: pg_cryptohash_final possible out-of-bounds access (per Coverity)

2021-02-14 Thread Ranier Vilela
Em dom., 14 de fev. de 2021 às 08:22, Michael Paquier escreveu: > On Sat, Feb 13, 2021 at 09:33:48PM -0300, Ranier Vilela wrote: > > Em sáb., 13 de fev. de 2021 às 20:32, Michael Paquier < > mich...@paquier.xyz> > > escreveu: > > > >> You are missing the point here. What you are proposing here w

Re: Some regular-expression performance hacking

2021-02-14 Thread Tom Lane
"Joel Jacobson" writes: > I've successfully tested both patches against the 1.5M regexes-in-the-wild > dataset. > Out of the 1489489 (pattern, text string) pairs tested, > there was only one single deviation: > This 100577 bytes big regex (pattern_id = 207811)... > ... > ...previously raised... >

Re: Extensibility of the PostgreSQL wire protocol

2021-02-14 Thread Dave Cramer
On Thu, 11 Feb 2021 at 09:28, Robert Haas wrote: > On Wed, Feb 10, 2021 at 2:04 PM Tom Lane wrote: > > That is a spot-on definition of where I do NOT want to end up. Hooks > > everywhere and enormous extensions that break anytime we change anything > > in the core. It's not really clear that a

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-14 Thread Peter Geoghegan
On Sat, Feb 6, 2021 at 7:40 PM Andres Freund wrote: > Looks like a mistake on my part... Probably a rename regex that somehow > went wrong - I went back and forth on those names way too many > times. Want me to push the fix? Spotted another one: Shouldn't ReadNextFullTransactionId() actually be c

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-14 Thread Justin Pryzby
On Tue, Feb 09, 2021 at 04:35:03AM +, tsunakawa.ta...@fujitsu.com wrote: > Rebased to HEAD with the following modifications. It passes make check in > the top directory and contrib/postgres_fdw. > That said, with the reviews from some people and good performance results, I > think this can b

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-14 Thread Thomas Munro
On Mon, Feb 15, 2021 at 10:02 AM Peter Geoghegan wrote: > On Sat, Feb 6, 2021 at 7:40 PM Andres Freund wrote: > > Looks like a mistake on my part... Probably a rename regex that somehow > > went wrong - I went back and forth on those names way too many > > times. Want me to push the fix? > > Spot

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-14 Thread Peter Geoghegan
On Sun, Feb 14, 2021 at 2:08 PM Thomas Munro wrote: > I prefer "next", because that's in the name of the variable it reads, > and the variable name seemed to me to have a more obvious meaning. > That's why I went for that name in commit 2fc7af5e966. I do agree > that it's slightly strange that th

Re: WIP: WAL prefetch (another approach)

2021-02-14 Thread Tomas Vondra
On 2/13/21 10:39 PM, Stephen Frost wrote: Greetings, * Andres Freund (and...@anarazel.de) wrote: On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote: Yeah, that's a good point. I think it'd make sense to keep track of recent FPIs and skip prefetching such blocks. But how exactly should we impl

Re: WIP: WAL prefetch (another approach)

2021-02-14 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 2/13/21 10:39 PM, Stephen Frost wrote: > >* Andres Freund (and...@anarazel.de) wrote: > >>On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote: > >>>Yeah, that's a good point. I think it'd make sense to keep track of recent > >>>FPI

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-14 Thread Thomas Munro
On Mon, Feb 15, 2021 at 11:33 AM Peter Geoghegan wrote: > On Sun, Feb 14, 2021 at 2:08 PM Thomas Munro wrote: > > I prefer "next", because that's in the name of the variable it reads, > > and the variable name seemed to me to have a more obvious meaning. > > That's why I went for that name in com

Re: Preventing free space from being reused

2021-02-14 Thread Noah Bergbauer
>I would suggest to take a look at the BRIN opclass multi-minmax currently in development. Thank you, this does look like it could help a lot with BRIN performance in this situation! But again, if index performance alone was the only issue, then I would simply accept the space overhead and switch

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-14 Thread Peter Geoghegan
On Sun, Feb 14, 2021 at 4:21 PM Thomas Munro wrote: > Done. Thanks. -- Peter Geoghegan

GCC warning in back branches

2021-02-14 Thread Thomas Munro
Hi, guc.c: In function ‘RestoreGUCState’: guc.c:9455:4: error: ‘varsourceline’ may be used uninitialized in this function [-Werror=maybe-uninitialized] 9455 |set_config_sourcefile(varname, varsourcefile, varsourceline); |^~~~

RE: [PoC] Non-volatile WAL buffer

2021-02-14 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > I've done some performance benchmarks with the master and NTT v4 > patch. Let me share the results. > ... > master NTT master-unlogged > 32 113209 67107 154298 > 64 144880 54289 178883 > 96 151405 50562 180018 > > "master-unlogged" is

Re: doing something about the broken dynloader.h symlink

2021-02-14 Thread Thomas Munro
On Fri, Jun 19, 2020 at 11:38 PM Julien Rouhaud wrote: > On Fri, Jun 19, 2020 at 12:08 PM Thomas Munro wrote: > > On Fri, Jun 19, 2020 at 8:02 PM Peter Eisentraut > > wrote: > > > +[# Remove links created by old versions of configure, so that there > > > +# are no broken symlinks in the tree > >

Re: pg_cryptohash_final possible out-of-bounds access (per Coverity)

2021-02-14 Thread Michael Paquier
On Sun, Feb 14, 2021 at 11:39:47AM -0300, Ranier Vilela wrote: > What do you think? That's not a good idea for two reasons: 1) There is CRC32 to worry about, which relies on a different logic. 2) It would become easier to miss the new option as compilation would not warn anymore if a new checksum

Re: shared tempfile was not removed on statement_timeout

2021-02-14 Thread Thomas Munro
On Fri, Feb 5, 2021 at 5:47 PM Thomas Munro wrote: > On Sun, Jan 31, 2021 at 6:07 PM Tom Lane wrote: > > Thomas Munro writes: > > > So that gives a very simple back-patchable patch. > > > > Hmm, so is the *rest* of that function perfectly okay with being > > interrupted? > > It looks OK to me.

Re: GCC warning in back branches

2021-02-14 Thread Michael Paquier
On Mon, Feb 15, 2021 at 02:15:51PM +1300, Thomas Munro wrote: > guc.c: In function ‘RestoreGUCState’: > guc.c:9455:4: error: ‘varsourceline’ may be used uninitialized in this > function [-Werror=maybe-uninitialized] > 9455 |set_config_sourcefile(varname, varsourcefile, varsourceline); >

Re: GCC warning in back branches

2021-02-14 Thread Tom Lane
Michael Paquier writes: > On Mon, Feb 15, 2021 at 02:15:51PM +1300, Thomas Munro wrote: >> I propose the attached. > We usually don't bother much about compilation warnings in stable > branches as long as they are not real bugs, and these are the oldest > stable ones. So why here? I would have

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2021-02-14 Thread Justin Pryzby
On Thu, Feb 04, 2021 at 03:38:39PM +0900, Michael Paquier wrote: > On Wed, Feb 03, 2021 at 07:54:42PM +0900, Michael Paquier wrote: > > Not sure I like that. Here is a proposal: > > "it is recommended to separately use ALTER TABLE ONLY on them so as > > any new partitions attached inherit the new

Re: Use pg_pwrite() in pg_test_fsync

2021-02-14 Thread Thomas Munro
On Sun, Jan 24, 2021 at 1:50 PM Thomas Munro wrote: > On Sun, Jan 10, 2021 at 9:21 AM Thomas Munro wrote: > > I left the fsync-after-closing and non-sync'd tests using write(), > > because they weren't using lseek(). The latter case is arguably a bit > > odd because it's not overwriting pre-allo

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-02-14 Thread Masahiro Ikeda
On 2021-02-08 13:01, Fujii Masao wrote: On 2021/02/05 8:45, Masahiro Ikeda wrote: I pgindented the patches. Thanks for updating the patches! Thanks for checking the patches. + XLogWrite, which nomally called by an + issue_xlog_fsync, which nomally called by an Typo: "nomally

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-02-14 Thread Masahiro Ikeda
On 2021-02-08 14:26, Fujii Masao wrote: On 2021/02/08 13:01, Fujii Masao wrote: On 2021/02/05 8:45, Masahiro Ikeda wrote: I pgindented the patches. Thanks for updating the patches! +   XLogWrite, which nomally called by an +   issue_xlog_fsync, which nomally called by an Typo: "

Snapshot scalability patch issue

2021-02-14 Thread Peter Geoghegan
The call to heap_page_prune() within lazy_scan_heap() passes a bool literal ('false') as its fourth argument. But the fourth argument is of type TransactionId, not bool. This has been the case since the snapshot scalability work performed by commit dc7420c2c92. Surely something is amiss here. I al

Re: Support ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION ... syntax

2021-02-14 Thread Bharath Rupireddy
On Sat, Feb 13, 2021 at 11:41 AM japin wrote: > > IIUC, with the current patch, the new ALTER SUBSCRIPTION ... ADD/DROP > > errors out on the first publication that already exists/that doesn't > > exist right? What if there are multiple publications given in the > > ADD/DROP list, and few of them

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2021-02-14 Thread Michael Paquier
On Sun, Feb 14, 2021 at 08:10:50PM -0600, Justin Pryzby wrote: > Isn't this dead code ? Nope, it's not dead. Those two code paths can be hit when attempting a reidex with a tablespace move directly on toast tables and indexes, see: =# create table aa (a text); CREATE TABLE =# select relname from

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-02-14 Thread Masahiro Ikeda
On 2021-02-10 00:51, David G. Johnston wrote: On Thu, Feb 4, 2021 at 4:45 PM Masahiro Ikeda wrote: I pgindented the patches. ... XLogWrite, which is invoked during an XLogFlush request (see ...). This is also incremented by the WAL receiver during replication. ("which normally called" shou

Re: Some regular-expression performance hacking

2021-02-14 Thread Tom Lane
"Joel Jacobson" writes: > Below is the result of the performance test query: > -- 8facf1ea00b7a0c08c755a0392212b83e04ae28a: > Time: 592196.145 ms (09:52.196) > -- 8facf1ea00b7a0c08c755a0392212b83e04ae28a+patches: > Time: 461739.364 ms (07:41.739) > That's an impressive 22% speed-up! I've been doi

Re: Default wal_sync_method on FreeBSD

2021-02-14 Thread Thomas Munro
On Fri, Jan 8, 2021 at 2:19 PM Thomas Munro wrote: > The system header change has one interesting consequence for existing > releases of PostgreSQL, though: xlogdefs.h now sees that there is an > O_DSYNC macro that is distinct from O_SYNC, and defaults to > wal_sync_method=open_datasync. That's n

Re: repeated decoding of prepared transactions

2021-02-14 Thread Amit Kapila
On Sat, Feb 13, 2021 at 10:23 PM Andres Freund wrote: > > On 2021-02-13 17:37:29 +0100, Petr Jelinek wrote: > > > AFAIK this is exactly why origins are Wal logged along with > > transaction, it allows us to guarantee never getting anything that has > > beed durably written. > > I think you'd need

Re: 64-bit XIDs in deleted nbtree pages

2021-02-14 Thread Peter Geoghegan
On Sat, Feb 13, 2021 at 10:47 PM Peter Geoghegan wrote: > It will be rare. But more importantly, the fact that scenario is now > an extreme case justifies treating it as an extreme case. We can teach > _bt_vacuum_needs_cleanup() to recognize it as an extreme case, too. In > particular, I think tha

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-14 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > This is crashing during fdw check. > http://cfbot.cputube.org/andrey-lepikhov.html > > Maybe it's related to this patch: > |commit 6214e2b2280462cbc3aa1986e350e167651b3905 > |Fix permission checks on constraint violation errors on partitions. > |Security: CVE-2021-33

Re: GCC warning in back branches

2021-02-14 Thread Thomas Munro
On Mon, Feb 15, 2021 at 2:35 PM Michael Paquier wrote: > ... I would have patched the top of the > function if it were me, btw. I just copied the way it is coded in master (due to commit fbb2e9a0 which fixed this warning in 11+).

Re: PG vs LLVM 12 on seawasp, next round

2021-02-14 Thread Thomas Munro
On Sat, Dec 12, 2020 at 8:45 AM Fabien COELHO wrote: > > +ERROR: could not load library > > "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": > > libLLVMOrcJIT.so.12git: cannot open shared object fi

Re: adding wait_start column to pg_locks

2021-02-14 Thread Fujii Masao
On 2021/02/10 10:43, Fujii Masao wrote: On 2021/02/09 23:31, torikoshia wrote: On 2021-02-09 22:54, Fujii Masao wrote: On 2021/02/09 19:11, Fujii Masao wrote: On 2021/02/09 18:13, Fujii Masao wrote: On 2021/02/09 17:48, torikoshia wrote: On 2021-02-05 18:49, Fujii Masao wrote: On 2

Re: logical replication seems broken

2021-02-14 Thread vignesh C
On Sat, Feb 13, 2021 at 5:58 PM Erik Rijkers wrote: > > > On 02/13/2021 11:49 AM Amit Kapila wrote: > > > > On Fri, Feb 12, 2021 at 10:00 PM wrote: > > > > > > > On 02/12/2021 1:51 PM Amit Kapila wrote: > > > > > > > > On Fri, Feb 12, 2021 at 6:04 PM Erik Rijkers wrote: > > > > > > > > > > I a

Re: [DOC] add missing "[ NO ]" to various "DEPENDS ON" synopses

2021-02-14 Thread Ian Lawrence Barwick
2021年2月13日(土) 11:52 Michael Paquier : > On Fri, Feb 12, 2021 at 10:32:14AM +0900, Ian Lawrence Barwick wrote: > > In the documentation, the "[ NO ]" option is listed in the synopsis for > > ALTER TRIGGER and ALTER FUNCTION, but not the others. > > Trivial patch attached. > > There are two flavors

Re: [DOC] add missing "[ NO ]" to various "DEPENDS ON" synopses

2021-02-14 Thread Ian Lawrence Barwick
2021年2月13日(土) 11:52 Michael Paquier : > On Fri, Feb 12, 2021 at 10:32:14AM +0900, Ian Lawrence Barwick wrote: > > In the documentation, the "[ NO ]" option is listed in the synopsis for > > ALTER TRIGGER and ALTER FUNCTION, but not the others. > > Trivial patch attached. > > There are two flavors

Re: a misbehavior of partition row movement (?)

2021-02-14 Thread Masahiko Sawada
On Wed, Jan 20, 2021 at 7:04 PM Amit Langote wrote: > > On Wed, Jan 20, 2021 at 4:13 PM Peter Eisentraut > wrote: > > On 2021-01-08 09:54, Amit Langote wrote: > > >>> I don't quite recall if the decision to implement it like this was > > >>> based on assuming that this is what users would like to

Re: Fallback table AM for relkinds without storage

2021-02-14 Thread Michael Paquier
On Tue, Feb 09, 2021 at 04:27:34PM +0900, Michael Paquier wrote: > Putting sanity checks within all the table_* functions of tableam.h > would not be a good idea, as nothing prevents the call of what's > stored in rel->rd_tableam. I have been playing with this idea, and finished with the attache

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-14 Thread Greg Nancarrow
On Sat, Feb 13, 2021 at 12:17 AM Amit Langote wrote: > > On Thu, Feb 11, 2021 at 4:43 PM Greg Nancarrow wrote: > > On Thu, Feb 11, 2021 at 5:33 PM Greg Nancarrow wrote: > > > On Tue, Feb 9, 2021 at 1:04 AM Amit Langote > > > wrote: > > > > > > > > * I think that the concerns raised by Tsunakaw