Re: Release SPI plans for referential integrity with DISCARD ALL

2021-01-18 Thread yuzuko
Hi Corey, Thank you for sharing. > Amit's patch is now available in this thread [1]. I'm curious if it has any > effect on your memory pressure issue. > I just found that thread. I'll check the patch. -- Best regards, Yuzuko Hosoya NTT Open Source Software Center

Re: simplifying foreign key/RI checks

2021-01-18 Thread Amit Langote
On Tue, Jan 19, 2021 at 12:00 PM Zhihong Yu wrote: > + if (mapped_partkey_attnums[i] == pk_attnums[j]) > + { > + partkey_vals[i] = pk_vals[j]; > + partkey_isnull[i] = pk_nulls[j] == 'n' ? true : false; > + i++; > +

Re: simplifying foreign key/RI checks

2021-01-18 Thread Amit Langote
On Tue, Jan 19, 2021 at 3:46 PM Corey Huinker wrote: > v2 patch applies and passes make check and make check-world. Perhaps, given > the missing break at line 418 without any tests failing, we could add another > regression test if we're into 100% code path coverage. As it is, I think the > com

Re: search_plan_tree(): handling of non-leaf CustomScanState nodes causes segfault

2021-01-18 Thread David Geier
Hi, On 18.01.21 23:42, Tom Lane wrote: David Geier writes: On 18.01.21 19:46, Tom Lane wrote: Hm. I agree that we shouldn't simply assume that ss_currentRelation isn't null. However, we cannot make search_plan_tree() descend through non-leaf CustomScan nodes, because we don't know what proc

RE: Parallel INSERT (INTO ... SELECT ...)

2021-01-18 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > Tsunakawa-San, does this address your concerns around locking the > target relation in the required cases? It would be good to test but I > don't see any problems in the scenarios you mentioned. Thank you, understood. RevalidateCachedQuery() does parse analysis, that's the t

Re: Release SPI plans for referential integrity with DISCARD ALL

2021-01-18 Thread Corey Huinker
On Wed, Jan 13, 2021 at 1:03 PM Corey Huinker wrote: > In addition to that, a following case would be solved with this approach: >> When many processes are referencing many tables defined foreign key >> constraints thoroughly, a huge amount of memory will be consumed >> regardless of whether refe

Re: ResourceOwner refactoring

2021-01-18 Thread Michael Paquier
On Mon, Jan 18, 2021 at 02:22:33PM +0200, Heikki Linnakangas wrote: > diff --git a/src/common/cryptohash_openssl.c b/src/common/cryptohash_openssl.c > index 551ec392b60..642e72d8c0f 100644 > --- a/src/common/cryptohash_openssl.c > +++ b/src/common/cryptohash_openssl.c [...] > +/* ResourceOwner call

Re: simplifying foreign key/RI checks

2021-01-18 Thread Corey Huinker
On Mon, Jan 18, 2021 at 9:45 PM Amit Langote wrote: > On Tue, Jan 19, 2021 at 2:47 AM Zhihong Yu wrote: > > > > Hi, > > I was looking at this statement: > > > > insert into f select generate_series(1, 200, 2); > > > > Since certain generated values (the second half) are not in table p, > wou

Re: [PATCH] ProcessInterrupts_hook

2021-01-18 Thread Craig Ringer
On Tue, 19 Jan 2021 at 12:44, Craig Ringer wrote: > > > We're about halfway there already, see 7e784d1dc. I didn't do the >> > other half because it wasn't necessary to the problem, but exposing >> > the shutdown state more fully seems reasonable. >> > > Excellent, I'll take a look. Thanks. > T

Re: Is Recovery actually paused?

2021-01-18 Thread Dilip Kumar
On Tue, Jan 19, 2021 at 10:30 AM Kyotaro Horiguchi wrote: > > At Tue, 19 Jan 2021 11:41:18 +0900, Yugo NAGATA wrote in > > On Sun, 17 Jan 2021 11:33:52 +0530 > > Dilip Kumar wrote: > > > > > > > > > > > > > > I think the current pg_is_wal_replay_paused() already has another > > > > > > > purpos

Re: Additional Chapter for Tutorial - arch-dev.sgml

2021-01-18 Thread Jürgen Purtz
On 18.01.21 15:13, Heikki Linnakangas wrote: On 20/11/2020 23:52, Erik Rijkers wrote: (smallish) Changes to arch-dev.sgml This looks good to me. One little complaint: @@ -125,7 +122,7 @@ use a supervisor process (also master process) that spawns a new server process every time

RE: Global snapshots

2021-01-18 Thread tsunakawa.ta...@fujitsu.com
Hello, Andrey-san, all, Based on the request at HighGo's sharding meeting, I'm re-sending the information on Commitment Ordering that could be used for global visibility. Their patents have already expired. -- Have anyone examined the followin

Re: simplifying foreign key/RI checks

2021-01-18 Thread Amit Langote
On Tue, Jan 19, 2021 at 2:56 PM Corey Huinker wrote: >> In file included from >> /home/japin/Codes/postgresql/Debug/../src/include/postgres.h:47:0, >> from >> /home/japin/Codes/postgresql/Debug/../src/backend/utils/adt/ri_triggers.c:24: >> /home/japin/Codes/postgresql/Debug/../s

Re: POC: postgres_fdw insert batching

2021-01-18 Thread Amit Langote
On Tue, Jan 19, 2021 at 2:06 PM tsunakawa.ta...@fujitsu.com wrote: > From: Amit Langote > > I apologize in advance for being maybe overly pedantic, but I noticed > > that, in ExecInitModifyTable(), you decided to place the call outside > > the loop that goes over resultRelations (shown below), al

How to expose session vs txn lock info in pg_locks view?

2021-01-18 Thread Craig Ringer
Presently there doesn't seem to be a way to tell whether a lock is session-level or transaction-level in the pg_locks view. I was expecting this to be a quick patch, but the comment on the definition of PROCLOCKTAG in lock.h notes that shmem state for heavyweight locks does not track whether the l

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-18 Thread Amit Kapila
On Tue, Jan 19, 2021 at 10:32 AM Greg Nancarrow wrote: > > On Tue, Jan 19, 2021 at 1:37 PM tsunakawa.ta...@fujitsu.com > wrote: > > > > > The table has ALREADY been locked (by the caller) during the > > > parse/analyze phase. > > > > Isn't there any case where planning is done but parse analysis

Re: [PATCH 1/1] Initial mach based shared memory support.

2021-01-18 Thread Tom Lane
James Hilliard writes: > from my understanding due to mach semantics a number of the sanity checks > we are doing for sysv shmem are probably unnecessary when using mach > API's due to the mach port task bindings(mach_task_self()) effectively > ensuring our maps are already task bound and won't co

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2021-01-18 Thread Fujii Masao
On 2021/01/19 9:53, Hou, Zhijie wrote: +1 to add it after "dropped (Note )", how about as follows with slight changes? dropped (Note that server name of an invalid connection can be NULL if the server is dropped), and then such . Yes, I like this one. One question is; "should" o

Re: CheckpointLock needed in CreateCheckPoint()?

2021-01-18 Thread Michael Paquier
On Mon, Jan 18, 2021 at 02:36:48PM -0500, Robert Haas wrote: > Here's a patch to remove CheckpointLock completely. For > ProcessInterrupts() to do anything, one of the following things would > have to be true: > > [...] > > So I don't see any problem with just ripping this out entirely, but > I'd

Re: simplifying foreign key/RI checks

2021-01-18 Thread Corey Huinker
> > > In file included from > /home/japin/Codes/postgresql/Debug/../src/include/postgres.h:47:0, > from > /home/japin/Codes/postgresql/Debug/../src/backend/utils/adt/ri_triggers.c:24: > /home/japin/Codes/postgresql/Debug/../src/backend/utils/adt/ri_triggers.c: > In function ‘ri_Pri

Re: [patch] Help information for pg_dump

2021-01-18 Thread Bharath Rupireddy
On Tue, Jan 19, 2021 at 9:07 AM Zhang, Jie wrote: > > Hi all > > After executing command [pg_dump -?], some help information is as follows. > > pg_dump -? > - > -N, --exclude-schema=PATTERN do NOT dump the specified schema(s)

Re: [PATCH 1/1] Initial mach based shared memory support.

2021-01-18 Thread James Hilliard
On Mon, Jan 18, 2021 at 5:29 PM Tom Lane wrote: > > James Hilliard writes: > > OSX implements sysv shmem support via a mach wrapper, however the mach > > sysv shmem wrapper has some severe restrictions that prevent us from > > allocating enough memory segments in some cases. > > ... > > In order

Re: Odd, intermittent failure in contrib/pageinspect

2021-01-18 Thread Michael Paquier
On Mon, Jan 18, 2021 at 05:47:40PM -0500, Tom Lane wrote: > Right, then we don't need any strange theories about autovacuum, > just bad timing luck. whelk does seem pretty slow, so it's not > much of a stretch to imagine that it's more susceptible to this > corner case than faster machines. > > So

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-18 Thread Amit Kapila
On Tue, Jan 19, 2021 at 9:19 AM Greg Nancarrow wrote: > > On Tue, Jan 19, 2021 at 2:03 PM Amit Kapila wrote: > > > > > > > > > > You have not raised a WARNING for the second case. > > > > > > The same checks in current Postgres code also don't raise a WARNING > > > for that case, so I'm just bein

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-18 Thread Amit Kapila
On Mon, Jan 18, 2021 at 3:50 PM Greg Nancarrow wrote: > > On Mon, Jan 18, 2021 at 8:10 PM Amit Kapila wrote: > > > > > 1) > > > > > > >Here, it seems we are accessing the relation descriptor without any > > > >lock on the table which is dangerous considering the table can be > > > >modified in a

RE: POC: postgres_fdw insert batching

2021-01-18 Thread tsunakawa.ta...@fujitsu.com
From: Amit Langote > I apologize in advance for being maybe overly pedantic, but I noticed > that, in ExecInitModifyTable(), you decided to place the call outside > the loop that goes over resultRelations (shown below), although my > intent was to ask to place it next to the BeginForeignModify() i

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-18 Thread Greg Nancarrow
On Tue, Jan 19, 2021 at 1:37 PM tsunakawa.ta...@fujitsu.com wrote: > > > The table has ALREADY been locked (by the caller) during the > > parse/analyze phase. > > Isn't there any case where planning is done but parse analysis is not done > immediately before? e.g. > > * Alteration of objects inv

Re: Is Recovery actually paused?

2021-01-18 Thread Kyotaro Horiguchi
At Tue, 19 Jan 2021 11:41:18 +0900, Yugo NAGATA wrote in > On Sun, 17 Jan 2021 11:33:52 +0530 > Dilip Kumar wrote: > > > > > > > > > > > > I think the current pg_is_wal_replay_paused() already has another > > > > > > purpose; > > > > > > this waits recovery to actually get paused. If we want to

Re: POC: postgres_fdw insert batching

2021-01-18 Thread Amit Langote
Tsunakawa-san, On Tue, Jan 19, 2021 at 12:50 PM tsunakawa.ta...@fujitsu.com wrote: > From: Tomas Vondra > > OK. Can you prepare a final patch, squashing all the commits into a > > single one, and perhaps use the function in create_foreign_modify? > > Attached, including the message fix pointed b

Re: Wrong usage of RelationNeedsWAL

2021-01-18 Thread Kyotaro Horiguchi
At Mon, 18 Jan 2021 17:30:22 +0900 (JST), Kyotaro Horiguchi wrote in > At Sun, 17 Jan 2021 23:02:18 -0800, Noah Misch wrote in > > On Sun, Jan 17, 2021 at 10:36:31PM -0800, Noah Misch wrote: > > > I wrote the above based on the "PageGetLSN(page) > (snapshot)->lsn" check > > > in > > > TestFor

Re: [PATCH] ProcessInterrupts_hook

2021-01-18 Thread Craig Ringer
On Tue, 19 Jan 2021, 02:01 Robert Haas, wrote: > On Mon, Jan 18, 2021 at 11:56 AM Tom Lane wrote: > > > I've wanted this in the past, too, so +1 from me. > > > > I dunno, this seems pretty scary and easily abusable. There's not all > > that much that can be done safely in ProcessInterrupts(), a

Re: simplifying foreign key/RI checks

2021-01-18 Thread Pavel Stehule
út 19. 1. 2021 v 3:08 odesílatel Amit Langote napsal: > On Tue, Jan 19, 2021 at 3:01 AM Pavel Stehule > wrote: > > po 18. 1. 2021 v 13:40 odesílatel Amit Langote > napsal: > >> I started with the check that's performed when inserting into or > >> updating the referencing table to confirm that t

RE: POC: postgres_fdw insert batching

2021-01-18 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > OK. Can you prepare a final patch, squashing all the commits into a > single one, and perhaps use the function in create_foreign_modify? Attached, including the message fix pointed by Zaihong-san. Regards Takayuki Tsunakawa v12-0001-Add-bulk-insert-for-foreign-tables.pat

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-18 Thread Greg Nancarrow
On Tue, Jan 19, 2021 at 2:03 PM Amit Kapila wrote: > > > > > > > You have not raised a WARNING for the second case. > > > > The same checks in current Postgres code also don't raise a WARNING > > for that case, so I'm just being consistent with existing Postgres > > code (which itself isn't consis

回复:Re: Cache relation sizes?

2021-01-18 Thread 陈佳昕(步真)
Hi Thomas I want to share a patch with you, I change the replacement algorithm from fifo to a simple lru. Buzhen 0001-update-fifo-to-lru-to-sweep-a-valid-cache.patch Description: Binary data

[patch] Help information for pg_dump

2021-01-18 Thread Zhang, Jie
Hi all After executing command [pg_dump -?], some help information is as follows. pg_dump -? - -N, --exclude-schema=PATTERN do NOT dump the specified schema(s) ※ -T, --exclude-table=PATTERN do NOT dump the specified tab

Re: simplifying foreign key/RI checks

2021-01-18 Thread japin
On Tue, 19 Jan 2021 at 10:45, Amit Langote wrote: > On Tue, Jan 19, 2021 at 2:47 AM Zhihong Yu wrote: >> >> Hi, >> I was looking at this statement: >> >> insert into f select generate_series(1, 200, 2); >> >> Since certain generated values (the second half) are not in table p, >> wouldn't

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2021-01-18 Thread Bharath Rupireddy
On Mon, Jan 18, 2021 at 9:11 PM Fujii Masao wrote: > > Attaching v12 patch set. 0001 is for postgres_fdw_disconnect() > > function, 0002 is for keep_connections GUC and 0003 is for > > keep_connection server level option. > > Thanks! > > > > > Please review it further. > > + server =

Re: Is Recovery actually paused?

2021-01-18 Thread Dilip Kumar
On Tue, 19 Jan 2021 at 8:12 AM, Yugo NAGATA wrote: > On Sun, 17 Jan 2021 11:33:52 +0530 > Dilip Kumar wrote: > > > On Thu, Jan 14, 2021 at 6:49 PM Yugo NAGATA wrote: > > > > > > On Wed, 13 Jan 2021 17:49:43 +0530 > > > Dilip Kumar wrote: > > > > > > > On Wed, Jan 13, 2021 at 3:35 PM Dilip Kuma

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-18 Thread Amit Kapila
On Mon, Jan 18, 2021 at 3:50 PM Greg Nancarrow wrote: > > On Mon, Jan 18, 2021 at 8:10 PM Amit Kapila wrote: > > > > > >It is not clear why the above two are shouldn't happen cases and if so > > > >why we want to treat them as unsafe. Ideally, there should be an > > > >Assert if these can't happe

Re: list of extended statistics on psql

2021-01-18 Thread Tatsuro Yamada
Hi, The above query is so simple so that we would better to use the following query: # This query works on PG10 or later SELECT     es.stxnamespace::pg_catalog.regnamespace::text AS "Schema",     es.stxname AS "Name",     pg_catalog.format('%s FROM %s',     (SELECT pg_catalog.string_agg

Re: simplifying foreign key/RI checks

2021-01-18 Thread Zhihong Yu
Thanks for the quick response. + if (mapped_partkey_attnums[i] == pk_attnums[j]) + { + partkey_vals[i] = pk_vals[j]; + partkey_isnull[i] = pk_nulls[j] == 'n' ? true : false; + i++; + break; The way

Re: list of extended statistics on psql

2021-01-18 Thread Tomas Vondra
On 1/19/21 1:44 AM, Tatsuro Yamada wrote: Hi Tomas, As for how to deal with this, I can think of about three ways: 1) simplify the command to only print information from pg_statistic_ext (so on information about which stats are built or sizes) 2) extend pg_stats_ext with necessary infor

Re: simplifying foreign key/RI checks

2021-01-18 Thread Amit Langote
On Tue, Jan 19, 2021 at 2:47 AM Zhihong Yu wrote: > > Hi, > I was looking at this statement: > > insert into f select generate_series(1, 200, 2); > > Since certain generated values (the second half) are not in table p, wouldn't > insertion for those values fail ? > I tried a scaled down versi

configurable the threshold for warning due to run out of transaction ID

2021-01-18 Thread Masahiro Ikeda
Hi, PostgreSQL has the feature to warn about running out of transaction ID. The following message is an example. ``` 2021-01-19 10:59:27 JST [client backend] WARNING: database "postgres" must be vacuumed within xxx transactions 2021-01-19 10:59:27 JST [client backend] HINT: To avoid a databas

Re: Is Recovery actually paused?

2021-01-18 Thread Yugo NAGATA
On Sun, 17 Jan 2021 11:33:52 +0530 Dilip Kumar wrote: > On Thu, Jan 14, 2021 at 6:49 PM Yugo NAGATA wrote: > > > > On Wed, 13 Jan 2021 17:49:43 +0530 > > Dilip Kumar wrote: > > > > > On Wed, Jan 13, 2021 at 3:35 PM Dilip Kumar wrote: > > > > > > > > On Wed, Jan 13, 2021 at 3:27 PM Yugo NAGATA

RE: Parallel INSERT (INTO ... SELECT ...)

2021-01-18 Thread tsunakawa.ta...@fujitsu.com
From: Greg Nancarrow > On Fri, Jan 15, 2021 at 7:39 PM Amit Kapila wrote: > >Here, it seems we are accessing the relation descriptor without any > >lock on the table which is dangerous considering the table can be > >modified in a parallel session. Is there a reason why you think this > >is safe?

Re: POC: postgres_fdw insert batching

2021-01-18 Thread Tomas Vondra
On 1/19/21 2:28 AM, tsunakawa.ta...@fujitsu.com wrote: From: Tomas Vondra I took a look at this - there's a bit of bitrot due to 708d165ddb92c, so attached is a rebased patch (0001) fixing that. 0002 adds a couple comments and minor tweaks 0003 addresses a couple shortcomings related to expla

Re: pgindent for worker.c

2021-01-18 Thread Amit Kapila
On Tue, Jan 19, 2021 at 6:53 AM Peter Smith wrote: > > PSA a trivial patch just to pgindent the file > src/backend/replication/logical/worker.c > > (I am modifying this file in a separate patch, but every time I used > pgindent for my own code I would keep seeing these existing format > problems).

Re: [PATCH 1/1] Initial mach based shared memory support.

2021-01-18 Thread James Hilliard
On Mon, Jan 18, 2021 at 5:29 PM Tom Lane wrote: > > James Hilliard writes: > > OSX implements sysv shmem support via a mach wrapper, however the mach > > sysv shmem wrapper has some severe restrictions that prevent us from > > allocating enough memory segments in some cases. > > ... > > In order

Re: simplifying foreign key/RI checks

2021-01-18 Thread Amit Langote
On Tue, Jan 19, 2021 at 3:01 AM Pavel Stehule wrote: > po 18. 1. 2021 v 13:40 odesílatel Amit Langote > napsal: >> I started with the check that's performed when inserting into or >> updating the referencing table to confirm that the new row points to a >> valid row in the referenced relation.

Re: psql \df choose functions by their arguments

2021-01-18 Thread Ian Lawrence Barwick
2021年1月15日(金) 1:46 Greg Sabino Mullane : > Thanks for the feedback: new version v5 (attached) has int8, plus the > suggested code formatting. > > Cheers, > Greg > Thanks for the update. In my preceding mail I meant we should add int2, int4 and int8 for completeness (apologies, I was a bit unclea

Re: PoC/WIP: Extended statistics on expressions

2021-01-18 Thread Tomas Vondra
On 1/18/21 4:48 PM, Dean Rasheed wrote: Looking through extended_stats.c, I found a corner case that can lead to a seg-fault: CREATE TABLE foo(); CREATE STATISTICS s ON (1) FROM foo; ANALYSE foo; This crashes in lookup_var_attr_stats(), because it isn't expecting nvacatts to be 0. I can't thi

RE: POC: postgres_fdw insert batching

2021-01-18 Thread tsunakawa.ta...@fujitsu.com
Tomas-san, Zhihong-san, From: Zhihong Yu > + if (batch_size <= 0) > + ereport(ERROR, > + (errcode(ERRCODE_SYNTAX_ERROR), > +errmsg("%s requires a non-negative integer value", > > It seems the message doesn't match the check w

Paint some PG_USED_FOR_ASSERTS_ONLY in inline functions of ilist.h and bufpage.h

2021-01-18 Thread Michael Paquier
Hi all, The following functions in ilist.h and bufpage.h use some arguments only in assertions: - dlist_next_node - dlist_prev_node - slist_has_next - slist_next_node - PageValidateSpecialPointer Without PG_USED_FOR_ASSERTS_ONLY, this can lead to compilation warnings when not using assertions, an

RE: POC: postgres_fdw insert batching

2021-01-18 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > I took a look at this - there's a bit of bitrot due to 708d165ddb92c, so > attached is > a rebased patch (0001) fixing that. > > 0002 adds a couple comments and minor tweaks > > 0003 addresses a couple shortcomings related to explain - we haven't been > showing the batch si

pgindent for worker.c

2021-01-18 Thread Peter Smith
PSA a trivial patch just to pgindent the file src/backend/replication/logical/worker.c (I am modifying this file in a separate patch, but every time I used pgindent for my own code I would keep seeing these existing format problems). Kind Regards, Peter Smith. Fujitsu Australia pgindent_wo

RE: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2021-01-18 Thread Hou, Zhijie
> >>> +1 to add it after "dropped (Note )", how about as follows > >>> with slight changes? > >>> > >>> dropped (Note that server name of an invalid connection can be NULL > >>> if the server is dropped), and then such . > >> > >> Yes, I like this one. One question is; "should" or "is"

Re: list of extended statistics on psql

2021-01-18 Thread Tatsuro Yamada
Hi Tomas, As for how to deal with this, I can think of about three ways: 1) simplify the command to only print information from pg_statistic_ext (so on information about which stats are built or sizes) 2) extend pg_stats_ext with necessary information (e.g. sizes) 3) create a new system view

Re: [PATCH 1/1] Initial mach based shared memory support.

2021-01-18 Thread Tom Lane
James Hilliard writes: > OSX implements sysv shmem support via a mach wrapper, however the mach > sysv shmem wrapper has some severe restrictions that prevent us from > allocating enough memory segments in some cases. > ... > In order to avoid hitting these limits we can bypass the wrapper layer >

Re: Is it worth accepting multiple CRLs?

2021-01-18 Thread Kyotaro Horiguchi
At Fri, 15 Jan 2021 08:56:27 +0100, Peter Eisentraut wrote in > On 2020-08-31 11:03, Kyotaro Horiguchi wrote: > > At Tue, 18 Aug 2020 16:43:47 +0900 (JST), Kyotaro Horiguchi > > wrote in > >> Thank you very much. I'll do that after some polishing. > >> > >> A near-by discussion about OpenSSL3.0

Re: popcount

2021-01-18 Thread David Fetter
On Mon, Jan 18, 2021 at 10:34:10AM -0500, Tom Lane wrote: > Peter Eisentraut writes: > > [ assorted nits ] fixed, I think. > At the level of bikeshedding ... I quite dislike using the name "popcount" > for these functions. I'm aware that some C compilers provide primitives > of that name, but I

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Mon, Jan 18, 2021 at 04:38:47PM -0500, Robert Haas wrote: > To me, it wouldn't make sense to commit a full README for a TDE > feature that we don't have yet with a key management patch, but the > way that they'll interact with each other has to be clear. The > doc/database-encryption.sgml file t

Re: NOT VALID for Unique Indexes

2021-01-18 Thread japin
On Fri, 15 Jan 2021 at 00:22, Simon Riggs wrote: > As you may be aware the NOT VALID qualifier currently only applies to > CHECK and FK constraints, but not yet to unique indexes. I have had > customer requests to change that. > > It's a reasonably common requirement to be able to change an inde

Re: [WIP] UNNEST(REFCURSOR): allowing SELECT to consume data from a REFCURSOR

2021-01-18 Thread Massimo Fidanza
This is an interesting feature, but it seems that the author has abandoned development, what happens now? Will this be postponed from commitfest to commitfest and never be taken over by anyone? Massimo. Il giorno ven 6 mar 2020 alle ore 22:36 Dent John ha scritto: > > On 22 Feb 2020, at 10:38,

Re: Key management with tests

2021-01-18 Thread Tom Kincaid
I met with Bruce and Stephen this afternoon to discuss the feedback we received so far (prior to Robert's note which I haven't fully digested yet) on this patch. Here is what we plan to do: 1) Bruce is going to gather all the details from the Wiki and build a README for the TDE Key Management pa

Re: Odd, intermittent failure in contrib/pageinspect

2021-01-18 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Jan-18, Tom Lane wrote: >> [ thinks for a bit... ] Does the checkpointer pin pages it's writing >> out? I guess it'd have to ... > It does, per SyncOneBuffer(), called from BufferSync(), called from > CheckPointBuffers(). Right, then we don't need any strange t

Re: search_plan_tree(): handling of non-leaf CustomScanState nodes causes segfault

2021-01-18 Thread Tom Lane
David Geier writes: > On 18.01.21 19:46, Tom Lane wrote: >> Hm. I agree that we shouldn't simply assume that ss_currentRelation >> isn't null. However, we cannot make search_plan_tree() descend >> through non-leaf CustomScan nodes, because we don't know what processing >> is involved there. We

Re: Odd, intermittent failure in contrib/pageinspect

2021-01-18 Thread Alvaro Herrera
On 2021-Jan-18, Tom Lane wrote: > Right. If that's the explanation, then adding DISABLE_PAGE_SKIPPING > to the test's VACUUM options should fix it. However, to believe that > theory you have to have some reason to think that some other process > might have the page pinned. What would that be?

Re: Odd, intermittent failure in contrib/pageinspect

2021-01-18 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Jan-18, Tom Lane wrote: >> Searching the buildfarm logs turned up exactly one previous occurrence, >> also on whelk [2]. So I'm not sure what to make of it. Could the >> immediately preceding VACUUM FREEZE command have silently skipped this >> page for some reaso

Re: Odd, intermittent failure in contrib/pageinspect

2021-01-18 Thread Alvaro Herrera
On 2021-Jan-18, Tom Lane wrote: > Searching the buildfarm logs turned up exactly one previous occurrence, > also on whelk [2]. So I'm not sure what to make of it. Could the > immediately preceding VACUUM FREEZE command have silently skipped this > page for some reason? That'd be a bug I should

Re: CheckpointLock needed in CreateCheckPoint()?

2021-01-18 Thread Tom Lane
Robert Haas writes: > On Mon, Jan 18, 2021 at 3:25 PM Tom Lane wrote: >> If memory serves, the reason for the lock was that the CHECKPOINT >> command used to run the checkpointing code directly in the calling >> backend, so we needed it to keep more than one process from doing >> that at once. A

Odd, intermittent failure in contrib/pageinspect

2021-01-18 Thread Tom Lane
whelk failed today [1] with this surprising symptom: --- snip --- diff -w -U3 C:/buildfarm/buildenv/HEAD/pgsql.build/contrib/pageinspect/expected/page.out C:/buildfarm/buildenv/HEAD/pgsql.build/contrib/pageinspect/results/page.out --- C:/buildfarm/buildenv/HEAD/pgsql.build/contrib/pageinspect/e

Re: CheckpointLock needed in CreateCheckPoint()?

2021-01-18 Thread Robert Haas
On Mon, Jan 18, 2021 at 3:25 PM Tom Lane wrote: > If memory serves, the reason for the lock was that the CHECKPOINT > command used to run the checkpointing code directly in the calling > backend, so we needed it to keep more than one process from doing > that at once. AFAICS, it's no longer possi

Re: Key management with tests

2021-01-18 Thread Robert Haas
On Mon, Jan 18, 2021 at 2:00 PM Tom Kincaid wrote: > Some of the missing things you mention above are about the design of > TDE feature in general. However, this patch is about Key Management > which is going part of the larger TDE feature. So it feels as though > there is the need for a general

Re: Deleting older versions in unique indexes to avoid page splits

2021-01-18 Thread Peter Geoghegan
On Mon, Jan 18, 2021 at 1:10 PM Victor Yegorov wrote: > I must admit, that it's a bit difficult to understand you here (at least for > me). > > I assume that by "bet" you mean flagged tuple, that we marked as such > (should we implement the suggested case). > As heapam will give up early in case

Re: search_plan_tree(): handling of non-leaf CustomScanState nodes causes segfault

2021-01-18 Thread David Geier
Hi, On 18.01.21 19:46, Tom Lane wrote: David Geier writes: search_plan_tree() assumes that CustomScanState::ScanState::ss_currentRelation is never NULL. In my understanding that only holds for CustomScanState nodes which are at the bottom of the plan and actually read from a relation. CustomSc

Re: search_plan_tree(): handling of non-leaf CustomScanState nodes causes segfault

2021-01-18 Thread Tom Lane
Zhihong Yu writes: > I was thinking that, if sstate->ss_currentRelation is null for the other > cases, that would be a bug. > An assertion can be added for the cases ending with T_TidScanState. Maybe, but there are surely a lot of other places that would crash in such a case --- places far more o

Re: Deleting older versions in unique indexes to avoid page splits

2021-01-18 Thread Victor Yegorov
пн, 18 янв. 2021 г. в 21:43, Peter Geoghegan : > In the end, I couldn't justify imposing this cost on anything other > than a non-HOT updater, which is what the flag proposal would require > me to do -- then it's not 100% clear that the relative cost of each > "bet" placed in heapam.c really is ex

Re: search_plan_tree(): handling of non-leaf CustomScanState nodes causes segfault

2021-01-18 Thread Zhihong Yu
Hi, Tom: I was thinking that, if sstate->ss_currentRelation is null for the other cases, that would be a bug. An assertion can be added for the cases ending with T_TidScanState. Though, the null sstate->ss_currentRelation would surface immediately (apart from assertion). So I omitted the assertion

Re: Narrow the scope of the variable outputstr in logicalrep_write_tuple

2021-01-18 Thread Tom Lane
japin writes: > I find that the outputstr variable in logicalrep_write_tuple() only use in > `else` branch, I think we can narrow the scope, just like variable outputbytes > in `if` branch (for more readable). Agreed, done. For context, I'm not usually in favor of making one-off stylistic improv

Re: Add primary keys to system catalogs

2021-01-18 Thread Joel Jacobson
On Mon, Jan 18, 2021, at 19:33, Tom Lane wrote: > On second thought, a catalog is overkill; it'd only be useful if the data > could change after initdb, which this data surely cannot. The right way > to expose such info to SQL is with a set-returning function reading a > constant table in the C co

Re: truncating timestamps on arbitrary intervals

2021-01-18 Thread John Naylor
On Mon, Nov 23, 2020 at 1:44 PM John Naylor wrote: > > On Thu, Nov 12, 2020 at 9:56 AM Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > > - After reading the discussion a few times, I'm not so sure anymore > > whether making this a cousin of date_trunc is the right way to go. As > >

Re: Deleting older versions in unique indexes to avoid page splits

2021-01-18 Thread Peter Geoghegan
On Mon, Jan 18, 2021 at 6:11 AM Victor Yegorov wrote: > If I understand you correctly, you suggest to track _all_ the hints that came > from the executor for possible non-HOT logical duplicates somewhere on > the page. And when we hit the no-space case, we'll check not only the last > item being h

[PATCH] Full support for index LP_DEAD hint bits on standby

2021-01-18 Thread Michail Nikolaev
Hello, hackers. [ABSTRACT] Execution of queries to hot standby is one of the most popular ways to scale application workload. Most of the modern Postgres installations have two standby nodes for high-availability support. So, utilization of replica's CPU seems to be a reasonable idea. At the same

Re: PG vs LLVM 12 on seawasp, next round

2021-01-18 Thread Fabien COELHO
Hello Alvaro, The "no such file" error seems more like a machine local issue to me. I'll look into it when I have time, which make take some time. Hopefully over the holidays. This is still happening ... Any chance you can have a look at it? Indeed. I'll try to look (again) into it soon.

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-18 Thread Álvaro Herrera
On 2021-Jan-18, Matthias van de Meent wrote: > On Fri, 15 Jan 2021 at 15:29, Álvaro Herrera wrote: > Would this not need to be the following? Right now, it resets > potentially older h->catalog_oldest_nonremovable (which is set in the > PROC_IN_SAFE_IC branch). > > > +if (statusFlag

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-18 Thread Álvaro Herrera
On 2021-Jan-18, Matthias van de Meent wrote: > Example: > > 1.) RI starts > 2.) PHASE 2: filling the index: > 2.1.) scanning the heap (live tuple is cached) > < tuple is deleted > < last transaction other than RI commits, only snapshot of RI exists > < vacuum drops the tuple, and cannot remove it

Re: CheckpointLock needed in CreateCheckPoint()?

2021-01-18 Thread Tom Lane
Robert Haas writes: > Here's a patch to remove CheckpointLock completely. ... > So I don't see any problem with just ripping this out entirely, but > I'd like to know if anybody else does. If memory serves, the reason for the lock was that the CHECKPOINT command used to run the checkpointing cod

Re: search_plan_tree(): handling of non-leaf CustomScanState nodes causes segfault

2021-01-18 Thread Tom Lane
Zhihong Yu writes: > It seems sstate->ss_currentRelation being null can only > occur for T_ForeignScanState and T_CustomScanState. > What about the following change ? Seems like more code for no very good reason. regards, tom lane

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-18 Thread Matthias van de Meent
On Fri, 15 Jan 2021 at 15:29, Álvaro Herrera wrote: > > So one last remaining improvement was to have VACUUM ignore processes > doing CIC and RC to compute the Xid horizon of tuples to remove. I > think we can do something simple like the attached patch. Regarding the patch: > +if (

Re: [PATCH] Add support for leading/trailing bytea trim()ing

2021-01-18 Thread Tom Lane
"Joel Jacobson" writes: > On Fri, Dec 4, 2020, at 22:02, Tom Lane wrote: >> (Maybe the existing ltrim/rtrim descrs are also like this, but if so >> I'd change them too.) > They weren't, but I think the description for the bytea functions > can be improved to have a more precise description > if w

Re: Deleting older versions in unique indexes to avoid page splits

2021-01-18 Thread Peter Geoghegan
On Sun, Jan 17, 2021 at 10:44 PM Amit Kapila wrote: > With the above example, it seems like it would also help when this is not > true. I'll respond to your remarks here separately, in a later email. > I am not sure what I proposed fits here but the bottom-up sounds like > we are starting from

Re: CheckpointLock needed in CreateCheckPoint()?

2021-01-18 Thread Robert Haas
On Thu, Jan 14, 2021 at 10:21 AM Robert Haas wrote: > Yeah, I think this lock is useless. In fact, I think it's harmful, > because it makes large sections of the checkpointer code, basically > all of it really, run with interrupts held off for no reason. > > It's not impossible that removing the l

Re: search_plan_tree(): handling of non-leaf CustomScanState nodes causes segfault

2021-01-18 Thread Zhihong Yu
Hi, It seems sstate->ss_currentRelation being null can only occur for T_ForeignScanState and T_CustomScanState. What about the following change ? Cheers diff --git a/src/backend/executor/execCurrent.c b/src/backend/executor/execCurrent.c index 0852bb9cec..56e31951d1 100644 --- a/src/backend/exec

Re: Key management with tests

2021-01-18 Thread Andres Freund
On 2021-01-18 13:58:20 -0500, Bruce Momjian wrote: > On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote: > > Personally, but I admit that there's legitimate reasons to differ on > > that note, I don't think it's reasonable for a feature this invasive to > > commit preliminary patches wit

Re: Key management with tests

2021-01-18 Thread Tom Kincaid
> > I have to admit I was kind of baffled that the wiki page wasn't > > sufficient, because it is one of the longest Postgres feature > > explanations I have seen, but I now think the missing part is tying > > the wiki contents to the code implementation. If that is it, please > > confirm. If it

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote: > Personally, but I admit that there's legitimate reasons to differ on > that note, I don't think it's reasonable for a feature this invasive to > commit preliminary patches without the major subsequent patches being in > a shape that a

Re: WIP: System Versioned Temporal Table

2021-01-18 Thread Surafel Temesgen
On Mon, Jan 18, 2021 at 1:43 AM Vik Fearing wrote: > > This is not good, and I see that DROP SYSTEM VERSIONING also removes > these columns which is even worse. Please read the standard that you > are trying to implement! > > The standard states the function of ALTER TABLE ADD SYSTEM VERSIONING

Re: search_plan_tree(): handling of non-leaf CustomScanState nodes causes segfault

2021-01-18 Thread Tom Lane
David Geier writes: > search_plan_tree() assumes that > CustomScanState::ScanState::ss_currentRelation is never NULL. In my > understanding that only holds for CustomScanState nodes which are at the > bottom of the plan and actually read from a relation. CustomScanState > nodes which are not a

  1   2   >