Re: locking [user] catalog tables vs 2pc vs logical rep

2021-05-25 Thread Michael Paquier
On Mon, May 24, 2021 at 10:03:01AM +0530, Amit Kapila wrote: > So, this appears to be an existing caveat of synchronous replication. > If that is the case, I am not sure if it is a good idea to just block > such ops for the prepared transaction. Also, what about other > operations which acquire an

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Michael Paquier
On Tue, May 25, 2021 at 12:06:38PM +0530, Pavan Deolasee wrote: > While working on an output plugin that uses streaming protocol, I hit an > assertion failure. Further investigations revealed a possible bug in core > Postgres. This must be new to PG14 since streaming support is new to this > releas

Re: Different compression methods for FPI

2021-05-25 Thread Michael Paquier
On Mon, May 24, 2021 at 11:44:45PM -0500, Justin Pryzby wrote: > The goal is to support 2+ "methods" (including "none"), which takes 4 bits, so > may as well support 3 methods. > > - uncompressed > - pglz > - lz4 > - zlib or zstd or ?? Let's make a proper study of all that and make a choice, the

RE: Fdw batch insert error out when set batch_size > 65535

2021-05-25 Thread houzj.f...@fujitsu.com
From: Bharath Rupireddy Sent: Friday, May 21, 2021 5:03 PM > On Fri, May 21, 2021 at 1:19 PM houzj.f...@fujitsu.com > wrote: > > Attaching V2 patch. Please consider it for further review. > > Thanks for the patch. Some more comments: > > 1) Can fmstate->p_nums ever be 0 in postgresGetForeignMod

RE: Parallel Inserts in CREATE TABLE AS

2021-05-25 Thread tsunakawa.ta...@fujitsu.com
Bharath-san, all, Hmm, I didn't experience performance degradation on my poor-man's Linux VM (4 CPU, 4 GB RAM, HDD)... [benchmark preparation] autovacuum = off shared_buffers = 1GB checkpoint_timeout = 1h max_wal_size = 8GB min_wal_size = 8GB (other settings to enable parallelism) CREATE UNLOGG

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 12:12 PM Dilip Kumar wrote: > > On Tue, May 25, 2021 at 12:06 PM Pavan Deolasee > wrote: > > > > Hi, > > > > While working on an output plugin that uses streaming protocol, I hit an > > assertion failure. Further investigations revealed a possible bug in core > > Postgre

RE: locking [user] catalog tables vs 2pc vs logical rep

2021-05-25 Thread osumi.takami...@fujitsu.com
On Monday, May 24, 2021 1:33 PM Amit Kapila wrote: > On Tue, Apr 20, 2021 at 9:57 AM vignesh C wrote: > > > > This similar problem exists in case of synchronous replication setup > > having synchronous_standby_names referring to the subscriber, when we > > do the steps "begin;lock pg_class; inser

Re: Skipping logical replication transactions on subscriber side

2021-05-25 Thread Masahiko Sawada
On Tue, May 25, 2021 at 2:49 PM Bharath Rupireddy wrote: > > On Mon, May 24, 2021 at 1:32 PM Masahiko Sawada wrote: > > > > Hi all, > > > > If a logical replication worker cannot apply the change on the > > subscriber for some reason (e.g., missing table or violating a > > constraint, etc.), logi

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Pavan Deolasee
On Tue, May 25, 2021 at 1:26 PM Dilip Kumar wrote: > > > I have identified the cause of the issue, basically, the reason is if > we are doing a multi insert operation we don't set the toast cleanup > until we get the last tuple of the xl_multi_insert [1]. Now, with > streaming, we can process th

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 1:45 PM Pavan Deolasee wrote: > > I am not entirely sure if it works correctly. I'd tried something similar, > but the downstream node using > my output plugin gets NULL values for the toast columns. It's a bit hard to > demonstrate that with the > test_decoding plugin,

RE: Parallel Inserts in CREATE TABLE AS

2021-05-25 Thread tsunakawa.ta...@fujitsu.com
From: houzj.f...@fujitsu.com > + /* > + * We don't need to skip contacting FSM while inserting tuples > for > + * parallel mode, while extending the relations, workers > instead of > + * blocking on a page while another worker is inserting, can >

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Pavan Deolasee
On Tue, May 25, 2021 at 1:49 PM Dilip Kumar wrote: > On Tue, May 25, 2021 at 1:45 PM Pavan Deolasee > wrote: > > > > I am not entirely sure if it works correctly. I'd tried something > similar, but the downstream node using > > my output plugin gets NULL values for the toast columns. It's a bit

sync request forward function ForwardSyncRequest() might hang for some time in a corner case?

2021-05-25 Thread Paul Guo
Hi hackers, I found this when reading the related code. Here is the scenario: bool RegisterSyncRequest(const FileTag *ftag, SyncRequestType type, bool retryOnError) For the case retryOnError is true, the function would in loop call ForwardSyncRequest() until it succeeds, but

Re: pg_rewind fails if there is a read only file.

2021-05-25 Thread Paul Guo
> You seem to have missed my point completely. The answer to this problem > IMNSHO is "Don't put a read-only file in the data directory". Oh sorry. Well, if we really do not want this we may want to document this and keep educating users, but meanwhile probably the product should be more user frie

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 1:59 PM Pavan Deolasee wrote: > > On Tue, May 25, 2021 at 1:49 PM Dilip Kumar wrote: >> >> On Tue, May 25, 2021 at 1:45 PM Pavan Deolasee >> wrote: >> > >> > I am not entirely sure if it works correctly. I'd tried something similar, >> > but the downstream node using >

Re: libpq_pipeline in tmp_install

2021-05-25 Thread Peter Eisentraut
On 19.05.21 19:35, Tom Lane wrote: Alvaro Herrera writes: On 2021-May-19, Tom Lane wrote: +1, except that you should add documentation for NO_INSTALL to the list of definable symbols at the head of pgxs.mk, and to the list in extend.sgml (compare that for NO_INSTALLCHECK). I propose this.

RE: Added schema level support for publication.

2021-05-25 Thread tanghy.f...@fujitsu.com
On Monday, May 24, 2021 at 8:31 PM vignesh C wrote: > The earlier patch does not apply on the head. The v4 patch attached > has the following changes: > a) Rebased it on head. b) Removed pubschemas, pubtables columns and > replaced it with pubtype in pg_publication table. c) List the schemas > in

Re: Fdw batch insert error out when set batch_size > 65535

2021-05-25 Thread Bharath Rupireddy
On Tue, May 25, 2021 at 1:08 PM houzj.f...@fujitsu.com wrote: > Thanks for the comments. I have addressed all comments to the v3 patch. Thanks! The patch basically looks good to me except that it is missing a commit message. I think it can be added now. > BTW, Is the batch_size issue here an Ope

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 2:34 PM Dilip Kumar wrote: > > > When the transaction is streamed, I see: > > ``` > > + opening a streamed block for transaction > > + table public.toasted: INSERT: id[integer]:9001 other[text]:'bbb' > > data[text]:'ccc' > > + table public.toasted: INSERT: id[integer]:9002

Replace run-time error check with assertion

2021-05-25 Thread Peter Eisentraut
In the attached patch, the error message was checking that the structures returned from the parser matched expectations. That's something we usually use assertions for, not a full user-facing error message. So I replaced that with an assertion (hidden inside lfirst_node()). From a30697758c4

Re: parallel vacuum - few questions on docs, comments and code

2021-05-25 Thread Amit Kapila
On Fri, May 21, 2021 at 3:33 PM Amit Kapila wrote: > > On Fri, May 21, 2021 at 1:30 PM Bharath Rupireddy > wrote: > > > > On Fri, May 21, 2021 at 11:10 AM Amit Kapila > > wrote: > > > > > > If the patch changes the vacuumdb code as above then isn't it better > > > to change the vacuumdb docs to

Re: Replication slot stats misgivings

2021-05-25 Thread Amit Kapila
On Mon, May 24, 2021 at 10:09 AM vignesh C wrote: > > On Mon, May 24, 2021 at 9:38 AM Amit Kapila wrote: > > > > On Thu, May 13, 2021 at 11:30 AM vignesh C wrote: > > > > > > > Do we want to update the information about pg_stat_replication_slots > > at the following place in docs > > https://www

Re: [PATCH] Add `truncate` option to subscription commands

2021-05-25 Thread Amit Kapila
On Mon, May 24, 2021 at 2:10 PM Bharath Rupireddy wrote: > > On Mon, May 24, 2021 at 11:01 AM Amit Kapila wrote: > > > 1) Whether a table the sync worker is trying to truncate is having any > > > referencing (foreign key) tables on the subscriber? If yes, whether > > > all the referencing tables

Re: Logical Replication - behavior of TRUNCATE ... CASCADE

2021-05-25 Thread Amit Kapila
On Mon, May 24, 2021 at 2:18 PM Bharath Rupireddy wrote: > > On Mon, May 24, 2021 at 11:22 AM Amit Kapila wrote: > > I don't deny that this can allow some additional cases than we allow > > today but was just not sure whether users really need it. If we want > > to go with such an option then as

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Pavan Deolasee
On Tue, May 25, 2021 at 2:57 PM Dilip Kumar wrote: > > > > > Yes, I am able to reproduce this, basically, until we get the last > > tuple of the multi insert we can not clear the toast data otherwise we > > can never form a complete tuple. So the only possible fix I can think > > of is to consid

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 3:33 PM Pavan Deolasee wrote: >> The attached patch should fix the issue, now the output is like below >> > > Thanks. This looks fine to me. We should still be able to stream multi-insert > transactions (COPY) as and when the copy buffer becomes full and is flushed. > Th

Re: Skipping logical replication transactions on subscriber side

2021-05-25 Thread Bharath Rupireddy
On Tue, May 25, 2021 at 1:44 PM Masahiko Sawada wrote: > > On Tue, May 25, 2021 at 2:49 PM Bharath Rupireddy > wrote: > > > > On Mon, May 24, 2021 at 1:32 PM Masahiko Sawada > > wrote: > > > > > > Hi all, > > > > > > If a logical replication worker cannot apply the change on the > > > subscribe

How can the Aggregation move to the outer query

2021-05-25 Thread Andy Fan
My question can be demonstrated with the below example: create table m1(a int, b int); explain (costs off) select (select count(*) filter (*where true*) from m1 t1) from m1 t2 where t2.b % 2 = 1; QUERY PLAN - Seq Scan on m1 t2 Filter: ((b % 2) = 1)

Add ZSON extension to /contrib/

2021-05-25 Thread Aleksander Alekseev
Hi hackers, Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. The extension introduces the new ZSON type, which is 100% compatible with JSONB but uses a shared dictionary of strings most frequently used in given JSONB documents for compression. These strings are replaced

Re: Adaptive Plan Sharing for PreparedStmt

2021-05-25 Thread Andy Fan
Hi, On Thu, May 20, 2021 at 5:02 PM Tomas Vondra wrote: > Hi, > > On 5/20/21 5:43 AM, Andy Fan wrote: > > Currently we are using a custom/generic strategy to handle the data skew > > issue. However, it doesn't work well all the time. For example: SELECT * > > FROM t WHERE a between $1 and $2. W

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Amit Kapila
On Tue, May 25, 2021 at 2:57 PM Dilip Kumar wrote: > > On Tue, May 25, 2021 at 2:34 PM Dilip Kumar wrote: > > > > > When the transaction is streamed, I see: > > > ``` > > > + opening a streamed block for transaction > > > + table public.toasted: INSERT: id[integer]:9001 other[text]:'bbb' > > > d

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Peter Eisentraut
On 24.05.21 02:01, Tom Lane wrote: I wrote: I think we ought to fix this so that OUT-only arguments are ignored when calling from SQL not plpgsql. I'm working on a patch to make it act that way. I've got some issues yet to fix with named arguments (which seem rather undertested BTW, since the

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Magnus Hagander
On Tue, May 25, 2021 at 12:55 PM Aleksander Alekseev wrote: > > Hi hackers, > > Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. > The extension introduces the new ZSON type, which is 100% compatible with > JSONB but uses a shared dictionary of strings most frequently

Re: Fixing the docs for ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION

2021-05-25 Thread Peter Eisentraut
On 21.05.21 22:19, Mark Dilger wrote: +ALTER SUBSCRIPTION mysubscription DROP PUBLICATION nosuchpub WITH (copy_data = false, refresh = false); +ERROR: unrecognized subscription parameter: "copy_data" +ALTER SUBSCRIPTION mysubscription SET (copy_data = false, refresh = false); +ERROR: unrecogni

Re: How can the Aggregation move to the outer query

2021-05-25 Thread David Rowley
On Tue, 25 May 2021 at 22:28, Andy Fan wrote: > > explain (costs off) select (select count(*) filter (where t2.b = 1) from m1 > t1) > from m1 t2 where t2.b % 2 = 1; > > QUERY PLAN > --- > Aggregate >-> Seq Scan on m1 t2 > Filter: ((b % 2) = 1)

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Ranier Vilela
Em seg., 24 de mai. de 2021 às 23:35, Zhihong Yu escreveu: > > > On Mon, May 24, 2021 at 7:21 PM Ranier Vilela wrote: > >> Em seg., 24 de mai. de 2021 às 22:42, Mark Dilger < >> mark.dil...@enterprisedb.com> escreveu: >> >>> >>> >>> > On May 24, 2021, at 5:37 PM, Ranier Vilela >>> wrote: >>> >

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 4:50 PM Amit Kapila wrote: > > Your patch will fix the reported scenario but I don't like the way > multi_insert flag is used to detect incomplete tuple. One problem > could be that even when there are no toast inserts, it won't allow to > stream unless we get the last tupl

Re: Skipping logical replication transactions on subscriber side

2021-05-25 Thread Masahiko Sawada
On Tue, May 25, 2021 at 7:21 PM Bharath Rupireddy wrote: > > On Tue, May 25, 2021 at 1:44 PM Masahiko Sawada wrote: > > > > On Tue, May 25, 2021 at 2:49 PM Bharath Rupireddy > > wrote: > > > > > > On Mon, May 24, 2021 at 1:32 PM Masahiko Sawada > > > wrote: > > > > > > > > Hi all, > > > > > >

Re: Transactions involving multiple postgres foreign servers, take 2

2021-05-25 Thread Masahiko Sawada
On Fri, May 21, 2021 at 5:48 PM Masahiro Ikeda wrote: > > > > On 2021/05/21 13:45, Masahiko Sawada wrote: > > > > Yes. We also might need to be careful about the order of foreign > > transaction resolution. I think we need to resolve foreign> transactions in > > arrival order at least within a fo

Re: logical replication empty transactions

2021-05-25 Thread Ajin Cherian
On Tue, Apr 27, 2021 at 1:49 PM Ajin Cherian wrote: Rebased the patch as it was no longer applying. regards, Ajin Cherian Fujitsu Australia v6-0001-Skip-empty-transactions-for-logical-replication.patch Description: Binary data

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 5:46 PM Dilip Kumar wrote: > > On Tue, May 25, 2021 at 4:50 PM Amit Kapila wrote: > > > > Your patch will fix the reported scenario but I don't like the way > > multi_insert flag is used to detect incomplete tuple. One problem > > could be that even when there are no toast

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Pavan Deolasee
On Tue, May 25, 2021 at 6:43 PM Dilip Kumar wrote: > > I have also added a test case for this. > > Is that test good enough to trigger the original bug? In my experience, I had to add a lot more tuples before the logical_decoding_work_mem threshold was crossed and the streaming kicked in. I would

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 6:50 PM Pavan Deolasee wrote: > Is that test good enough to trigger the original bug? In my experience, I had > to add a lot more tuples before the logical_decoding_work_mem threshold was > crossed and the streaming kicked in. I would suggest running the test without >

Re: Refactor "mutually exclusive options" error reporting code in parse_subscription_options

2021-05-25 Thread Alvaro Herrera
On 2021-May-25, Bharath Rupireddy wrote: > On Mon, May 24, 2021 at 11:37 PM Alvaro Herrera > wrote: > > There's no API limitation here, since that stuff is not user-visible, so > > it doesn't matter. If we ever need a 33rd option, we can change the > > datatype to bits64. Any extensions using

Re: pg_rewind fails if there is a read only file.

2021-05-25 Thread Laurenz Albe
On Tue, 2021-05-25 at 16:57 +0800, Paul Guo wrote: > > You seem to have missed my point completely. The answer to this problem > > IMNSHO is "Don't put a read-only file in the data directory". > > Oh sorry. Well, if we really do not want this we may want to document this > and keep educating users

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Pavan Deolasee
On Tue, May 25, 2021 at 6:55 PM Dilip Kumar wrote: > On Tue, May 25, 2021 at 6:50 PM Pavan Deolasee > wrote: > > > Is that test good enough to trigger the original bug? In my experience, > I had to add a lot more tuples before the logical_decoding_work_mem > threshold was crossed and the streami

Re: Skip partition tuple routing with constant partition key

2021-05-25 Thread Amit Langote
Hou-san, On Mon, May 24, 2021 at 10:15 PM houzj.f...@fujitsu.com wrote: > From: Amit Langote > Sent: Monday, May 24, 2021 4:27 PM > > On Mon, May 24, 2021 at 10:31 AM houzj.f...@fujitsu.com > > wrote: > > > Currently for a normal RANGE partition key it will first generate a > > > CHECK expressi

Re: How can the Aggregation move to the outer query

2021-05-25 Thread Andy Fan
On Tue, May 25, 2021 at 7:42 PM David Rowley wrote: > On Tue, 25 May 2021 at 22:28, Andy Fan wrote: > > > > explain (costs off) select (select count(*) filter (where t2.b = 1) > from m1 t1) > > from m1 t2 where t2.b % 2 = 1; > > > > QUERY PLAN > > --- > >

Re: pg_rewind fails if there is a read only file.

2021-05-25 Thread Andrew Dunstan
On 5/25/21 9:38 AM, Laurenz Albe wrote: > On Tue, 2021-05-25 at 16:57 +0800, Paul Guo wrote: >>> You seem to have missed my point completely. The answer to this problem >>> IMNSHO is "Don't put a read-only file in the data directory". >> Oh sorry. Well, if we really do not want this we may want t

Re: How can the Aggregation move to the outer query

2021-05-25 Thread Tom Lane
David Rowley writes: > On Tue, 25 May 2021 at 22:28, Andy Fan wrote: >> explain (costs off) select (select count(*) filter (where t2.b = 1) from m1 >> t1) >> from m1 t2 where t2.b % 2 = 1; >> >> This one is too confusing to me since the Aggregate happens >> on t2 rather than t1. What happens

Re: pg_rewind fails if there is a read only file.

2021-05-25 Thread Tom Lane
Andrew Dunstan writes: > Perhaps we should add that read-only files can be particularly problematic. Given the (legitimate, IMO) example of a read-only SSL key, I'm not quite convinced that pg_rewind doesn't need to cope with this. regards, tom lane

Re: Replace run-time error check with assertion

2021-05-25 Thread Tom Lane
Peter Eisentraut writes: > In the attached patch, the error message was checking that the > structures returned from the parser matched expectations. That's > something we usually use assertions for, not a full user-facing error > message. So I replaced that with an assertion (hidden inside

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
Peter Eisentraut writes: > On 24.05.21 02:01, Tom Lane wrote: >>> I think we ought to fix this so that OUT-only arguments are ignored >>> when calling from SQL not plpgsql. > I don't understand why you want to change this. The argument resolution > of CALL is specified in the SQL standard; we s

Re: Test of a partition with an incomplete detach has a timing issue

2021-05-25 Thread Alvaro Herrera
So I had a hard time reproducing the problem, until I realized that I needed to limit the server to use only one CPU, and in addition run some other stuff concurrently in the same server in order to keep it busy. With that, I see about one failure every 10 runs. So I start the server as "numactl -

Re: Test of a partition with an incomplete detach has a timing issue

2021-05-25 Thread Tom Lane
Alvaro Herrera writes: > The problem disappears completely if I add a sleep to the cancel query: > step "s1cancel" { SELECT pg_cancel_backend(pid), pg_sleep(0.01) FROM > d3_pid; } > I suppose a 0.01 second sleep is not going to be sufficient to close the > problem in slower animals, but I h

Re: Race condition in recovery?

2021-05-25 Thread Robert Haas
On Sun, May 23, 2021 at 12:08 PM Dilip Kumar wrote: > I have created a tap test based on Robert's test.sh script. It > reproduces the issue. I am new with perl so this still needs some > cleanup/improvement, but at least it shows the idea. Thanks. I think this is the right idea but just needs a

Re: automatic analyze: readahead - add "IO read time" log message

2021-05-25 Thread Egor Rogov
Hi, On 11.02.2021 01:10, Stephen Frost wrote: Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: On 05/02/2021 23:22, Stephen Frost wrote: Unless there's anything else on this, I'll commit these sometime next week. One more thing: Instead of using 'effective_io_concurrency' GUC direct

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Tom Lane
Ranier Vilela writes: > Possible pointer TupleDesc rettupdesc used not initialized? > if (!isNull) at line 4346 taking a true branch, the function > check_sql_fn_retval at line 4448 can use rettupdesc uninitialized. This seems to have been introduced by the SQL-function-body patch. After some st

Re: array_cat anycompatible change is breaking xversion upgrade tests

2021-05-25 Thread Justin Pryzby
On Thu, May 20, 2021 at 07:35:10PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > On Wed, Nov 04, 2020 at 07:43:51PM -0500, Tom Lane wrote: > >> As was discussed in the thread leading up to that commit, modifying the > >> signature of array_cat and friends could break user-defined operators >

Re: How can the Aggregation move to the outer query

2021-05-25 Thread Andy Fan
On Tue, May 25, 2021 at 10:23 PM Tom Lane wrote: > David Rowley writes: > > On Tue, 25 May 2021 at 22:28, Andy Fan wrote: > >> explain (costs off) select (select count(*) filter (where t2.b = 1) > from m1 t1) > >> from m1 t2 where t2.b % 2 = 1; > >> > >> This one is too confusing to me since t

Re: Test of a partition with an incomplete detach has a timing issue

2021-05-25 Thread Alvaro Herrera
On 2021-May-25, Tom Lane wrote: > Alvaro Herrera writes: > > The problem disappears completely if I add a sleep to the cancel query: > > step "s1cancel" { SELECT pg_cancel_backend(pid), pg_sleep(0.01) FROM > > d3_pid; } > > I suppose a 0.01 second sleep is not going to be sufficient to close

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Ranier Vilela
Em ter., 25 de mai. de 2021 às 13:09, Tom Lane escreveu: > Ranier Vilela writes: > > Possible pointer TupleDesc rettupdesc used not initialized? > > if (!isNull) at line 4346 taking a true branch, the function > > check_sql_fn_retval at line 4448 can use rettupdesc uninitialized. > > This seems

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Tom Lane
Ranier Vilela writes: > Following the guidelines, I provided a patch. Oh, I already pushed a fix, thanks. regards, tom lane

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
Hi, On 2021-05-25 12:46:45 -0400, Robert Haas wrote: > This approach has a few disadvantages. For example, right now, we only > need to WAL log hints for the first write to each page after a > checkpoint, but in this approach, if the same page is written multiple > times per checkpoint cycle, we'd

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Ranier Vilela
Em ter., 25 de mai. de 2021 às 14:35, Tom Lane escreveu: > Ranier Vilela writes: > > Following the guidelines, I provided a patch. > > Oh, I already pushed a fix, thanks. > No problem! Thank you. regards, Ranier Vilela

Access root->simple_rte_array instead of Query->rtable for 2 more cases.

2021-05-25 Thread Andy Fan
When I am understanding the relationship between Query->rtable and root->simple_rte_array, I'd like to assume that Query->rtable should be never used when root->simple_rte_array is ready. I mainly checked two places, make_one_rel and create_plan with the below hacks. { List *l = root->parse->rta

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Peter Eisentraut
On 25.05.21 17:20, Tom Lane wrote: I don't really see how you can argue that the existing behavior is more spec-compliant than what I'm suggesting. What I read in the spec (SQL:2021 10.4 SR 9) h) iii) 1)) is 1) If Pi is an output SQL parameter, then XAi shall be a . (where more or less

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
I wrote: > * The rules for how to identify a target routine in ALTER, DROP, > etc are different for functions and procedures. That's especially > nasty in ALTER/DROP ROUTINE, where we don't have a syntax cue > as to whether or not to ignore OUT parameters. Just to enlarge on that point a bit: re

Re: storing an explicit nonce

2021-05-25 Thread Robert Haas
On Tue, May 25, 2021 at 1:37 PM Andres Freund wrote: > The obvious concerns are issues around binary upgrades for cases that > already use the special space? Are you planning to address that by not > having that path? Or by storing the nonce at the "start" of the special > space (i.e. [normal data

Re: Access root->simple_rte_array instead of Query->rtable for 2 more cases.

2021-05-25 Thread Tom Lane
Andy Fan writes: > When I am understanding the relationship between Query->rtable and > root->simple_rte_array, I'd like to assume that Query->rtable should be > never used > when root->simple_rte_array is ready. TBH, now that Lists are really arrays, there's basically no performance advantage to

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 12:46:45PM -0400, Robert Haas wrote: > On Thu, Mar 18, 2021 at 2:59 PM Bruce Momjian wrote: > > > Ultimately, we need to make sure that LSNs aren't re-used. There's two > > > sources of LSNs today: those for relations which are being written into > > > the WAL and those fo

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 02:25:21PM -0400, Robert Haas wrote: > One question here is whether we're comfortable saying that the nonce > is entirely constant. I wasn't sure about that. It seems possible to > me that different encryption algorithms might want nonces of different > sizes, either now or

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Robert Haas
On Tue, May 25, 2021 at 2:20 PM Tom Lane wrote: > Just to enlarge on that point a bit: > > regression=# create function foo(int, out int) language sql > regression-# as 'select $1'; > CREATE FUNCTION > regression=# create procedure foo(int, out int) language sql > regression-# as 'select $1'; > CR

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
Peter Eisentraut writes: > On 25.05.21 17:20, Tom Lane wrote: >> I don't really see how you can argue that the existing behavior is >> more spec-compliant than what I'm suggesting. What I read in the spec >> (SQL:2021 10.4 SR 9) h) iii) 1)) is >> 1) If Pi is an output SQL parameter, then XAi sha

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 10:37:32AM -0700, Andres Freund wrote: > The obvious concerns are issues around binary upgrades for cases that > already use the special space? Are you planning to address that by not > having that path? Or by storing the nonce at the "start" of the special > space (i.e. [no

Re: storing an explicit nonce

2021-05-25 Thread Robert Haas
On Tue, May 25, 2021 at 2:45 PM Bruce Momjian wrote: > Well, if we create a separate nonce counter, we still need to make sure > it doesn't go backwards during a crash, so we have to WAL log it I think we don't really need a global counter, do we? We could simply increment the nonce every time we

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
Robert Haas writes: > I'm also concerned about the behavior here. I noticed it when this > commit went in, and it seemed concerning to me then, and it still > does. Nevertheless, I'm not convinced that your proposal is an > improvement. Suppose we have foo(int, out int) and also foo(int). > Then,

Re: pg_rewind fails if there is a read only file.

2021-05-25 Thread Andrew Dunstan
On 5/25/21 10:29 AM, Tom Lane wrote: > Andrew Dunstan writes: >> Perhaps we should add that read-only files can be particularly problematic. > Given the (legitimate, IMO) example of a read-only SSL key, I'm not > quite convinced that pg_rewind doesn't need to cope with this. > >

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 03:09:03PM -0400, Robert Haas wrote: > On Tue, May 25, 2021 at 2:45 PM Bruce Momjian wrote: > > Well, if we create a separate nonce counter, we still need to make sure > > it doesn't go backwards during a crash, so we have to WAL log it > > I think we don't really need a g

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-05-25 Thread Alvaro Herrera
On 2021-May-24, Noah Misch wrote: > prairiedog and wrasse failed a $SUBJECT test after this (commit 8aba932). > Also, some non-CLOBBER_CACHE_ALWAYS animals failed a test before the fix. > These IsolationCheck failures match detach-partition-concurrently[^\n]*FAILED: > > sysname │ snapsho

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 03:20:06PM -0400, Bruce Momjian wrote: > Also, when you change hint bits, either you don't change the nonce/LSN, > and don't re-encrypt the page (and the hint bit changes are visible), or > you change the nonce and reencrypt the page, and you are then WAL > logging the page.

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 03:34:04PM -0400, Bruce Momjian wrote: > Let me go into more detail here. The general rule is that you never > encrypt _different_ data with the same key/nonce. Now, since a hint bit > change changes the data, it should get a new nonce, and since it is a > newly encrypted

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Robert Haas
On Tue, May 25, 2021 at 3:10 PM Tom Lane wrote: > No, you misunderstand my proposal. The thing that I most urgently > want to do is to prevent that situation from ever arising, by not > allowing those two procedures to coexist, just as you can't have > two functions with such signatures. > > If p

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Andrew Dunstan
On 5/25/21 6:55 AM, Aleksander Alekseev wrote: > Hi hackers, > > Back in 2016 while being at PostgresPro I developed the ZSON extension > [1]. The extension introduces the new ZSON type, which is 100% > compatible with JSONB but uses a shared dictionary of strings most > frequently used in given

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Tom Lane
Magnus Hagander writes: > On Tue, May 25, 2021 at 12:55 PM Aleksander Alekseev > wrote: >> Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. >> The extension introduces the new ZSON type, which is 100% compatible with >> JSONB but uses a shared dictionary of strings mo

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Matthias van de Meent
On Tue, 25 May 2021 at 13:32, Magnus Hagander wrote: > > On Tue, May 25, 2021 at 12:55 PM Aleksander Alekseev > wrote: > > > > Hi hackers, > > > > Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. > > The extension introduces the new ZSON type, which is 100% compatible

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
Robert Haas writes: > On Tue, May 25, 2021 at 3:10 PM Tom Lane wrote: >> You're definitely confused, because reversing that choice is *exactly* >> what I'm on about. The question of whether SQL-level CALL should act >> differently from plpgsql CALL is pretty secondary. > I understood the revers

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Andrew Dunstan
On 5/25/21 4:10 PM, Tom Lane wrote: > Magnus Hagander writes: >> On Tue, May 25, 2021 at 12:55 PM Aleksander Alekseev >> wrote: >>> Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. >>> The extension introduces the new ZSON type, which is 100% compatible with >>> JSO

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, On Tue, May 25, 2021 at 14:56 Bruce Momjian wrote: > On Tue, May 25, 2021 at 02:25:21PM -0400, Robert Haas wrote: > > One question here is whether we're comfortable saying that the nonce > > is entirely constant. I wasn't sure about that. It seems possible to > > me that different enc

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Tom Lane
Andrew Dunstan writes: > On 5/25/21 4:10 PM, Tom Lane wrote: >> Also, even if ZSON was "100% compatible with JSONB" back in 2016, >> a whole lot of features have been added since then. Having to >> duplicate all that code again for a different data type is not >> something I want to see us doing.

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, On Tue, May 25, 2021 at 15:09 Robert Haas wrote: > On Tue, May 25, 2021 at 2:45 PM Bruce Momjian wrote: > > Well, if we create a separate nonce counter, we still need to make sure > > it doesn't go backwards during a crash, so we have to WAL log it > > I think we don't really need a

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Tom Lane
Matthias van de Meent writes: > I like the idea of the ZSON type, but I'm somewhat disappointed by its > current limitations: I've not read the code, so maybe this thought is completely off-point, but I wonder if anything could be learned from PostGIS. AIUI they have developed the infrastructure

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
Hi, On 2021-05-25 15:34:04 -0400, Bruce Momjian wrote: > My point is that we have to full-page-write cases where we change the > nonce --- we get a new LSN/nonce for free if we are using the LSN as the > nonce. What has made this approach much easier is that you basically > tie a change of the no

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 01:54:21PM -0700, Andres Freund wrote: > Hi, > > On 2021-05-25 15:34:04 -0400, Bruce Momjian wrote: > > My point is that we have to full-page-write cases where we change the > > nonce --- we get a new LSN/nonce for free if we are using the LSN as the > > nonce. What has ma

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 03:20:06PM -0400, Bruce Momjian wrote: > > Also, when you change hint bits, either you don't change the nonce/LSN, > > and don't re-encrypt the page (and the hint bit changes are visible), or > > you change the nonce an

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 04:29:08PM -0400, Stephen Frost wrote: > Greetings, > > On Tue, May 25, 2021 at 14:56 Bruce Momjian wrote: > > On Tue, May 25, 2021 at 02:25:21PM -0400, Robert Haas wrote: > > One question here is whether we're comfortable saying that the nonce > > is entirely

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Andrew Dunstan
On 5/25/21 4:31 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 5/25/21 4:10 PM, Tom Lane wrote: >>> Also, even if ZSON was "100% compatible with JSONB" back in 2016, >>> a whole lot of features have been added since then. Having to >>> duplicate all that code again for a different data type

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 05:04:50PM -0400, Stephen Frost wrote: > > Now, if we want to consult some security experts and have them tell us > > the hint bit visibility is not a problem, we could get by without using a > > new nonce for hint bit changes, and in that case it doesn't matter if we > > ha

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 01:54:21PM -0700, Andres Freund wrote: > > On 2021-05-25 15:34:04 -0400, Bruce Momjian wrote: > > > My point is that we have to full-page-write cases where we change the > > > nonce --- we get a new LSN/nonce for free i

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 05:14:24PM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > Yes, I can see that happening. I think occasional leakage of hint bit > > changes to be acceptable. We might decide they are all acceptable. > > I don't think that I agree with the idea

  1   2   >