Re: Skipping logical replication transactions on subscriber side

2021-05-28 Thread Amit Kapila
On Sat, May 29, 2021 at 8:27 AM Masahiko Sawada wrote: > > On Thu, May 27, 2021 at 7:04 PM Amit Kapila wrote: > > > > On Thu, May 27, 2021 at 1:46 PM Masahiko Sawada > > wrote: > > > > > > 1. the worker records the XID and commit LSN of the failed transaction > > > to a catalog. > > > > > > > W

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Amit Kapila
On Fri, May 28, 2021 at 5:16 PM Tomas Vondra wrote: > > I wonder if there's a way to free the TOASTed data earlier, instead of > waiting until the end of the transaction (as this patch does). > IIUC we are anyway freeing the toasted data at the next insert/update/delete. We can try to free at oth

Re: Parallel Inserts in CREATE TABLE AS

2021-05-28 Thread Amit Kapila
On Fri, May 28, 2021 at 8:53 AM Amit Kapila wrote: > > On Thu, May 27, 2021 at 7:37 PM Bharath Rupireddy > > > > I captured below information with the attached patch > > 0001-test-times-and-block-counts.patch applied on top of CTAS v23 > > patch set. Testing details are attached in the file named

Regarding the necessity of RelationGetNumberOfBlocks for every rescan / bitmap heap scan.

2021-05-28 Thread Andy Fan
Hi: I'm always confused about the following codes. static void initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) { ParallelBlockTableScanDesc bpscan = NULL; bool allow_strat; bool allow_sync; /* * Determine the number of blocks we have to scan. * * It is sufficient to do this once a

Re: Skipping logical replication transactions on subscriber side

2021-05-28 Thread Masahiko Sawada
On Thu, May 27, 2021 at 7:04 PM Amit Kapila wrote: > > On Thu, May 27, 2021 at 1:46 PM Masahiko Sawada wrote: > > > > On Thu, May 27, 2021 at 2:48 PM Amit Kapila wrote: > > > > > > Okay, that makes sense but still not sure how will you identify if we > > > need to reset XID in case of failure do

Re: Skipping logical replication transactions on subscriber side

2021-05-28 Thread Masahiko Sawada
On Thu, May 27, 2021 at 7:26 PM Amit Kapila wrote: > > On Thu, May 27, 2021 at 12:01 PM Masahiko Sawada > wrote: > > > > On Wed, May 26, 2021 at 6:11 PM Amit Kapila wrote: > > > > > > I agree with you that specifying XID could be easier and > > > understandable for users. I was thinking and stu

ANALYZE's dead tuple accounting can get confused

2021-05-28 Thread Peter Geoghegan
The accounting used by ANALYZE to count dead tuples in acquire_sample_rows() (actually in heapam_scan_analyze_next_tuple() these days) makes some dubious assumptions about how it should count dead tuples. This is something that I discussed with Masahiko in the context of our Postgres 14 work on VAC

Re: Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Johannes Graën
On 28/05/2021 18.24, Tom Lane wrote: > Pavel Stehule writes: >> pá 28. 5. 2021 v 16:12 odesílatel Johannes Graën >> napsal: >>> When trying to upgrade an existing database from version 10 to 13 I came >>> across a degression in some existing code used by clients. Further >>> investigations showed

Re: Add option --drop-cascade for pg_dump/restore

2021-05-28 Thread Greg Sabino Mullane
Overall the patch looks good, but I did notice a few small things: 1. In pg_dumpall.c, the section /* Add long options to the pg_dump argument list */, we are now passing along the --drop-cascade option. However, --clean is not passed in, so any call to pg_dumpall using --drop-cascade fails a

Re: Delegating superuser tasks to new security roles (Was: Granting control of SUSET gucs to non-superusers)

2021-05-28 Thread Mark Dilger
> On May 27, 2021, at 11:06 PM, Noah Misch wrote: > > On Tue, May 25, 2021 at 01:33:54PM -0700, Mark Dilger wrote: >> v3-0001 adds a new pg_logical_replication role with permission to manage >> publications and subscriptions. > >> v3-0004 adds a new pg_database_security role with permission

Re: terminate called after throwing an instance of 'std::bad_alloc'

2021-05-28 Thread Justin Pryzby
There's a memory leak affecting JIT expressions, even when inlining, optimization, and tuple deforming are disabled. The server that got OOM after installing PG13.3 (because your patch wasn't applied) still gets OOM even with inline_above_cost=-1, optimize_above_cost=-1, and deforming=off. Actual

Re: Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Tom Lane
Pavel Stehule writes: > pá 28. 5. 2021 v 16:12 odesílatel Johannes Graën > napsal: >> When trying to upgrade an existing database from version 10 to 13 I came >> across a degression in some existing code used by clients. Further >> investigations showed that performance measures are similar in ve

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Tomas Vondra
On 5/28/21 4:22 PM, Andrew Dunstan wrote: > > On 5/28/21 6:35 AM, Tomas Vondra wrote: >> >>> >>> IMO the main benefit of having different dictionaries is that you >>> could have a small dictionary for small and very structured JSONB >>> fields (e.g. some time-series data), and a large one for l

Re: Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Pavel Stehule
hI pá 28. 5. 2021 v 16:12 odesílatel Johannes Graën napsal: > Hi, > > When trying to upgrade an existing database from version 10 to 13 I came > across a degression in some existing code used by clients. Further > investigations showed that performance measures are similar in versions > 11 to 13

Re: Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Tomas Vondra
On 5/28/21 4:12 PM, Johannes Graën wrote: > Hi, > > When trying to upgrade an existing database from version 10 to 13 I came > across a degression in some existing code used by clients. Further > investigations showed that performance measures are similar in versions > 11 to 13, while in the origi

Re: be-secure-gssapi.c and auth.c with setenv() not compatible on Windows

2021-05-28 Thread Tom Lane
Michael Paquier writes: > We can do two things here: > 1) Switch be-secure-gssapi.c and auth.c to use putenv(). > 2) Backport into 12 and 13 the fallback implementation of setenv > introduced in 7ca37fb, and keep be-secure-gssapi.c as they are now. There's a lot of value in keeping the branches l

Re: Command statistics system (cmdstats)

2021-05-28 Thread Mark Dilger
> On May 28, 2021, at 6:42 AM, Alvaro Herrera wrote: > On Mar 4, 2020, at 7:43 PM, Mark Dilger wrote: > as mentioned in [1], I have created an implementation of command counter statistics very similar in purpose to the one already pending in the commitfest going by

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Andrew Dunstan
On 5/28/21 6:35 AM, Tomas Vondra wrote: > >> >> IMO the main benefit of having different dictionaries is that you >> could have a small dictionary for small and very structured JSONB >> fields (e.g. some time-series data), and a large one for large / >> unstructured JSONB fields, without having t

Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Johannes Graën
Hi, When trying to upgrade an existing database from version 10 to 13 I came across a degression in some existing code used by clients. Further investigations showed that performance measures are similar in versions 11 to 13, while in the original database on version 10 it's around 100 times faste

Re: Reduce lock level for ALTER TABLE ... ADD CHECK .. NOT VALID

2021-05-28 Thread Greg Sabino Mullane
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed Looks fine to me The new status of this patch is: Ready for

Re: Asynchronous Append on postgres_fdw nodes.

2021-05-28 Thread Etsuro Fujita
Horiguchi-san, On Fri, May 28, 2021 at 5:29 PM Kyotaro Horiguchi wrote: > At Fri, 28 May 2021 16:30:29 +0900, Etsuro Fujita > wrote in > > The root cause would > > be that we call classify_matching_subplans() to re-determine > > sync/async subplans when called from the first ExecAppend() after

Re: Command statistics system (cmdstats)

2021-05-28 Thread Alvaro Herrera
> >> On Mar 4, 2020, at 7:43 PM, Mark Dilger > >> wrote: > >> as mentioned in [1], I have created an implementation of command > >> counter statistics very similar in purpose to the one already > >> pending in the commitfest going by the name "pg_stat_sql". I don't > >> really care if this impl

Re: storing an explicit nonce

2021-05-28 Thread Bruce Momjian
On Thu, May 27, 2021 at 04:36:23PM -0400, Stephen Frost wrote: > At this point I'm wondering if it's just: > > dboid/relfilenode:block-offset > > and then we hash it to whatever size EVP_CIPHER_iv_length(AES-XTS-128) > (or -256, whatever we're using based on what was passed to initdb) > returns.

Re: [HACKERS] EvalPlanQual behaves oddly for FDW queries involving system columns

2021-05-28 Thread Etsuro Fujita
On Fri, May 28, 2021 at 8:25 AM Alvaro Herrera wrote: > On 2019-Mar-15, Etsuro Fujita wrote: > > (2019/03/04 12:10), Etsuro Fujita wrote: > > > (2019/03/02 3:57), Andres Freund wrote: > > > > FWIW, I pushed the EPQ patch, doing this conversion blindly. It'd be > > > > awesome if you'd check that i

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Amit Kapila
On Fri, May 28, 2021 at 6:01 PM Tomas Vondra wrote: > > On 5/28/21 2:17 PM, Dilip Kumar wrote: > > On Fri, May 28, 2021 at 5:16 PM Tomas Vondra > > wrote: > >> On 5/27/21 6:36 AM, Dilip Kumar wrote: > >>> On Thu, May 27, 2021 at 9:47 AM Amit Kapila > >>> wrote: > > On Thu, May 27, 202

be-secure-gssapi.c and auth.c with setenv() not compatible on Windows

2021-05-28 Thread Michael Paquier
Hi all, Now that hamerkop has been fixed and that we have some coverage with builds of GSSAPI on Windows thanks to 02511066, the buildfarm has been complaining about a build failure on Windows for 12 and 13: https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=hamerkop&dt=2021-05-28%2011%

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Tomas Vondra
On 5/28/21 2:17 PM, Dilip Kumar wrote: > On Fri, May 28, 2021 at 5:16 PM Tomas Vondra > wrote: >> On 5/27/21 6:36 AM, Dilip Kumar wrote: >>> On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: True, but if you do this clean-up

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Dilip Kumar
On Fri, May 28, 2021 at 5:16 PM Tomas Vondra wrote: > On 5/27/21 6:36 AM, Dilip Kumar wrote: > > On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: > >> > >> On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: > >> > >> True, but if you do this clean-up in ReorderBufferCleanupTXN then you > >> do

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Tomas Vondra
On 5/27/21 6:36 AM, Dilip Kumar wrote: > On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: >> >> On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: >> >> True, but if you do this clean-up in ReorderBufferCleanupTXN then you >> don't need to take care at separate places. Also, toast_hash is st

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Tomas Vondra
On 5/27/21 4:15 AM, Andrew Dunstan wrote: > > On 5/26/21 5:29 PM, Bruce Momjian wrote: >> On Tue, May 25, 2021 at 01:55:13PM +0300, Aleksander Alekseev wrote: >>> Hi hackers, >>> >>> Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. >>> The >>> extension introduces th

Re: [BUG]Update Toast data failure in logical replication

2021-05-28 Thread Dilip Kumar
On Fri, May 28, 2021 at 12:31 PM tanghy.f...@fujitsu.com wrote: > > On Friday, May 28, 2021 3:02 PM, tanghy.f...@fujitsu.com wrote: > > FYI. The problem also occurs in PG-13. I will try to check from which > > version it > > got introduced. > > I reproduced it in PG-10,11,12,13. > I think the pro

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Tomas Vondra
On 5/26/21 2:49 AM, Stephen Frost wrote: > Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Matthias van de Meent writes: >>> I like the idea of the ZSON type, but I'm somewhat disappointed by its >>> current limitations: >> >> I've not read the code, so maybe this thought is completel

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Tomas Vondra
On 5/26/21 6:43 PM, Matthias van de Meent wrote: > On Wed, 26 May 2021 at 12:49, Aleksander Alekseev > wrote: >> >> Hi hackers, >> >> Many thanks for your feedback, I very much appreciate it! >> >>> If the extension is mature enough, why make it an extension in >>> contrib, and not instead eith

RE: Parallel Inserts in CREATE TABLE AS

2021-05-28 Thread houzj.f...@fujitsu.com
From: Bharath Rupireddy Sent: Thursday, May 27, 2021 10:07 PM > On Thu, May 27, 2021 at 9:53 AM Bharath Rupireddy > wrote: > > > One idea to find this out could be that we have three counters for > > > each worker which counts the number of times each worker extended > > > the relation in bulk, t

Re: Parallel INSERT SELECT take 2

2021-05-28 Thread Greg Nancarrow
On Mon, May 24, 2021 at 3:15 PM houzj.f...@fujitsu.com wrote: > > > Thanks for the comments and your descriptions looks good. > Attaching v5 patchset with all these changes. > A few other minor things I noticed: (1) error message wording when declaring a table SAFE for parallel DML src/backend/

Re: Asynchronous Append on postgres_fdw nodes.

2021-05-28 Thread Kyotaro Horiguchi
At Fri, 28 May 2021 16:30:29 +0900, Etsuro Fujita wrote in > On Wed, Mar 31, 2021 at 6:55 PM Etsuro Fujita wrote: > > On Tue, Mar 30, 2021 at 8:40 PM Etsuro Fujita > > wrote: > > > I'm happy with the patch, so I'll commit it if there are no objections. > > > > Pushed. > > I noticed that resc

Re: Race condition in recovery?

2021-05-28 Thread Kyotaro Horiguchi
(Sorry for being a bit off-topic) At Fri, 28 May 2021 12:18:35 +0900, Tatsuro Yamada wrote in > Hi Horiguchi-san, (Why me?) > In a project I helped with, I encountered an issue where > the archive command kept failing. I thought this issue was > related to the problem in this thread, so I'm s

Re: Asynchronous Append on postgres_fdw nodes.

2021-05-28 Thread Etsuro Fujita
On Wed, Mar 31, 2021 at 6:55 PM Etsuro Fujita wrote: > On Tue, Mar 30, 2021 at 8:40 PM Etsuro Fujita wrote: > > I'm happy with the patch, so I'll commit it if there are no objections. > > Pushed. I noticed that rescan of async Appends is broken when do_exec_prune=false, leading to incorrect resu

RE: [BUG]Update Toast data failure in logical replication

2021-05-28 Thread tanghy.f...@fujitsu.com
On Friday, May 28, 2021 3:02 PM, tanghy.f...@fujitsu.com wrote: > FYI. The problem also occurs in PG-13. I will try to check from which version > it > got introduced. I reproduced it in PG-10,11,12,13. I think the problem has been existing since Logical replication introduced in PG-10. Regards