Re: Synchronizing slots from primary to standby

2022-02-23 Thread James Coleman
ccur without any attempt to trigger the catalog xmin problem. Given this error seems pretty significant in terms of indicating fundamental lack of test coverage (the primary stated benefit of the patch is physical failover), and it currently is a blocker to testing more deeply. Thanks, Jame

Consider parallel for lateral subqueries with limit

2020-11-30 Thread James Coleman
I've been investigating parallelizing certain correlated subqueries, and during that work stumbled across the fact that set_rel_consider_parallel disallows parallel query on what seems like a fairly simple case. Consider this query: select t.unique1 from tenk1 t join lateral (select t.unique1 fro

Re: enable_incremental_sort changes query behavior

2020-11-30 Thread James Coleman
On Tue, Nov 3, 2020 at 4:39 PM Tomas Vondra wrote: > I've pushed the 0001 part, i.e. the main fix. Not sure about the other > parts (comments and moving the code back to postgres_fdw) yet. I noticed the CF entry [1] got moved to the next CF; I'm thinking this entry should be marked as committed s

Re: [DOC] Document concurrent index builds waiting on each other

2020-11-30 Thread James Coleman
On Mon, Nov 30, 2020 at 4:53 PM Alvaro Herrera wrote: > > On 2020-Sep-30, Michael Paquier wrote: > > > + > > + CREATE INDEX (including the > > CONCURRENTLY > > + option) commands are included when VACUUM calculates > > what > > + dead tuples are safe to remove even on tables other than t

Re: Why does create_gather_merge_plan need make_sort?

2020-11-30 Thread James Coleman
On Sun, Nov 29, 2020 at 4:10 PM Tomas Vondra wrote: > > > > On 11/29/20 3:44 PM, James Coleman wrote: > > On Mon, Nov 23, 2020 at 8:19 AM James Coleman wrote: > >> > >> .. > >> > >> So I'm +1 on turning it into an ERROR log instead,

Re: Fix generate_useful_gather_paths for parallel unsafe pathkeys

2020-11-30 Thread James Coleman
On Sun, Nov 29, 2020 at 4:20 PM Tomas Vondra wrote: > > On 11/29/20 3:26 PM, James Coleman wrote: > > On Mon, Nov 23, 2020 at 8:35 AM James Coleman wrote: > >> > >> On Sun, Nov 22, 2020 at 4:59 PM Tomas Vondra > >> wrote: > >>> ... > >>

Re: Consider parallel for lateral subqueries with limit

2020-12-01 Thread James Coleman
On Mon, Nov 30, 2020 at 7:00 PM James Coleman wrote: > > I've been investigating parallelizing certain correlated subqueries, > and during that work stumbled across the fact that > set_rel_consider_parallel disallows parallel query on what seems like > a fairly simple cas

Re: [DOC] Document concurrent index builds waiting on each other

2020-12-01 Thread James Coleman
On Tue, Dec 1, 2020 at 6:51 PM Alvaro Herrera wrote: > > On 2020-Nov-30, James Coleman wrote: > > > On Mon, Nov 30, 2020 at 4:53 PM Alvaro Herrera > > wrote: > > > > > > On 2020-Sep-30, Michael Paquier wrote: > > > > Yeah, I

Re: Change definitions of bitmap flags to bit-shifting style

2020-12-06 Thread James Coleman
On Sun, Dec 6, 2020 at 1:25 AM Michael Paquier wrote: > On Sat, Dec 05, 2020 at 10:31:09PM -0300, Alvaro Herrera wrote: > > The hexadecimal representation is more natural to me than bit-shifting, > > so I would prefer to use that style too. But maybe I'm trained to it > > because of looking at t

Re: Consider parallel for lateral subqueries with limit

2020-12-07 Thread James Coleman
Brian's email didn't keep the relevant headers, and so didn't show up as a reply, so I've pasted it here and am replying in this original thread. You can find the original at [1]. On Sun, Dec 6, 2020 at 7:34 PM Brian Davis wrote: > > > Note that near the end of grouping planner we have a similar

Re: Insert Documentation - Returning Clause and Order

2020-12-12 Thread James Coleman
On Friday, December 11, 2020, David G. Johnston wrote: > > On Fri, Dec 11, 2020 at 6:24 AM Ashutosh Bapat > wrote: >> >> On Thu, Dec 10, 2020 at 7:49 PM David G. Johnston >> wrote: >> >> > Yeah, the ongoing work on parallel inserts would seem to be an issue. We >> > should probably document t

Re: Insert Documentation - Returning Clause and Order

2020-12-12 Thread James Coleman
On Sat, Dec 12, 2020 at 10:11 AM David G. Johnston wrote: > > On Sat, Dec 12, 2020 at 7:02 AM James Coleman wrote: >> >> >> Certainly almost every ORM, and maybe even other forms of application >> code, need to be able to associate the serial column value ret

Re: [DOC] Document concurrent index builds waiting on each other

2020-12-18 Thread James Coleman
On Tue, Dec 1, 2020 at 8:05 PM James Coleman wrote: > > On Tue, Dec 1, 2020 at 6:51 PM Alvaro Herrera wrote: > > > > On 2020-Nov-30, James Coleman wrote: > > > > > On Mon, Nov 30, 2020 at 4:53 PM Alvaro Herrera > > > wrote: > > &

Re: Dependency isn't created between extension and schema

2020-12-21 Thread James Coleman
On Mon, Dec 21, 2020 at 2:59 AM Michael Paquier wrote: > > On Mon, Dec 21, 2020 at 04:02:29PM +0900, Masahiko Sawada wrote: > > Is it a bug? Since the created schema obviously depends on the > > extension when we created the schema specified in the schema option, I > > think we might want to creat

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-05 Thread James Coleman
On Fri, Mar 5, 2021 at 12:36 PM Fujii Masao wrote: > > > > On 2021/03/05 22:45, David Steele wrote: > > Hi Fujii, > > > > On 9/8/20 1:17 PM, James Coleman wrote: > >> On Tue, Aug 18, 2020 at 12:25 PM Fujii Masao > >> wrote: > >>> Thank

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-09 Thread James Coleman
On Tue, Mar 9, 2021 at 8:47 AM Alvaro Herrera wrote: > > On 2021-Mar-07, Magnus Hagander wrote: > > > On Sun, Mar 7, 2021 at 3:39 PM Fujii Masao > > wrote: > > > > > > Here's an idea: > > > > > > > > * hot_standby=on, before reaching consistent state > > > >FATAL: database is not accepting

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-09 Thread James Coleman
On Tue, Mar 9, 2021 at 9:07 AM Alvaro Herrera wrote: > > On 2021-Mar-09, James Coleman wrote: > > > On Tue, Mar 9, 2021 at 8:47 AM Alvaro Herrera > > wrote: > > > > > > On 2021-Mar-07, Magnus Hagander wrote: > > > > > > >

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-09 Thread James Coleman
On Tue, Mar 9, 2021 at 9:17 AM Alvaro Herrera wrote: > > On 2021-Mar-09, James Coleman wrote: > > > Yes, I think they both agreed on the "DETAIL: Hot standby mode is > > disabled." message, but that alternative meant not needing to add any > > new signals

Re: enable_incremental_sort changes query behavior

2020-10-30 Thread James Coleman
On Thu, Oct 29, 2020 at 6:06 PM Tomas Vondra wrote: > > On Tue, Oct 06, 2020 at 09:37:31AM -0400, James Coleman wrote: > >On Tue, Oct 6, 2020 at 9:28 AM Jaime Casanova > > wrote: > >> Can you please create an entry in the commitfest for this one so we > >> d

Re: enable_incremental_sort changes query behavior

2020-10-30 Thread James Coleman
On Fri, Oct 30, 2020 at 5:03 PM Tomas Vondra wrote: > > On Fri, Oct 30, 2020 at 01:26:08PM -0400, James Coleman wrote: > >On Thu, Oct 29, 2020 at 6:06 PM Tomas Vondra > > wrote: > >> > >> On Tue, Oct 06, 2020 at 09:37:31AM -0400, James Coleman wrote: > &

Re: Use of "long" in incremental sort code

2020-11-04 Thread James Coleman
On Tue, Nov 3, 2020 at 4:42 PM Tomas Vondra wrote: > > On Tue, Nov 03, 2020 at 03:53:53AM +0100, Tomas Vondra wrote: > >Hi, > > > >I took another look at this, and 99% of the patch (the fixes to sort > >debug messages) seems fine to me. Attached is the part I plan to get > >committed, including c

Re: enable_incremental_sort changes query behavior

2020-11-17 Thread James Coleman
On Mon, Nov 16, 2020 at 11:23 PM Tomas Vondra wrote: > > Hmm, I missed that other thread. That indeed seems like a bug in the > same area already tweaked by ebb7ae839d033d0f2 for similar cases. I meant to bring it up in this thread before we got the other patch committed, but I just ended up not

Re: enable_incremental_sort changes query behavior

2020-11-20 Thread James Coleman
On Tue, Nov 17, 2020 at 11:20 AM Tomas Vondra wrote: > > > > On 11/17/20 3:03 PM, James Coleman wrote: > > On Mon, Nov 16, 2020 at 11:23 PM Tomas Vondra > > wrote: > >> > >> Hmm, I missed that other thread. That indeed seems like a bug in the > >

Re: enable_incremental_sort changes query behavior

2020-11-20 Thread James Coleman
On Fri, Nov 20, 2020 at 12:06 PM Robert Haas wrote: > > On Wed, Oct 7, 2020 at 6:22 PM Tomas Vondra > wrote: > > I'm still not entirely sure I understand what's happening, or what the > > exact rule is. Consider this query: > > > >explain (verbose) select distinct i, t, md5(t) from ref_0; > >

Re: enable_incremental_sort changes query behavior

2020-11-20 Thread James Coleman
Thanks much for the detailed followup; this is super helpful. On Fri, Nov 20, 2020 at 2:57 PM Robert Haas wrote: > > On Fri, Nov 20, 2020 at 1:51 PM James Coleman wrote: > > > This isn't a counterexample, because there's no join tree here -- or, > > > well, t

Why does create_gather_merge_plan need make_sort?

2020-11-20 Thread James Coleman
While looking at another issue I noticed that create_gather_merge_plan calls make_sort if the subplan isn't sufficiently sorted. In all of the cases I've seen where a gather merge path (not plan) is created the input path is expected to be properly sorted, so I was wondering if anyone happened to k

Fix generate_useful_gather_paths for parallel unsafe pathkeys

2020-11-20 Thread James Coleman
Kjc%2B8LuQa49EHW4%2BKfefb3wHqPYFnCuUqozo%2BLFg%40mail.gmail.com 4: https://www.postgresql.org/message-id/CA%2BTgmobX2079GNJWJVFjo5CmwTg%3DoQQOybFQL2CyZxMY59P6BA%40mail.gmail.com From 11cb504dca05d93e40a97fa4fe13202a59664950 Mon Sep 17 00:00:00 2001 From: James Coleman Date: Fri, 20 Nov 2020 20:29:51 -0500 Subject: [PATCH v1 2

Re: Why does create_gather_merge_plan need make_sort?

2020-11-23 Thread James Coleman
On Sun, Nov 22, 2020 at 5:07 PM Tomas Vondra wrote: > > On 11/22/20 10:31 PM, Tom Lane wrote: > > Tomas Vondra writes: > >> On 11/20/20 11:24 PM, James Coleman wrote: > >>> While looking at another issue I noticed that create_gather_merge_plan > >

Re: Fix generate_useful_gather_paths for parallel unsafe pathkeys

2020-11-23 Thread James Coleman
On Sun, Nov 22, 2020 at 4:59 PM Tomas Vondra wrote: > > On 11/21/20 2:55 AM, James Coleman wrote: > > Over on the -bugs list we had a report [1] of a seg fault with > > incremental sort. The short of the investigation there was that a > > subplan wasn't being serial

Re: enable_incremental_sort changes query behavior

2020-11-23 Thread James Coleman
On Mon, Nov 23, 2020 at 2:24 PM Tom Lane wrote: > > James Coleman writes: > > But I think that still leaves something missing: after all, > > prepare_sort_from_pathkeys does know how to insert new target list > > entries, so something either there (or in the caller/how i

Re: enable_incremental_sort changes query behavior

2020-11-25 Thread James Coleman
On Tue, Nov 24, 2020 at 2:31 PM Tom Lane wrote: > > James Coleman writes: > > On Mon, Nov 23, 2020 at 2:24 PM Tom Lane wrote: > >> 1. James was wondering, far upthread, why we would do projections > >> pre-sort or post-sort. I think the answers can be foun

Re: enable_incremental_sort changes query behavior

2020-11-25 Thread James Coleman
On Wed, Nov 25, 2020 at 2:53 PM James Coleman wrote: > > On Tue, Nov 24, 2020 at 2:31 PM Tom Lane wrote: > > > > James Coleman writes: > > > On Mon, Nov 23, 2020 at 2:24 PM Tom Lane wrote: > > >> 1. James was wondering, far upthread, why we would do proje

Re: Fix generate_useful_gather_paths for parallel unsafe pathkeys

2020-11-29 Thread James Coleman
On Mon, Nov 23, 2020 at 8:35 AM James Coleman wrote: > > On Sun, Nov 22, 2020 at 4:59 PM Tomas Vondra > wrote: > > ... > > Thanks for the patches, I'll take a closer look next week. The 0002 > > patch is clearly something we need to backpatch, not sure about 00

Re: Why does create_gather_merge_plan need make_sort?

2020-11-29 Thread James Coleman
On Mon, Nov 23, 2020 at 8:19 AM James Coleman wrote: > > On Sun, Nov 22, 2020 at 5:07 PM Tomas Vondra > wrote: > > > > On 11/22/20 10:31 PM, Tom Lane wrote: > > > Tomas Vondra writes: > > >> On 11/20/20 11:24 PM, James Coleman wrote: > > &

Re: [DOC] Document concurrent index builds waiting on each other

2021-01-13 Thread James Coleman
On Wed, Jan 13, 2021 at 12:58 AM Michael Paquier wrote: > > On Tue, Jan 12, 2021 at 04:51:39PM -0300, Alvaro Herrera wrote: > > I looked into this again, and I didn't like what I had added to > > maintenance.sgml at all. It seems out of place where I put it; and I > > couldn't find any great spot

Re: [DOC] Document concurrent index builds waiting on each other

2021-01-13 Thread James Coleman
On Wed, Jan 13, 2021 at 12:33 PM Alvaro Herrera wrote: > > On 2021-Jan-13, James Coleman wrote: > > > On Wed, Jan 13, 2021 at 12:58 AM Michael Paquier > > wrote: > > > > > > On Tue, Jan 12, 2021 at 04:51:39PM -0300, Alvaro Herrera wrote: > > > &g

Re: [DOC] Document concurrent index builds waiting on each other

2021-01-13 Thread James Coleman
On Wed, Jan 13, 2021 at 4:05 PM Alvaro Herrera wrote: > > On 2021-Jan-13, James Coleman wrote: > > > On Wed, Jan 13, 2021 at 12:33 PM Alvaro Herrera > > wrote: > > > > This is true. So I propose > > > > > > Like any long-running transactio

Re: [DOC] Document concurrent index builds waiting on each other

2021-01-13 Thread James Coleman
On Wed, Jan 13, 2021 at 4:29 PM Tom Lane wrote: > > Alvaro Herrera writes: > > On 2021-Jan-13, James Coleman wrote: > >>>> This is true. So I propose > >>>> Like any long-running transaction, REINDEX can > >>>> affect which tuples can be

Re: [DOC] Document concurrent index builds waiting on each other

2021-01-13 Thread James Coleman
On Wed, Jan 13, 2021 at 5:00 PM Tom Lane wrote: > > James Coleman writes: > > On Wed, Jan 13, 2021 at 4:29 PM Tom Lane wrote: > >> but the antecedent of "a table" is a bit unclear; people might > >> wonder if it means the table being reindexed. > >

Re: Discarding DISCARD ALL

2021-01-20 Thread James Coleman
I hope to do further review of the patch later this week, but I wanted to at least comment on this piece: On Wed, Jan 20, 2021 at 2:48 AM Peter Eisentraut wrote: > > On 2020-12-23 15:33, Simon Riggs wrote: > > Poolers such as pgbouncer would then be able to connect transaction > > mode pools by s

Re: Discarding DISCARD ALL

2021-01-20 Thread James Coleman
On Wed, Jan 20, 2021 at 9:58 AM Simon Riggs wrote: > > On Wed, 20 Jan 2021 at 14:21, James Coleman wrote: > > > An alternative approach that occurred to me while typing this reply: a > > setting in Postgres that would disallow setting session level GUCs > > (i.e., enfo

Re: Consider parallel for lateral subqueries with limit

2022-01-04 Thread James Coleman
when it's a lateral reference. That limit/scan (if lateral) has to be being executed per tuple in the outer scan, right? And if it's a unique execution per tuple, then consistency across tuples (that are in different workers) can't be a concern. Is there a scenario I'm missing where lateral can currently be executed in that way in that structure (or a different one)? Thanks, James Coleman

Re: Consider parallel for lateral subqueries with limit

2022-01-13 Thread James Coleman
On Tue, Jan 4, 2022 at 9:59 PM James Coleman wrote: > > On Tue, Jan 4, 2022 at 5:31 PM Tom Lane wrote: > > > > Greg Nancarrow writes: > > > The patch LGTM. > > > I have set the status to "Ready for Committer". > > > > I don't really

Re: Parallelize correlated subqueries that execute within each worker

2022-01-14 Thread James Coleman
On Fri, Dec 3, 2021 at 2:35 AM Michael Paquier wrote: > > On Mon, Nov 15, 2021 at 10:01:37AM -0500, Robert Haas wrote: > > On Wed, Nov 3, 2021 at 1:34 PM James Coleman wrote: > >> As I understand the current code, parallel plans are largely chosen > >> based not

Re: Parallelize correlated subqueries that execute within each worker

2022-01-14 Thread James Coleman
On Mon, Nov 15, 2021 at 10:01 AM Robert Haas wrote: > > On Wed, Nov 3, 2021 at 1:34 PM James Coleman wrote: > > As I understand the current code, parallel plans are largely chosen > > based not on where it's safe to insert a Gather node but rather by > > determinin

Add last commit LSN to pg_last_committed_xact()

2022-01-14 Thread James Coleman
on't write it out to disk). That the downside of making this feature dependent on "track_commit_timestamps = on", but that seems reasonable: 1. Getting the xid of the last commit is similarly dependent on commit timestamps infrastructure. 2. It's a simple place to hook into

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-19 Thread James Coleman
On Tue, Mar 9, 2021 at 9:27 AM Fujii Masao wrote: > > > > On 2021/03/09 23:19, James Coleman wrote: > > On Tue, Mar 9, 2021 at 9:17 AM Alvaro Herrera > > wrote: > >> > >> On 2021-Mar-09, James Coleman wrote: > >> > >>> Yes

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-03-19 Thread James Coleman
On Fri, Sep 11, 2020 at 5:11 PM James Coleman wrote: > > On Tue, Sep 8, 2020 at 4:37 PM Heikki Linnakangas wrote: > > > > On 08/09/2020 22:25, James Coleman wrote: > > > On Wed, Aug 19, 2020 at 3:16 AM Heikki Linnakangas > > > wrote: > > >> >

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-22 Thread James Coleman
On Mon, Mar 22, 2021 at 1:49 PM Fujii Masao wrote: > > > > On 2021/03/19 23:35, James Coleman wrote: > > Here's an updated patch; I think I've gotten what Alvaro suggested. > > Thanks for updating the patch! But I was thinking that our consensus is > somethi

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-22 Thread James Coleman
On Mon, Mar 22, 2021 at 2:52 PM Fujii Masao wrote: > > > > On 2021/03/23 3:25, James Coleman wrote: > > On Mon, Mar 22, 2021 at 1:49 PM Fujii Masao > > wrote: > >> > >> > >> > >> On 2021/03/19 23:35, James Coleman wrote: > >>

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-23 Thread James Coleman
On Tue, Mar 23, 2021 at 1:46 AM Fujii Masao wrote: > > > > On 2021/03/23 3:59, James Coleman wrote: > > Are you saying we should only change the message for a single case: > > the case where we'd otherwise allow connections but EnableHotStandby > > is false

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-23 Thread James Coleman
say "... is not accepting connections *yet*". > > +1, but I think "... is not yet accepting connections" is slightly > better style. All right, see attached v8. James Coleman v8-0001-Improve-standby-connection-denied-error-message.patch Description: Binary data

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-24 Thread James Coleman
essage reported at PM_STARTUP to that one, > based on v8 patch that James posted upthread. I also ran pgindent for > the patch. Attached is the updated version of the patch. > > Barring any objection, I will commit this. That looks good to me. Thanks for working on this. James Coleman

Re: [EXTERNAL] Any objection to documenting pg_sequence_last_value()?

2021-04-06 Thread James Coleman
elp document this function if there are no objections. > > Best, > Hanefi > > -Original Message- > From: James Coleman > Sent: 6 Ağustos 2020 Perşembe 16:14 > To: pgsql-hackers > Subject: [EXTERNAL] Any objection to documenting pg_sequence_last_value()? > > T

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-06 Thread James Coleman
On Mon, Apr 5, 2021 at 11:58 PM David Rowley wrote: > > On Sat, 20 Mar 2021 at 09:41, James Coleman wrote: > > I've attached a cleaned up patch. Last CF it was listed in is > > https://commitfest.postgresql.org/29/2542/ -- what's the appropriate > > step

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-08 Thread James Coleman
On Thu, Apr 8, 2021 at 8:01 AM David Rowley wrote: > > On Thu, 8 Apr 2021 at 22:54, David Rowley wrote: > > > > I think the changes in the patch are fairly isolated and the test > > coverage is now pretty good. I'm planning on looking at the patch > > again now and will consider pushing it for P

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-08 Thread James Coleman
On Thu, Apr 8, 2021 at 1:04 PM Alvaro Herrera wrote: > > On 2021-Apr-08, James Coleman wrote: > > > I assume proper procedure for the CF entry is to move it into the > > current CF and then mark it as committed, however I don't know how (or > > don't have perm

Processing btree walks as a batch to parallelize IO

2021-04-09 Thread James Coleman
$SUBJECT is still a very loosely formed idea, so forgive lack of detail or things I've likely missed, but I wanted to get it out there to see if it sounded at all intriguing to people. Background: One of the big problems with non-local storage such as AWS EBS volumes or a SAN is that in a large da

Re: Nicer error when connecting to standby with hot_standby=off

2021-04-09 Thread James Coleman
On Wed, Mar 24, 2021 at 9:43 PM Fujii Masao wrote: > > > > On 2021/03/24 22:06, James Coleman wrote: > > That looks good to me. Thanks for working on this. > > Thanks! I pushed the patch. > > Regards, > > -- > Fujii Masao > Advanced Computing Te

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-12 Thread James Coleman
On Mon, Apr 12, 2021 at 7:49 PM David Rowley wrote: > > On Tue, 13 Apr 2021 at 11:42, Tom Lane wrote: > > > > David Rowley writes: > > > I realised when working on something unrelated last night that we can > > > also do hash lookups for NOT IN too. > > > > ... and still get the behavior right f

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-13 Thread James Coleman
On Mon, Apr 12, 2021 at 10:07 PM James Coleman wrote: > > On Mon, Apr 12, 2021 at 7:49 PM David Rowley wrote: > > > > On Tue, 13 Apr 2021 at 11:42, Tom Lane wrote: > > > > > > David Rowley writes: > > > > I realised when working on something

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread James Coleman
On Tue, Jul 28, 2020 at 11:10 AM Justin Pryzby wrote: > > An internal server aborted last night while running a maintenance script. I > reproduced this easily running the crashing command in a loop, and verified > this is a live issue on REL_13_STABLE (dc6f2fb43). > REINDEX INDEX pg_class_tblspc_

Re: HashAgg's batching counter starts at 0, but Hash's starts at 1. (now: incremental sort)

2020-07-30 Thread James Coleman
On Thu, Jul 30, 2020 at 8:22 PM Justin Pryzby wrote: > On Wed, Jul 29, 2020 at 09:18:44PM -0700, Peter Geoghegan wrote: > > On Wed, Jul 29, 2020 at 9:05 PM Justin Pryzby > wrote: > > > So my 2ndary suggestion is to conditionalize based on the method > rather than > > > value of space used. > > >

Re: Use of "long" in incremental sort code

2020-07-31 Thread James Coleman
On Thu, Jul 30, 2020 at 10:12 PM David Rowley wrote: > > On Fri, 3 Jul 2020 at 07:47, James Coleman wrote: > > Patch using int64 attached. > > I added this to the open items list for PG13. > > David I'd previously attached a patch [1], and there seemed to be agreeme

Re: [DOC] Document concurrent index builds waiting on each other

2020-07-31 Thread James Coleman
On Thu, Jul 16, 2020 at 7:34 PM David Johnston wrote: > > The following review has been posted through the commitfest application: > make installcheck-world: not tested > Implements feature: not tested > Spec compliant: not tested > Documentation:tested, passed > > Jam

Re: Comment simplehash/dynahash trade-offs

2020-07-31 Thread James Coleman
On Mon, Jul 20, 2020 at 1:29 AM Thomas Munro wrote: > > On Fri, May 1, 2020 at 1:53 PM James Coleman wrote: > > In another thread [1] I'd mused that "there might be some value in a > > README or comments > > addition that would be a guide to what the various h

Re: Nicer error when connecting to standby with hot_standby=off

2020-07-31 Thread James Coleman
On Wed, Jul 29, 2020 at 11:24 AM Fujii Masao wrote: > > > > On 2020/04/03 22:49, James Coleman wrote: > > On Thu, Apr 2, 2020 at 5:53 PM David Zhang wrote: > >> > >> The following review has been posted through the commitfest application: > >> make in

Re: Comment simplehash/dynahash trade-offs

2020-08-01 Thread James Coleman
On Fri, Jul 31, 2020 at 8:17 PM Thomas Munro wrote: > > On Sat, Aug 1, 2020 at 7:22 AM James Coleman wrote: > > [v2 patch set] > > I ran it through pgindent which insisted on adding some newlines, I > manually replaced some spaces with tabs to match nearby lines, I added >

Re: pg13dev: explain partial, parallel hashagg, and memory use

2020-08-04 Thread James Coleman
On Tue, Aug 4, 2020 at 9:44 PM David Rowley wrote: > > On Wed, 5 Aug 2020 at 13:21, Justin Pryzby wrote: > > > > I'm testing with a customer's data on pg13dev and got output for which Peak > > Memory doesn't look right/useful. I reproduced it on 565f16902. > > Likely the sanity of those results

Any objection to documenting pg_sequence_last_value()?

2020-08-06 Thread James Coleman
The function pg_sequence_last_value() was added to underlie the pg_sequences view, and it's the only way I'm aware of from userspace to directly get the last value of a sequence globally (i.e., not within the current session like currval()/lastval()). Obviously you can join to the pg_sequences view

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-08-10 Thread James Coleman
On Mon, Aug 10, 2020 at 8:37 PM Tom Lane wrote: > > Alvaro Herrera writes: > > To recap: currently, any CREATE INDEX CONCURRENTLY will wait for all > > other CICs running concurrently to finish, because they can't be > > distinguished amidst other old snapshots. We can change things by > > havin

Re: massive FPI_FOR_HINT load after promote

2020-08-11 Thread James Coleman
On Tue, Aug 11, 2020 at 2:55 AM Masahiko Sawada wrote: > > On Tue, 11 Aug 2020 at 07:56, Alvaro Herrera wrote: > > > > Last week, James reported to us that after promoting a replica, some > > seqscan was taking a huge amount of time; on investigation he saw that > > there was a high rate of FPI_F

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-08-11 Thread James Coleman
On Tue, Aug 11, 2020 at 11:14 AM Tom Lane wrote: > > James Coleman writes: > > Why is a CIC in active index-building something we need to wait for? > > Wouldn't it fall under a similar kind of logic to the other snapshot > > types we can explicitly ignore?

Re: Nicer error when connecting to standby with hot_standby=off

2020-09-08 Thread James Coleman
On Tue, Aug 18, 2020 at 12:25 PM Fujii Masao wrote: > Thanks for updating the patch! But it failed to be applied to the master > branch > cleanly because of the recent commit 0038f94387. So could you update the patch > again? Updated patch attached. James v3-0001-Improve-standby-connection-de

Re: [DOC] Document concurrent index builds waiting on each other

2020-09-08 Thread James Coleman
On Fri, Jul 31, 2020 at 2:51 PM James Coleman wrote: > > On Thu, Jul 16, 2020 at 7:34 PM David Johnston > wrote: > > > > The following review has been posted through the commitfest application: > > make installcheck-world: not tested > > Implements feature:

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-09-08 Thread James Coleman
On Sat, Aug 29, 2020 at 8:06 PM Tom Lane wrote: > > Alvaro Herrera writes: > > I pushed despite the objection because it seemed that downstream > > discussion was largely favorable to the change, and there's a different > > proposal to solve the bloat problem for analyze; and also: > > Note that

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2020-09-08 Thread James Coleman
On Wed, Aug 19, 2020 at 3:16 AM Heikki Linnakangas wrote: > > On 01/05/2020 05:20, James Coleman wrote: > > On Tue, Apr 28, 2020 at 8:25 AM Tomas Vondra > > wrote: > > ... > >> Any particular reasons to pick dynahash over simplehash? ISTM we're > >

Re: Parallelize correlated subqueries that execute within each worker

2021-11-03 Thread James Coleman
On Wed, Sep 8, 2021 at 8:47 AM James Coleman wrote: > See updated patch series attached. Jaime, I noticed on 3-October you moved this into "waiting on author"; I don't see anything waiting in this thread, however. Am I missing something? I'm planning to change i

Re: Consider parallel for lateral subqueries with limit

2021-11-03 Thread James Coleman
On Fri, Jul 16, 2021 at 3:16 PM James Coleman wrote: > > On Thu, May 27, 2021 at 9:01 PM Greg Nancarrow wrote: > > > > On Tue, Dec 8, 2020 at 10:46 AM James Coleman wrote: > > > > > > While I haven't actually tracked down to guarantee this is handled &

Re: Parallelize correlated subqueries that execute within each worker

2021-11-03 Thread James Coleman
er than anyone. > > On Fri, May 7, 2021 at 12:30 PM James Coleman wrote: > > The basic idea is that we need to track (both on nodes and relations) > > not only whether that node or rel is parallel safe but also whether > > it's parallel safe assuming params are rechecke

Re: Misleading comment in tuplesort_set_bound

2019-09-05 Thread James Coleman
Yes, planning on it, just a bit behind right now so will likely be a few more days at least. On Thu, Sep 5, 2019 at 4:57 PM Alvaro Herrera from 2ndQuadrant wrote: > > On 2019-Aug-26, Tom Lane wrote: > > > James Coleman writes: > > > I think the comment is fine as-is. P

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-12 Thread James Coleman
s are useful? I'm not > sure what the next steps are for this patch. I wanted to note here that I haven't abandoned this patch, but ended up needing to use my extra time for working on a conference talk. That talk is today, so I'm hoping to be able to catch up on this again soon. James Coleman

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-13 Thread James Coleman
On Fri, Sep 13, 2019 at 10:54 AM Tomas Vondra wrote: > > On Thu, Sep 12, 2019 at 11:54:06AM -0400, James Coleman wrote: > >> OK, so we have that now. I suppose this spreadsheet now tells us which > >> places are useful and which aren't, at least for the queries t

pg_rewind docs correction

2019-09-13 Thread James Coleman
on the source of any copied blocks. So I've attached a patch to summarize more correctly as well as document clearly the state of the cluster after the operation and also the operation sequencing dangers caused by copying configuration files from the source. James Coleman 001_pg_rewind_ex

Re: [DOC] Document auto vacuum interruption

2019-09-13 Thread James Coleman
On Sat, Aug 31, 2019 at 10:51 PM Amit Kapila wrote: > > On Fri, Jul 26, 2019 at 1:45 AM James Coleman wrote: > > > > We've discussed this internally many times, but today finally decided > > to write up a doc patch. > > > > Thanks, I think something

Re: pg_rewind docs correction

2019-09-14 Thread James Coleman
On Sat, Sep 14, 2019 at 12:20 AM Michael Paquier wrote: > > On Fri, Sep 13, 2019 at 01:47:03PM -0400, James Coleman wrote: > > So I've attached a patch to summarize more correctly as well as > > document clearly the state of the cluster after the operation and also >

Re: pg_rewind docs correction

2019-09-15 Thread James Coleman
On Sun, Sep 15, 2019 at 10:25 AM Michael Paquier wrote: > > On Sat, Sep 14, 2019 at 07:00:54PM -0400, James Coleman wrote: > > Updated (plus some additional wordsmithing). > > +The rewind operation is not expected to result in a consistent data > +directory state eit

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-15 Thread James Coleman
of me wants to pull it over to a separate thread just to get additional feedback, but I'm not sure how useful that is given we don't currently have an example case outside of this patch. One thing to note though: the current patch does not also modify add_partial_path_precheck which also does not take into account startup cost, so we probably need to update that for completeness's sake. James Coleman

Re: [DOC] Document auto vacuum interruption

2019-09-17 Thread James Coleman
On Tue, Sep 17, 2019 at 2:21 AM Amit Kapila wrote: > > On Fri, Sep 13, 2019 at 11:59 PM James Coleman wrote: > > > > On Sat, Aug 31, 2019 at 10:51 PM Amit Kapila > > wrote: > > > > > > > Updated patch attached. I changed the wording to be about co

Re: pg_rewind docs correction

2019-09-17 Thread James Coleman
On Tue, Sep 17, 2019 at 3:51 AM Michael Paquier wrote: > > On Sun, Sep 15, 2019 at 10:36:04AM -0400, James Coleman wrote: > > On Sun, Sep 15, 2019 at 10:25 AM Michael Paquier > > wrote: > >> +The rewind operation is not expected to result in a consistent dat

[DOC] Document concurrent index builds waiting on each other

2019-09-18 Thread James Coleman
various stages might change the results. James Coleman commit 9e28e704820eebb81ff94c1c3cbfb7db087b2c45 Author: James Coleman Date: Wed Sep 18 13:36:22 2019 -0400 Document concurrent indexes waiting on each other It's not immediately obvious that because concurrent index b

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-27 Thread James Coleman
On Mon, Sep 9, 2019 at 5:55 PM Tomas Vondra wrote: > The "patched" column means all developer GUCs disabled, so it's expected > to produce the same plan as master (and it is). And then there's one > column for each developer GUC. If the column is just TRUE it means the > GUC does not affect any of

Consider low startup cost in add_partial_path

2019-09-27 Thread James Coleman
then, and maybe that would shed some light on whether it's still inherently true. I've attached a patch (by Tomas Vondra, also cc'd) to consider startup cost in add_partial_path, but should we apply the patch we'll also likely need to apply the same kind of change to ad

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-27 Thread James Coleman
On Mon, Sep 16, 2019 at 6:32 AM Tomas Vondra wrote: > > On Sun, Sep 15, 2019 at 09:33:33PM -0400, James Coleman wrote: > >On Sat, Jul 20, 2019 at 11:25 AM Tomas Vondra > > wrote: > >> >> ... > >> >> > >> >> I think this may be a thi

Re: Consider low startup cost in add_partial_path

2019-09-28 Thread James Coleman
On Saturday, September 28, 2019, Tomas Vondra wrote: > On Sat, Sep 28, 2019 at 12:16:05AM -0400, Robert Haas wrote: > >> On Fri, Sep 27, 2019 at 2:24 PM James Coleman wrote: >> >>> Over in the incremental sort patch discussion we found [1] a case >>> wh

Re: [DOC] Document concurrent index builds waiting on each other

2019-09-28 Thread James Coleman
On Sat, Sep 28, 2019 at 9:22 PM Alvaro Herrera wrote: > > On 2019-Sep-28, Bruce Momjian wrote: > > > The CREATE INDEX docs already say: > > > > In a concurrent index build, the index is actually entered into > > the system catalogs in one transaction, then two table scans occur in > >

Re: [DOC] Document concurrent index builds waiting on each other

2019-09-28 Thread James Coleman
On Sat, Sep 28, 2019 at 9:56 PM Bruce Momjian wrote: > > On Sat, Sep 28, 2019 at 09:54:48PM -0400, James Coleman wrote: > > On Sat, Sep 28, 2019 at 9:22 PM Alvaro Herrera > > wrote: > > > > > > On 2019-Sep-28, Bruce Momjian wrote: > > &

Re: Consider low startup cost in add_partial_path

2019-10-02 Thread James Coleman
On Sat, Sep 28, 2019 at 7:21 PM James Coleman wrote: > Now the trick is to figure out a way to demonstrate it in test :) > > Basically we need: > Path A: Can short circuit with LIMIT but has high total cost > Path B: Can’t short circuit with LIMIT but has lower total cost &g

Re: Consider low startup cost in add_partial_path

2019-10-24 Thread James Coleman
On Fri, Oct 4, 2019 at 8:36 AM Robert Haas wrote: > > On Wed, Oct 2, 2019 at 10:22 AM James Coleman wrote: > > In all cases I've been starting with: > > > > set enable_hashjoin = off; > > set enable_nestloop = off; > > set max_parallel_workers_per_gather

Re: Proving IS NOT NULL inference for ScalarArrayOpExpr's

2019-03-02 Thread James Coleman
On Fri, Mar 1, 2019 at 5:28 PM Tom Lane wrote: > > James Coleman writes: > > [ saop_is_not_null-v10.patch ] > > I went through this again, and this time (after some more rewriting > of the comments) I satisfied myself that the logic is correct. > Hence, pushed. Thanks!

  1   2   3   4   5   6   >