Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-14 Thread James Coleman
On Fri, Mar 13, 2020 at 1:06 PM James Coleman wrote: > > On Thu, Mar 12, 2020 at 5:53 PM Alvaro Herrera > wrote: > > > > I gave this a very quick look; I don't claim to understand it or > > anything, but I thought these trivial cleanups worthwhile. The only &g

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-15 Thread James Coleman
On Sat, Mar 14, 2020 at 10:55 PM James Coleman wrote: > > On Fri, Mar 13, 2020 at 1:06 PM James Coleman wrote: > > > > On Thu, Mar 12, 2020 at 5:53 PM Alvaro Herrera > > wrote: > > > > > > I gave this a very quick look; I don't claim to under

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-15 Thread James Coleman
On Fri, Mar 13, 2020 at 4:22 PM Tom Lane wrote: > > Alvaro Herrera writes: > > Also, I wonder if it would be better to modify our policies so that we > > update typedefs.list more frequently. Some people include additions > > with their commits, but it's far from SOP. > > Perhaps. My own workfl

Re: improve transparency of bitmap-only heap scans

2020-03-16 Thread James Coleman
hanges - Added a basic commit message. - Add unfetched_pages initialization to ExecInitBitmapHeapScan. See attached. Thanks, James From 95babe8447eadb40c2d1452a9102d4766269d80f Mon Sep 17 00:00:00 2001 From: James Coleman Date: Sun, 15 Mar 2020 20:27:19 -0400 Subject: [PATCH v3] Show bitmap only

Re: Parallel leader process info in EXPLAIN

2020-03-17 Thread James Coleman
On Thu, Nov 7, 2019 at 9:48 PM Thomas Munro wrote: > > On Thu, Nov 7, 2019 at 11:37 PM Rafia Sabih wrote: > > ... > > Also, I noticed that the worker details are displayed for sort node even > > without verbose, but for scans it is only with verbose. Am I missing > > something or there is somet

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-17 Thread James Coleman
On Tue, Mar 17, 2020 at 9:03 PM Andres Freund wrote: > > Hi, > > On 2020-03-17 20:42:07 +0100, Laurenz Albe wrote: > > > I think Andres was thinking this would maybe be an optimization > > > independent of > > > is_insert_only (?) > > > > I wasn't sure. > > I'm not sure myself - but I'm doubtful

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-18 Thread James Coleman
On Tue, Mar 17, 2020 at 11:37 PM Justin Pryzby wrote: > > On Tue, Mar 17, 2020 at 09:58:53PM -0400, James Coleman wrote: > > On Tue, Mar 17, 2020 at 9:03 PM Andres Freund wrote: > > > > > > On 2020-03-17 20:42:07 +0100, Laurenz Albe wrote: > > > > >

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-18 Thread James Coleman
On Wed, Mar 18, 2020 at 1:08 PM Andres Freund wrote: > > Hi, > > On 2020-03-17 21:58:53 -0400, James Coleman wrote: > > On Tue, Mar 17, 2020 at 9:03 PM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2020-03-17 20:42:07 +0100, Laurenz

Re: improve transparency of bitmap-only heap scans

2020-03-19 Thread James Coleman
On Mon, Mar 16, 2020 at 9:08 AM James Coleman wrote: > ... > One question though: if I change the query to: > explain (analyze, buffers) select count(*) from exp where a between 50 > and 100 and d between 5 and 10; > then I get a parallel bitmap heap scan, and I only see exact hea

Re: improve transparency of bitmap-only heap scans

2020-03-19 Thread James Coleman
On Thu, Mar 19, 2020 at 9:26 PM Justin Pryzby wrote: > > On Mon, Mar 16, 2020 at 09:08:36AM -0400, James Coleman wrote: > > Does the original optimization cover parallel bitmap heap scans like this? > > It works for parallel bitmap only scans. > > template1=# explain anal

Re: optimisation? collation "C" sorting for GroupAggregate for all deterministic collations

2020-03-22 Thread James Coleman
On Sun, Mar 22, 2020 at 5:33 AM Pavel Stehule wrote: > > Hi > > ne 22. 3. 2020 v 10:12 odesílatel Maxim Ivanov napsal: >> >> Hi All, >> >> It is known, that collation "C" significantly speeds up string comparisons >> and as a result sorting. I was wondering, whether it is possible to use it >>

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread James Coleman
On Sun, Mar 22, 2020 at 6:02 PM Andreas Karlsson wrote: > > On 3/21/20 1:56 AM, Tomas Vondra wrote: > > I've looked at v38 but it seems it's a bit broken by some recent explain > > changes (mostly missing type in declarations). Attached is v39 fixing > > those issues, and including a bunch of fixe

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread James Coleman
On Sun, Mar 22, 2020 at 8:54 PM Andreas Karlsson wrote: > > On 3/23/20 1:33 AM, James Coleman wrote: > > So on the face of it we have a bit of a no-win situation. The function > > tuple_sort_method_name returns a const, but lappend wants a non-const. > > I'm no

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread James Coleman
On Fri, Mar 20, 2020 at 8:56 PM Tomas Vondra wrote: > > Hi, > > I've looked at v38 but it seems it's a bit broken by some recent explain > changes (mostly missing type in declarations). Attached is v39 fixing > those issues, and including a bunch of fixes based on a review - most of > the changes

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-23 Thread James Coleman
On Mon, Mar 23, 2020 at 1:05 PM Tom Lane wrote: > > Alvaro Herrera writes: > > ... all plan types that use only one child use the outer one. They > > could use either, as long as it does that consistently, I think. > > Yeah, exactly. The outer/inner terminology is really only sensible > for joi

Re: improve transparency of bitmap-only heap scans

2020-03-24 Thread James Coleman
On Tue, Mar 24, 2020 at 1:24 AM Amit Kapila wrote: > > On Fri, Mar 20, 2020 at 7:09 AM James Coleman wrote: > > > > Awesome, thanks for confirming with an actual plan. > > > > > I don't think it matters in nontext mode, but at least in text mode, I > &

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-24 Thread James Coleman
On Mon, Mar 23, 2020 at 11:44 PM Alvaro Herrera wrote: > > On 2020-Mar-23, James Coleman wrote: > > > 4. nodeIncrementalSort.c ExecReScanIncrementalSort: This whole chunk > > is suspect. I've mentioned previously I don't have a great mental > > model of how

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-24 Thread James Coleman
On Tue, Mar 24, 2020 at 7:08 PM Tomas Vondra wrote: > > On Tue, Mar 24, 2020 at 06:26:11PM -0400, James Coleman wrote: > >On Mon, Mar 23, 2020 at 11:44 PM Alvaro Herrera > > wrote: > >> > >> On 2020-Mar-23, James Coleman wrote: > >> > >> >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-24 Thread James Coleman
On Tue, Mar 24, 2020 at 8:23 PM James Coleman wrote: > While working on finding a test case to show rescan isn't implemented > properly yet, I came across a bug. At the top of > ExecInitIncrementalSort, we assert that eflags does not contain > EXEC_FLAG_REWIND. But the following

Re: improve transparency of bitmap-only heap scans

2020-03-25 Thread James Coleman
On Tue, Mar 24, 2020 at 11:02 PM Amit Kapila wrote: > > On Wed, Mar 25, 2020 at 12:44 AM Tom Lane wrote: > > > > I took a quick look through this patch. While I see nothing to complain > > about implementation-wise, I'm a bit befuddled as to why we need this > > reporting when there is no compar

Re: [DOC] Document concurrent index builds waiting on each other

2020-03-25 Thread James Coleman
On Wed, Mar 25, 2020 at 3:19 PM Andres Freund wrote: > > Hi, > > On 2019-09-18 13:51:00 -0400, James Coleman wrote: > > In my experience it's not immediately obvious (even after reading the > > documentation) the implications of how concurrent index builds manage >

Re: Make mesage at end-of-recovery less scary.

2020-03-27 Thread James Coleman
On Thu, Mar 26, 2020 at 12:41 PM Robert Haas wrote: > > On Wed, Mar 25, 2020 at 8:53 AM Peter Eisentraut > wrote: > > HINT: This is to be expected if this is the end of the WAL. Otherwise, > > it could indicate corruption. > > First, I agree that this general issue is a problem, because it's co

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread James Coleman
On Fri, Mar 27, 2020 at 10:58 PM Tomas Vondra wrote: > ... > >> As a side note here, I'm wondering if this (determining useful pathkeys) > >> can be made a bit smarter by looking both at query_pathkeys and pathkeys > >> useful for merging, similarly to what truncate_useless_pathkeys() does. > >> B

Re: improve transparency of bitmap-only heap scans

2020-03-28 Thread James Coleman
On Fri, Mar 27, 2020 at 9:24 PM Amit Kapila wrote: > > On Wed, Mar 25, 2020 at 5:44 PM James Coleman wrote: > > > > On Tue, Mar 24, 2020 at 11:02 PM Amit Kapila > > wrote: > > > > > > On Wed, Mar 25, 2020 at 12:44 AM Tom Lane wrote: > > &g

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread James Coleman
On Sat, Mar 28, 2020 at 2:54 PM Tomas Vondra wrote: > ... > >> >> >9. optimizer/path/allpaths.c get_useful_pathkeys_for_relation: > >> >> >* Considering query_pathkeys is always worth it, because it might let > >> >> >us > >> >> >* avoid a local sort. > >> >> > > >> >> >That originally was a copy

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread James Coleman
On Sat, Mar 28, 2020 at 5:30 PM Tomas Vondra wrote: > > On Sat, Mar 28, 2020 at 10:19:04AM -0400, James Coleman wrote: > >On Fri, Mar 27, 2020 at 10:58 PM Tomas Vondra > > wrote: > >> > >> ... > >> > >> The more I look

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread James Coleman
On Sat, Mar 28, 2020 at 11:14 PM Tomas Vondra wrote: > > On Sat, Mar 28, 2020 at 10:47:49PM -0400, James Coleman wrote: > >On Sat, Mar 28, 2020 at 6:59 PM Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> Attached is my take on simplific

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-29 Thread James Coleman
On Sun, Mar 29, 2020 at 9:44 PM Tomas Vondra wrote: > > Hi, > > Attached is a slightly reorganized patch series. I've merged the fixes > into the appropriate matches, and I've also combined the two patches > adding incremental sort paths to additional places in planner. > > A couple more comments:

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 12:31 PM Alvaro Herrera wrote: > > On 2020-Mar-30, James Coleman wrote: > > > +/* > > + *Instruementation information for IncrementalSort > > + * > > + */ > > +typedef struct Incremen

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 1:04 PM Tom Lane wrote: > > James Coleman writes: > > + * TuplesortMethod is used in a bitmask in Increment Sort's shared memory > > + * instrumentation so needs to have each value be a separate bit. > > >> I don't quite understand

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 5:53 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 02:23:15PM -0400, James Coleman wrote: > >On Mon, Mar 30, 2020 at 9:14 PM Tomas Vondra > > wrote: > >> > >> The main thing I've been working on today is benchmarking how thi

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 6:54 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 06:35:32PM -0400, Tom Lane wrote: > >Tomas Vondra writes: > >> In general, I think it'd be naive that we can make planner smarter with > >> no extra overhead spent on planning, and we can never accept patches > >> add

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 7:56 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 07:09:04PM -0400, James Coleman wrote: > >On Tue, Mar 31, 2020 at 6:54 PM Tomas Vondra > > wrote: > >> > >> On Tue, Mar 31, 2020 at 06:35:32PM -0400, Tom Lane wrote: > >>

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 8:38 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 08:11:15PM -0400, James Coleman wrote: > >On Tue, Mar 31, 2020 at 7:56 PM Tomas Vondra > > wrote: > >> > >> On Tue, Mar 31, 2020 at 07:09:04PM -0400, James Coleman wrote: > &g

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 9:59 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 08:42:47PM -0400, James Coleman wrote: > >On Tue, Mar 31, 2020 at 8:38 PM Tomas Vondra > > wrote: > >> > >> On Tue, Mar 31, 2020 at 08:11:15PM -0400, James Coleman wrote: > &g

Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread James Coleman
Currently there's no good way that I'm aware of for monitoring software to check what the xmin horizon is being blocked at. You can check pg_stat_replication and pg_replication_slots and txid_snapshot_xmin(txid_current_snapshot()) and so on, but that list can grow, and it means monitoring setups ne

Re: Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread James Coleman
On Wed, Apr 1, 2020 at 5:58 PM Alvaro Herrera wrote: > > On 2020-Apr-01, Tom Lane wrote: > > > James Coleman writes: > > > To my knowledge the current oldest xmin (GetOldestXmin() if I'm not > > > mistaken) isn't exposed directly in any view or

Re: Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-02 Thread James Coleman
On Thu, Apr 2, 2020 at 12:13 AM Craig Ringer wrote: > > > > > On Thu, 2 Apr 2020 at 07:57, Tom Lane wrote: >> >> Alvaro Herrera writes: >> > On 2020-Apr-01, Tom Lane wrote: >> >> The fact that I had to use max(age(...)) in that sample query >> >> hints at one reason: it's really hard to do arith

Re: Should we add xid_current() or a int8->xid cast?

2020-04-02 Thread James Coleman
On Thu, Apr 2, 2020 at 2:47 PM Mark Dilger wrote: > > > > > On Apr 2, 2020, at 11:01 AM, Andres Freund wrote: > > > >> > >> Hmm, for some reason I had it in my head that we would make these use an > >> "epoch/val" output format rather than raw uint64 values. > > > > Why would we do that? IMO the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-02 Thread James Coleman
On Thu, Apr 2, 2020 at 8:20 PM Tomas Vondra wrote: > > Hi, > > Thanks, the v52 looks almost ready. I've been looking at the two or > three things I mentioned, and I have a couple of comments. > > > 1) /* XXX comparison_cost shouldn't be 0? */ > > I'm not worried about this, because this is not rea

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-02 Thread James Coleman
On Thu, Apr 2, 2020 at 8:46 PM James Coleman wrote: > > On Thu, Apr 2, 2020 at 8:20 PM Tomas Vondra > wrote: > > > > Hi, > > > > Thanks, the v52 looks almost ready. I've been looking at the two or > > three things I mentioned, and I have

Re: Nicer error when connecting to standby with hot_standby=off

2020-04-03 Thread James Coleman
On Thu, Apr 2, 2020 at 5:53 PM David Zhang wrote: > > The following review has been posted through the commitfest application: > make installcheck-world: not tested > Implements feature: tested, passed > Spec compliant: not tested > Documentation:not tested > > I appli

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > >On 2020-Apr-06, Tom Lane wrote: > > > >> Locally, things pass without force_parallel_mode, but turning it on > >> produces failures that look similar to rhinoceros's (didn't ex

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:20 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra > wrote: > > > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > > >On 2020-Apr-06, Tom Lane wrote: > > > > > >> Locally, thin

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:22 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 5:20 PM James Coleman wrote: > > > > On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra > > wrote: > > > > > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wr

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: > >On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > >>On 2020-Apr-06, Tom Lane wrote: > >> > >>>Locally, things pass without force_parallel_mode, but turning it on

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > > wrote: > >> > >> On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: > >&

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 7:09 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra > wrote: > > > > On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > > >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > > > wrote: > > >&

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread James Coleman
On Mon, Jan 17, 2022 at 4:20 PM Robert Haas wrote: > > On Fri, Jan 14, 2022 at 7:42 PM James Coleman wrote: > > I've attached a simple patch (sans tests and documentation) to get > > feedback early. After poking around this afternoon it seemed to me > > that the

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread James Coleman
On Mon, Jan 17, 2022 at 4:34 PM Alvaro Herrera wrote: > > On 2022-Jan-14, James Coleman wrote: > > > The logical slot can't flush past the > > last commit, so even if there's 100s of megabytes of unflushed WAL on > > the slot there may be zero lag (in te

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 9:25 AM Robert Haas wrote: > > On Mon, Jan 17, 2022 at 8:39 PM James Coleman wrote: > > I wondered about that, but commit_ts already does more than commit > > timestamps by recording the xid of the last commit. > > Well, if you're maintaining

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 12:50 PM Alvaro Herrera wrote: > > On 2022-Jan-17, James Coleman wrote: > > > I'd be happy to make it a separate GUC, though it seems adding an > > additional atomic access is worse (assuming we can convince ourselves > > putting

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 1:52 PM Alvaro Herrera wrote: > > On 2022-Jan-18, James Coleman wrote: > > > Reading the code it seems the only usage (besides > > the boolean activation status also stored there) is in > > TransactionIdGetCommitTsData, and the only consumers of

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
e that. We'd also > need to maintain a value for all disconnected backends, but that's also not a > hot > path. I expect most monitoring setups default to around something like checking anywhere from every single digit seconds to minutes. If I read between the lines I imagine you'd see even e.g. every 2s as not that big of a deal here, right? Thanks, James Coleman

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
be present if track_commit_timestamps isn't on)? Or would you expect the current xid and timestamp use the new infrastructure also? Thanks, James Coleman

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 8:05 PM Andres Freund wrote: > > Hi, > > On 2022-01-18 18:31:42 -0500, James Coleman wrote: > > One other question on this: if we went with this would you expect a > > new function to parallel pg_last_committed_xact()? > > I don't think I

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
_xact() currently finds its home there. I think we need a shared ProcArrayLock to read the array, correct? We also need to do the global updating under lock, but given it's when a proc is removed, that shouldn't be a performance issue if I'm following what you are saying. Thanks, Jame

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-19 Thread James Coleman
On Wed, Jan 19, 2022 at 7:51 PM David G. Johnston wrote: > > On Wed, Jan 19, 2022 at 5:08 PM Bossart, Nathan wrote: >> >> On 9/24/21, 7:30 AM, "James Coleman" wrote: >> > When PG11 added the ability for ALTER TABLE ADD COLUMN to set a constant >> >

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-19 Thread James Coleman
SN */ > > }; > > We do not rely on 64bit integers to be read/written atomically, just 32bit > ones. To make this work for older platforms you'd have to use a > pg_atomic_uint64. On new-ish platforms pg_atomic_read_u64/pg_atomic_write_u64 > end up as plain read/writes, but o

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-20 Thread James Coleman
On Wed, Jan 19, 2022 at 10:12 PM Andres Freund wrote: > > Hi, > > On 2022-01-19 21:23:12 -0500, James Coleman wrote: > > { oid => '3537', descr => 'get identification of SQL object', > > diff --git a/src/include/storage/proc.h b/src/include/stor

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-20 Thread James Coleman
On Wed, Jan 19, 2022 at 9:34 PM David G. Johnston wrote: > > On Wed, Jan 19, 2022 at 6:14 PM James Coleman wrote: >> >> I'm open to the idea of wordsmithing here, of course, but I strongly >> disagree that this is irrelevant data. > > > Ok, but wording aside

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-20 Thread James Coleman
On Thu, Jan 20, 2022 at 3:31 PM Andrew Dunstan wrote: > > > On 1/20/22 12:25, Bossart, Nathan wrote: > > On 1/19/22, 5:15 PM, "James Coleman" wrote: > >> I'm open to the idea of wordsmithing here, of course, but I strongly > >> disagree t

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-21 Thread James Coleman
On Thu, Jan 20, 2022 at 3:43 PM James Coleman wrote: > > As noted earlier I expect to be posting an updated patch soon. Here's the updated series. In 0001 I've moved the documentation tweak into the ALTER TABLE notes section. In 0002 I've taken David J's suggestion of s

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-21 Thread James Coleman
On Fri, Jan 21, 2022 at 4:08 PM Andrew Dunstan wrote: > > > On 1/21/22 13:55, James Coleman wrote: > > On Thu, Jan 20, 2022 at 3:43 PM James Coleman wrote: > >> As noted earlier I expect to be posting an updated patch soon. > > Here's the updated series. I

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-21 Thread James Coleman
d then add any desired default as > > described > > - below. > > > has now been completely removed from the documentation. > > Really? That's horrid, because that's directly useful advice. Remedied, but rewritten a bit to better fit with the new style/goal of

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-22 Thread James Coleman
On Sat, Jan 22, 2022 at 12:35 AM David G. Johnston wrote: > > On Fri, Jan 21, 2022 at 5:14 PM James Coleman wrote: >> >> >> > Really? That's horrid, because that's directly useful advice. >> >> Remedied, but rewritten a bit to better fit with th

Re: Parallelize correlated subqueries that execute within each worker

2022-01-22 Thread James Coleman
On Fri, Jan 21, 2022 at 3:20 PM Robert Haas wrote: > > On Fri, Jan 14, 2022 at 2:25 PM James Coleman wrote: > > I've been chewing on this a bit, and I was about to go re-read the > > code and see how easy it'd be to do exactly what you're suggesting in > >

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-25 Thread James Coleman
On Sat, Jan 22, 2022 at 10:28 AM David G. Johnston wrote: > > > > On Saturday, January 22, 2022, James Coleman wrote: >> >> On Sat, Jan 22, 2022 at 12:35 AM David G. Johnston >> wrote: >> > >> > On Fri, Jan 21, 2022 at 5:14 PM James Coleman w

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-28 Thread James Coleman
On Thu, Jan 20, 2022 at 8:15 AM James Coleman wrote: > > On Wed, Jan 19, 2022 at 10:12 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-01-19 21:23:12 -0500, James Coleman wrote: > > > { oid => '3537', descr => 'get identification o

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-29 Thread James Coleman
On Fri, Jan 28, 2022 at 7:47 PM Andres Freund wrote: > > On 2022-01-28 16:36:32 -0800, Andres Freund wrote: > > On 2022-01-28 18:43:57 -0500, James Coleman wrote: > > > Alternatively I see pg_attribute_aligned, but that's not defined > > > (AFAICT) on clang, for

Re: Synchronizing slots from primary to standby

2022-02-18 Thread James Coleman
review this patch, and last time I checked I noticed it didn't seem to apply cleanly to master anymore. Would you be able to send a rebased version? Thanks, James Coleman

Re: Synchronizing slots from primary to standby

2022-02-18 Thread James Coleman
quot;localy" -> "locally"). This patch would be a significant improvement for us; I'm hoping we can see some activity on it. I'm also hoping to try to do some testing next week and see if I can poke any holes in the functionality (with the goal of verifying Andres's concerns about the safety without the minimal logical decoding on a replica patch). Thanks, James Coleman

Re: Commit fest 2022-11

2022-11-14 Thread James Coleman
's something the app should do for us in this situation. Without that though the patch authors are left to wade through unrelated discussion, and, probably more importantly, the patch discussion thread doesn't show the current state (I think bumping there is more likely to prompt activity as well). James Coleman

Possible typo/unclear comment in joinpath.c

2021-04-14 Thread James Coleman
In joinpath.c three times we reference "extra_lateral_rels" (with underscores like it's a field), but as far as I can tell that's not a field anywhere in the source code, and looking at the code that follows it seems like it should be referencing "lateral_relids" (and the "extra" is really "extra [

Re: Possible typo/unclear comment in joinpath.c

2021-04-14 Thread James Coleman
On Wed, Apr 14, 2021 at 1:27 PM Tom Lane wrote: > > Justin Pryzby writes: > > On Wed, Apr 14, 2021 at 11:36:38AM -0400, James Coleman wrote: > >> In joinpath.c three times we reference "extra_lateral_rels" (with > >> underscores like it's a

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-14 Thread James Coleman
On Mon, Apr 12, 2021 at 8:37 AM Tomas Vondra wrote: > > On 4/12/21 2:24 PM, Luc Vlaming wrote: > > Hi, > > > > When trying to run on master (but afaik also PG-13) TPC-DS queries 94, > > 95 and 96 on a SF10 I get the error "could not find pathkey item to sort". > > When I disable enable_gathermerge

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-14 Thread James Coleman
On Wed, Apr 14, 2021 at 8:16 PM Robert Haas wrote: > > On Mon, Apr 12, 2021 at 8:37 AM Tomas Vondra > wrote: > > Could be related to incremental sort, which allowed some gather merge > > paths that were impossible before. We had a couple issues related to > > that fixed in November, IIRC. > > Hmm

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-14 Thread James Coleman
On Wed, Apr 14, 2021 at 8:21 PM Robert Haas wrote: > > On Wed, Apr 14, 2021 at 5:43 PM James Coleman wrote: > > The query in question is: > > select count(*) > > from store_sales > > ,household_demographics > > ,time_dim, sto

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-14 Thread James Coleman
On Wed, Apr 14, 2021 at 8:21 PM Robert Haas wrote: > > On Wed, Apr 14, 2021 at 5:43 PM James Coleman wrote: > > The query in question is: > > select count(*) > > from store_sales > > ,household_demographics > > ,time_dim, sto

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-14 Thread James Coleman
On Wed, Apr 14, 2021 at 5:42 PM James Coleman wrote: > > On Mon, Apr 12, 2021 at 8:37 AM Tomas Vondra > wrote: > > > > On 4/12/21 2:24 PM, Luc Vlaming wrote: > > > Hi, > > > > > > When trying to run on master (but afaik also PG-13) TPC-DS queri

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-15 Thread James Coleman
On Thu, Apr 15, 2021 at 5:33 AM Luc Vlaming wrote: > > On 15-04-2021 04:01, James Coleman wrote: > > On Wed, Apr 14, 2021 at 5:42 PM James Coleman wrote: > >> > >> On Mon, Apr 12, 2021 at 8:37 AM Tomas Vondra > >> wrote: > >>> >

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-19 Thread James Coleman
nd_ec_member_matching_expr at all; > that task can be left with the one caller that cares. I like the refactoring here. Two things I wonder: 1. Should we add tests for the relabel code path? 2. It'd be nice not to have the IS_SRF_CALL duplicated, but that might add enough complexity that it's not worth it. Thanks, James Coleman

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-19 Thread James Coleman
comment on this in my previous email, but it seems to me that relation_has_safe_ec_member, while less wordy, isn't quite descriptive enough. Perhaps something like relation_has_sort_safe_ec_member? James Coleman

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-19 Thread James Coleman
27;s a v4. This all looks good to me. James Coleman

Re: "could not find pathkey item to sort" for TPC-DS queries 94-96

2021-04-20 Thread James Coleman
On Tue, Apr 20, 2021 at 7:11 AM Dagfinn Ilmari Mannsåker wrote: > > ilm...@ilmari.org (Dagfinn Ilmari Mannsåker) writes: > > > Tom Lane writes: > > > >> +/* We ignore binary-compatible relabeling on both ends */ > >> +while (expr && IsA(expr, RelabelType)) > >> +expr = ((Relab

Parallelize correlated subqueries that execute within each worker

2021-05-07 Thread James Coleman
In a bug report back in November [1] a subthread explored why parallel query is excluded any time we have "Plan nodes which reference a correlated SubPlan". Amit's understanding was that the reasoning had to do with inability to easily pass (potentially variable length) Param values between workers

Re: Processing btree walks as a batch to parallelize IO

2021-05-07 Thread James Coleman
On Fri, Apr 9, 2021 at 4:57 PM Tomas Vondra wrote: > > > > On 4/9/21 7:33 PM, James Coleman wrote: > > $SUBJECT is still a very loosely formed idea, so forgive lack of detail > > or things I've likely missed, but I wanted to get it out there to see if > > it

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-05-07 Thread James Coleman
On Sat, Apr 24, 2021 at 6:25 AM David Rowley wrote: > > On Wed, 14 Apr 2021 at 05:40, James Coleman wrote: > > ...and here's a draft patch. I can take this to a new thread if you'd > > prefer; the one here already got committed, on the other hand this is > &

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-05-07 Thread James Coleman
On Fri, May 7, 2021 at 9:16 PM Tom Lane wrote: > > David Rowley writes: > > On Sat, 8 May 2021 at 09:15, James Coleman wrote: > >> On Sat, Apr 24, 2021 at 6:25 AM David Rowley wrote: > >>> I'm a bit undecided if it's safe to set the opfuncid to the n

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-05-07 Thread James Coleman
On Fri, May 7, 2021 at 8:38 PM David Rowley wrote: > > It's important to think of other cases, I just don't think there's any > need to do anything for that one. Remember that we have the > restriction of requiring a set of Consts, so for that case to be met, > someone would have to write somethi

Re: Parallelize correlated subqueries that execute within each worker

2023-01-21 Thread James Coleman
On Wed, Jan 18, 2023 at 9:34 PM James Coleman wrote: > > On Wed, Jan 18, 2023 at 2:09 PM Tomas Vondra > wrote: > > > > Hi, > > > > This patch hasn't been updated since September, and it got broken by > > 4a29eabd1d91c5484426bc5836e0a7143b064f5a which

Fix incorrect comment reference

2023-01-23 Thread James Coleman
Hello, See the attached for a simple comment fix -- the referenced generate_useful_gather_paths call isn't in grouping_planner it's in apply_scanjoin_target_to_paths. Thanks, James Coleman v1-0001-Fixup-incorrect-comment.patch Description: Binary data

Re: Parallelize correlated subqueries that execute within each worker

2023-01-23 Thread James Coleman
On Sat, Jan 21, 2023 at 10:07 PM James Coleman wrote: > ... > While working through Tomas's comment about a conditional in the > max_parallel_hazard_waker being guaranteed true I realized that in the > current version of the patch the safe_param_ids tracking in > is_parallel

Re: Fix incorrect comment reference

2023-01-23 Thread James Coleman
On Mon, Jan 23, 2023 at 1:26 PM Robert Haas wrote: > > On Mon, Jan 23, 2023 at 8:31 AM James Coleman wrote: > > See the attached for a simple comment fix -- the referenced > > generate_useful_gather_paths call isn't in grouping_planner it's in > > apply_scanjoin

Re: Fix incorrect comment reference

2023-01-23 Thread James Coleman
On Mon, Jan 23, 2023 at 3:41 PM Robert Haas wrote: > > On Mon, Jan 23, 2023 at 3:19 PM James Coleman wrote: > > On Mon, Jan 23, 2023 at 1:26 PM Robert Haas wrote: > > > On Mon, Jan 23, 2023 at 8:31 AM James Coleman wrote: > > > > See the attached for a sim

Re: Fix incorrect comment reference

2023-01-23 Thread James Coleman
On Mon, Jan 23, 2023 at 4:07 PM James Coleman wrote: > > On Mon, Jan 23, 2023 at 3:41 PM Robert Haas wrote: > > > > On Mon, Jan 23, 2023 at 3:19 PM James Coleman wrote: > > > On Mon, Jan 23, 2023 at 1:26 PM Robert Haas wrote: > > > > On Mon, Jan 2

Re: Parallelize correlated subqueries that execute within each worker

2023-02-08 Thread James Coleman
On Mon, Feb 6, 2023 at 11:39 AM Antonin Houska wrote: > > James Coleman wrote: > > Which this patch we do in fact now see (as expected) rels with > > non-empty lateral_relids showing up in generate_[useful_]gather_paths. > > And the partial paths can now have non-empty

Re: Consider parallel for lateral subqueries with limit

2022-09-24 Thread James Coleman
On Thu, Sep 22, 2022 at 5:19 PM James Coleman wrote: > > On Mon, Sep 19, 2022 at 4:29 PM Robert Haas wrote: > > > > On Mon, Sep 19, 2022 at 3:58 PM James Coleman wrote: > > > But in the case where there's correlation via LATERAL we already don't > > &g

Re: Allow foreign keys to reference a superset of unique columns

2022-09-24 Thread James Coleman
In the example above then an update to bar.x is going to update the rows in foo that match bar.x = foo.a and bar.y = foo.b *using the old values of bar.x and bar.y* to be the new values. You seem to be suggesting that instead it should look for other rows that already match the *new value* of only one of the columns in the constraint. If I'm understanding the example correctly, that seems like a *very* bad idea. James Coleman

Re: Allow foreign keys to reference a superset of unique columns

2022-09-25 Thread James Coleman
On Sun, Sep 25, 2022 at 4:49 AM Wolfgang Walther wrote: > > James Coleman: > > If I'm following properly this sounds like an overengineered EAV > > schema, and neither of those things inspires me to think "this is a > > use case I want to support". > &g

<    1   2   3   4   5   6   >