GiST index build missing smgrimmedsync()?

2022-02-23 Thread Melanie Plageman
I brought this up in [1] but wanted to start a dedicated thread. Since 16fa9b2b30a357 GiST indexes are not built in shared buffers. However, smgrimmedsync() is not done anywhere and skipFsync=true is always passed to smgrwrite() and smgrextend(). So, if a checkpoint starts and finishes after WAL-l

why do hash index builds use smgrextend() for new splitpoint pages

2022-02-24 Thread Melanie Plageman
I'm trying to understand why hash indexes are built primarily in shared buffers except when allocating a new splitpoint's worth of bucket pages -- which is done with smgrextend() directly in _hash_alloc_buckets(). Is this just so that the value returned by smgrnblocks() includes the new splitpoint

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-25 Thread Melanie Plageman
On Thu, Feb 24, 2022 at 10:24 PM Amit Kapila wrote: > > On Fri, Feb 25, 2022 at 4:41 AM Melanie Plageman > wrote: > > > > I'm trying to understand why hash indexes are built primarily in shared > > buffers except when allocating a new splitpoint's worth of b

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-26 Thread Melanie Plageman
On Fri, Feb 25, 2022 at 11:17 PM Amit Kapila wrote: > > On Sat, Feb 26, 2022 at 3:01 AM Melanie Plageman > wrote: > > > > Since _hash_alloc_buckets() WAL-logs the last page of the > > splitpoint, is it safe to skip the smgrimmedsync()? What if the last > > page

Re: Missed condition-variable wakeups on FreeBSD

2022-02-26 Thread Melanie Plageman
On Sat, Feb 26, 2022 at 2:07 PM Tom Lane wrote: > > About once a month over the last six months, my buildfarm animal > florican has gotten stuck while running the core regression tests. > The symptoms have looked very much the same each time: there is > a backend with two parallel worker processes

Why do spgbuildempty(), btbuildempty(), and blbuildempty() use smgrwrite()?

2022-03-02 Thread Melanie Plageman
n content, seems like the former). - Melanie From f4f594ed06d5cf54135e4f9974fd2158ead8a7bb Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Wed, 2 Mar 2022 15:42:48 -0500 Subject: [PATCH v1 1/2] Build unlogged index init fork in shared buffers Move empty index build in the init fork for u

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-03-04 Thread Melanie Plageman
thanks for doing that and for the rebase! since I made updates, the attached version 6 is also rebased. To Dmitry's question: On Sun, Feb 13, 2022 at 9:33 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Wed, Feb 09, 2022 at 01:49:30PM -0500, Melanie Plageman wrote

Re: make tuplestore helper function

2022-03-07 Thread Melanie Plageman
iptors is in helper functions (like materializeResult()) which separate the tuplestore creation from the tuple descriptor initialization in a way that made it hard to do a drop-in replacement with the helper function. - Melanie From a734a4dbf33230242f53bdc4ab6d23a98d6def8c Mon Sep 17 00:00:00

Issue with pg_stat_subscription_stats

2022-03-11 Thread Melanie Plageman
So, I noticed that pg_stat_reset_subscription_stats() wasn't working properly, and, upon further investigation, I'm not sure the view pg_stat_subscription_stats is being properly populated. I don't think subscriptionStatHash will be created properly and that the reset timestamp won't be initialize

Re: Issue with pg_stat_subscription_stats

2022-03-13 Thread Melanie Plageman
On Sat, Mar 12, 2022 at 3:15 PM Andres Freund wrote: > > Hi, > > On 2022-03-12 08:28:35 +0530, Amit Kapila wrote: > > On Sat, Mar 12, 2022 at 2:14 AM Melanie Plageman > > wrote: > > > > > > So, I noticed that pg_stat_reset_subscription_stats() wasn&#

Re: Issue with pg_stat_subscription_stats

2022-03-14 Thread Melanie Plageman
On Mon, Mar 14, 2022 at 4:02 AM Masahiko Sawada wrote: > > On Mon, Mar 14, 2022 at 2:05 AM Melanie Plageman > wrote: > > > > On Sat, Mar 12, 2022 at 3:15 PM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2022-03-12 08:28:35 +0530, A

Re: shared-memory based stats collector - v66

2022-03-20 Thread Melanie Plageman
sts with DROP SCHEMA actually exercise another code path, so it might be worth cutting those. - Melanie From 2cb108fadf656de9cc998c0b2123a184881ef4e0 Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Thu, 17 Mar 2022 21:54:16 -0400 Subject: [PATCH] add replica cleanup tests --- src/backend/

Re: shared-memory based stats collector - v66

2022-03-20 Thread Melanie Plageman
On Sun, Mar 20, 2022 at 12:58 PM Andres Freund wrote: > > Hi, > > On 2022-03-20 12:32:39 -0400, Melanie Plageman wrote: > > Attached is a TAP test to check that stats are cleaned up on a physical > > replica after the objects they concern are dropped on the primary. >

Re: shared-memory based stats collector - v66

2022-03-21 Thread Melanie Plageman
On Sun, Mar 20, 2022 at 4:56 PM Melanie Plageman wrote: > > Addressed all of these points in > v2-0001-add-replica-cleanup-tests.patch > > also added a new test file in > v2-0002-Add-TAP-test-for-discarding-stats-after-crash.patch > testing correct behavior after a crash a

Re: shared-memory based stats collector - v66

2022-03-22 Thread Melanie Plageman
e columns fetched from the PgStatBgwriter data structure since those and the Checkpointer stats are stored separately despite being displayed in the same view currently. but, alas... - Melanie From 37e5ba3b7743309b00c81dbfe65cfd481d4859a6 Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Tue,

Re: Parallel Full Hash Join

2020-11-04 Thread Melanie Plageman
ere probably isn't a nice name for this concept, since it is a function with the purpose of terminating parallelism. Regards, Melanie (Microsoft) From 0b13b62a8aac071393ac65b19dc1ec86daa967f2 Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Wed, 4 Nov 2020 14:25:33 -0800 Subject: [PAT

Re: Parallel Full Hash Join

2021-02-11 Thread Melanie Plageman
> + */ > +bool > +ExecParallelPrepHashTableForUnmatched(HashJoinState *hjstate) > > Comment name doesn't match function name. Updated -- and a few other comment updates too. I just attached the diff. >From 213c36f9e125f52eb6731005d5dcdadca73a Mon Sep 17 00:00:00 2001

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-03 Thread Melanie Plageman
Thanks again! I really appreciate the thorough review. I have combined responses to all three of your emails below. Let me know if it is more confusing to do it this way. On Wed, Dec 1, 2021 at 6:59 PM Justin Pryzby wrote: > > On Wed, Dec 01, 2021 at 05:00:14PM -0500, Melanie Plageman

Re: make tuplestore helper function

2021-12-13 Thread Melanie Plageman
On Thu, Nov 18, 2021 at 1:24 PM Justin Pryzby wrote: > > On Thu, Nov 18, 2021 at 12:59:03PM -0500, Melanie Plageman wrote: > > On Mon, Nov 8, 2021 at 3:13 PM Justin Pryzby wrote: > > > On Mon, Nov 08, 2021 at 02:52:28PM -0500, Melanie Plageman wrote: > > > > On T

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-15 Thread Melanie Plageman
v18 attached. On Thu, Dec 9, 2021 at 2:17 PM Andres Freund wrote: > > Hi, > > On 2021-12-03 15:02:24 -0500, Melanie Plageman wrote: > > From e0f7f3dfd60a68fa01f3c023bcdb69305ade3738 Mon Sep 17 00:00:00 2001 > > From: Melanie Plageman > > Date: Mon, 11 Oct 2021 16:1

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-30 Thread Melanie Plageman
On Tue, Dec 21, 2021 at 8:32 PM Melanie Plageman wrote: > On Thu, Dec 16, 2021 at 3:18 PM Andres Freund wrote: > > > > > From 9f22da9041e1e1fbc0ef003f5f78f4e72274d438 Mon Sep 17 00:00:00 2001 > > > > > From: Melanie Plageman > > > > > Date: W

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-01-03 Thread Melanie Plageman
On Thu, Dec 30, 2021 at 3:30 PM Melanie Plageman wrote: > > On Tue, Dec 21, 2021 at 8:32 PM Melanie Plageman > wrote: > > On Thu, Dec 16, 2021 at 3:18 PM Andres Freund wrote: > > > > > > From 9f22da9041e1e1fbc0ef003f5f78f4e72274d438 Mon Sep 17 00:00:00

Re: make tuplestore helper function

2022-01-05 Thread Melanie Plageman
Thanks for the detailed review! On Fri, Dec 17, 2021 at 3:04 PM Justin Pryzby wrote: > > On Mon, Dec 13, 2021 at 05:27:52PM -0500, Melanie Plageman wrote: > > Done in attached v4 > > Thanks. > > I think these comments can be removed from the callers: > /* it&#x

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-10 Thread Melanie Plageman
: > > Hi, > > On 2021-11-19 15:11:57 -0500, Melanie Plageman wrote: > > From 2130175c5d794f60c5f15d6eb1b626c81fb7c39b Mon Sep 17 00:00:00 2001 > > From: Melanie Plageman > > Date: Thu, 15 Apr 2021 07:01:01 -0400 > > Subject: [PATCH v2] Index build avoids immed fsy

Re: make tuplestore helper function

2022-01-11 Thread Melanie Plageman
On Wed, Jan 5, 2022 at 7:57 PM Justin Pryzby wrote: > > On Wed, Jan 05, 2022 at 12:09:16PM -0500, Melanie Plageman wrote: > > On Fri, Dec 17, 2021 at 3:04 PM Justin Pryzby wrote: > > > There's a couples places that you're checking expectedDesc where it wasn't

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-11 Thread Melanie Plageman
On Mon, Jan 10, 2022 at 5:50 PM Melanie Plageman wrote: > > I have attached a v3 which includes two commits -- one of which > implements the directmgr API and uses it and the other which adds > functionality to use either directmgr or bufmgr API during index build. > > Also reg

Re: Parallel Full Hash Join

2022-01-11 Thread Melanie Plageman
On Fri, Nov 26, 2021 at 3:11 PM Thomas Munro wrote: > > On Sun, Nov 21, 2021 at 4:48 PM Justin Pryzby wrote: > > On Wed, Nov 17, 2021 at 01:45:06PM -0500, Melanie Plageman wrote: > > > Yes, this looks like that issue. > > > > > > I've attached a v8

Re: Assertion failure with barriers in parallel hash join

2021-03-31 Thread Melanie Plageman
On Wed, Mar 17, 2021 at 8:18 AM Thomas Munro wrote: > > On Wed, Mar 17, 2021 at 6:58 PM Thomas Munro wrote: > > According to BF animal elver there is something wrong with this > > commit. Looking into it. > > Assertion failure reproduced here and understood, but unfortunately > it'll take some m

Re: Assertion failure with barriers in parallel hash join

2021-03-31 Thread Melanie Plageman
On Wed, Mar 17, 2021 at 8:18 AM Thomas Munro wrote: > > On Wed, Mar 17, 2021 at 6:58 PM Thomas Munro wrote: > > According to BF animal elver there is something wrong with this > > commit. Looking into it. > > Assertion failure reproduced here and understood, but unfortunately > it'll take some m

Re: Parallel Full Hash Join

2021-04-02 Thread Melanie Plageman
On Fri, Mar 5, 2021 at 8:31 PM Thomas Munro wrote: > > On Tue, Mar 2, 2021 at 11:27 PM Thomas Munro wrote: > > On Fri, Feb 12, 2021 at 11:02 AM Melanie Plageman > > wrote: > > > I just attached the diff. > > > > Squashed into one patch for the cfbot to ch

Re: Parallel Full Hash Join

2021-04-06 Thread Melanie Plageman
On Fri, Apr 2, 2021 at 3:06 PM Zhihong Yu wrote: > > Hi, > For v6-0003-Parallel-Hash-Full-Right-Outer-Join.patch > > +* current_chunk_idx: index in current HashMemoryChunk > > The above comment seems to be better fit for ExecScanHashTableForUnmatched(), > instead of ExecParallelPrepHashTableF

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-04-12 Thread Melanie Plageman
On Sun, Jan 26, 2020 at 11:21 PM Kyotaro Horiguchi wrote: > At Sun, 26 Jan 2020 12:22:03 -0800, Andres Freund wrote > in > > On 2020-01-26 16:20:03 +0100, Magnus Hagander wrote: > > > On Sun, Jan 26, 2020 at 1:44 AM Andres Freund wrote: > > > > On 2020-01-25 15:43:41 +0100, Magnus Hagander wrot

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-10-11 Thread Melanie Plageman
On Fri, Oct 8, 2021 at 1:56 PM Andres Freund wrote: > > Hi, > > On 2021-10-01 16:05:31 -0400, Melanie Plageman wrote: > > From 40c809ad1127322f3462e85be080c10534485f0d Mon Sep 17 00:00:00 2001 > > From: Melanie Plageman > > Date: Fri, 24 Sep 2021 17:39:12 -0400 >

make tuplestore helper function

2021-11-02 Thread Melanie Plageman
it made it harder to use MakeTuplestore() in this location. -- melanie [1] https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de From 69e41c23590637c171e985c700b8cfececbbf1f2 Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Tue, 2 Nov 2021 12:20:55 -0400 Subje

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-11-02 Thread Melanie Plageman
; pgstatfuncs.c? > > Export the variable itself? > done but wasn't sure about PGDLLIMPORT > > > > IIRC Thomas Munro had a patch adding a nonatomic_add or such > > > somewhere. Perhaps in the recovery readahead thread? Might be worth using > > > inste

Re: make tuplestore helper function

2021-11-08 Thread Melanie Plageman
ould be an elog() or an assert? Feedback is addressed inline below. On Tue, Nov 2, 2021 at 2:17 PM Alvaro Herrera wrote: > > On 2021-Nov-02, Melanie Plageman wrote: > > > Attached is a patch to create a helper function which creates a > > tuplestore to be used by FUNCAPI-ca

Re: make tuplestore helper function

2021-11-08 Thread Melanie Plageman
On Tue, Nov 2, 2021 at 4:23 PM Justin Pryzby wrote: > > Several places have a conditional value for the first argument (randomAccess), > but your patch changes the behavior to a constant "true". I didn't review the > patch beyond that. > > > @@ -740,18 +724,14 @@ pg_prepared_statement(PG_FUNCTION

Re: Parallel Full Hash Join

2021-11-17 Thread Melanie Plageman
On Sat, Nov 6, 2021 at 11:04 PM Justin Pryzby wrote: > > > Rebased patches attached. I will change status back to "Ready for Committer" > > The CI showed a crash on freebsd, which I reproduced. > https://cirrus-ci.com/task/5203060415791104 > > The crash is evidenced in 0001 - but only ~15% of the

Re: Assertion failure with barriers in parallel hash join

2021-11-17 Thread Melanie Plageman
On Wed, Mar 31, 2021 at 6:25 PM Melanie Plageman wrote: > > On Wed, Mar 17, 2021 at 8:18 AM Thomas Munro wrote: > > > > On Wed, Mar 17, 2021 at 6:58 PM Thomas Munro wrote: > > > According to BF animal elver there is something wrong with this > > > commit.

Re: Parallel Full Hash Join

2021-11-17 Thread Melanie Plageman
small mistake in v8. v9 attached. - Melanie v9-0002-Improve-the-naming-of-Parallel-Hash-Join-phases.patch Description: Binary data v9-0003-Parallel-Hash-Full-Right-Outer-Join.patch Description: Binary data v9-0001-Fix-race-condition-in-parallel-hash-join-batch-cl.patch Description: Binary da

Re: make tuplestore helper function

2021-11-18 Thread Melanie Plageman
On Mon, Nov 8, 2021 at 3:13 PM Justin Pryzby wrote: > > On Mon, Nov 08, 2021 at 02:52:28PM -0500, Melanie Plageman wrote: > > On Tue, Nov 2, 2021 at 4:23 PM Justin Pryzby wrote: > > > > > > Several places have a conditional value for the first argument > > &g

Re: Showing I/O timings spent reading/writing temp buffers in EXPLAIN

2021-11-18 Thread Melanie Plageman
On Sun, Aug 22, 2021 at 9:47 PM Masahiko Sawada wrote: > > On Thu, Aug 19, 2021 at 10:52 PM Ranier Vilela wrote: > > > > Em qui., 19 de ago. de 2021 às 09:21, Masahiko Sawada > > escreveu: > > > The presentation seems a little confusing, wouldn't it be better? > > > > I/O Timings: shared/local

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-11-19 Thread Melanie Plageman
On Mon, May 3, 2021 at 5:24 PM Melanie Plageman wrote: > > So, I've written a patch which avoids doing the immediate fsync for > index builds either by using shared buffers or by queueing sync requests > for the checkpointer. If a checkpoint starts during the index build and >

Re: Adding CI to our tree

2021-11-19 Thread Melanie Plageman
Hi, I reviewed the first patch in this set (v2-0001-ci-Add-CI-for-FreeBSD-Linux-MacOS-and-Windows-uti.patch). For the README, I found the instructions very clear. My only concern is that the cirrus-ci UI will change and the instructions on how to enable cirrus-ci on a repository will not be acces

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-11-23 Thread Melanie Plageman
On Fri, Nov 19, 2021 at 3:11 PM Melanie Plageman wrote: > > On Mon, May 3, 2021 at 5:24 PM Melanie Plageman > wrote: > > > > So, I've written a patch which avoids doing the immediate fsync for > > index builds either by using shared buffers or by queueing sync r

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-11-24 Thread Melanie Plageman
On Fri, Nov 19, 2021 at 11:49 AM Andres Freund wrote: > > +/* > > + * On modern systems this is really just *counter++. On some older systems > > + * there might be more to it, due to inability to read and write 64 bit > > values > > + * atomically. > > + */ > > +static inline void inc_counter(p

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-01 Thread Melanie Plageman
Thanks for the review! On Wed, Nov 24, 2021 at 8:16 PM Justin Pryzby wrote: > You wrote beentry++ at the start of two loops, but I think that's wrong; it > should be at the end, as in the rest of the file (or as a loop increment). > BackendStatusArray[0] is actually used (even though its backend

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-01 Thread Melanie Plageman
v16 (also rebased) attached On Fri, Nov 26, 2021 at 4:16 PM Justin Pryzby wrote: > > On Wed, Nov 24, 2021 at 07:15:59PM -0600, Justin Pryzby wrote: > > There's extraneous blank lines in these functions: > > > > +pgstat_sum_io_path_ops > > +pgstat_report_live_backend_io_path_ops > > +pgstat_recv_r

Re: Removing useless DISTINCT clauses

2018-03-23 Thread Melanie Plageman
nstraintDeps. FIXME whenever not-null constraints get represented * in pg_constraint. */ -- Melanie Plageman

Re: Removing useless DISTINCT clauses

2018-04-05 Thread Melanie Plageman
Hi David, The updated patch looks good to me. I've changed the status to "ready for committer" Thanks

Re: accounting for memory used for BufFile during hash joins

2019-09-05 Thread Melanie Plageman
On Tue, Sep 3, 2019 at 9:36 AM Alvaro Herrera wrote: > On 2019-Jul-11, Tomas Vondra wrote: > > > On Wed, Jul 10, 2019 at 04:51:02PM -0700, Melanie Plageman wrote: > > > > I think implementing support for parallel hashjoin or explicitly > > > disabling it would b

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-09-06 Thread Melanie Plageman
les. I thought that the other workers can move on and stop waiting at the barrier once the lone remaining worker has scanned their outer match status files. All the probe loops would be done, and the worker that is emitting tuples is not referencing the inner side hashtable at all and only the outer batch file and the combined bitmap. -- Melanie Plageman

Re: Memory Accounting

2019-09-18 Thread Melanie Plageman
I wanted to address a couple of questions Jeff asked me off-list about Greenplum's implementations of Memory Accounting. Greenplum has two memory accounting sub-systems -- one is the MemoryContext-based system proposed here. The other memory accounting system tracks "logical" memory owners in glob

Re: Memory Accounting

2019-09-24 Thread Melanie Plageman
On Thu, Sep 19, 2019 at 11:00 AM Jeff Davis wrote: > On Wed, 2019-09-18 at 13:50 -0700, Soumyadeep Chakraborty wrote: > > Hi Jeff, > > Hi Soumyadeep and Melanie, > > Thank you for the review! > > > max_stack_depth max level lazy (ms) eager (ms) > (eage > > r/lazy) > > 2MB 82

Re: Memory Accounting

2019-07-23 Thread Melanie Plageman
like it would be good for memory intensive operators which use a large, representative context. I think the HashTableContext for HashJoin might be one? -- Melanie Plageman

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-30 Thread Melanie Plageman
t; On Wed, Jul 03, 2019 at 02:22:09PM -0700, Melanie Plageman wrote: > >On Tue, Jun 18, 2019 at 3:24 PM Melanie Plageman < > melanieplage...@gmail.com> > > > >Before doing that, I wanted to ask what a desirable fallback condition > >would be. In this patch, fallback to

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-30 Thread Melanie Plageman
On Tue, Jul 30, 2019 at 4:36 PM Robert Haas wrote: > On Tue, Jul 30, 2019 at 2:47 PM Melanie Plageman > wrote: > > I did the "needlessly dumb implementation" Robert mentioned, though, > > I thought about it and couldn't come up with a much smarter way to > &

Re: Adding a test for speculative insert abort case

2019-08-07 Thread Melanie Plageman
On Wed, Jul 24, 2019 at 11:48 AM Andres Freund wrote: > > diff --git a/src/test/isolation/specs/insert-conflict-specconflict.spec > b/src/test/isolation/specs/insert-conflict-specconflict.spec > > index 3a70484fc2..7f29fb9d02 100644 > > --- a/src/test/isolation/specs/insert-conflict-specconflict.

Re: Cleanup isolation specs from unused steps

2019-08-19 Thread Melanie Plageman
o the check that all steps have been used in dry_run mode instead of when running the tests for real? -- Melanie Plageman

Re: Cleanup isolation specs from unused steps

2019-08-21 Thread Melanie Plageman
ly different syntax. It doesn't use isolationtester at all. So, I haven't had a use case to add long options to isolationtester yet :) -- Melanie Plageman

Re: Cleanup isolation specs from unused steps

2019-08-21 Thread Melanie Plageman
round does not > induce really much extra maintenance cost. > So, I think I completely misunderstood the purpose of 'dry-run'. If no one is using it, having a check for unused steps in dry-run may not be useful. +1 to renaming it to --print-permutations and, potentially, adding --print-all-permutations -- Melanie Plageman

Re: Adding a test for speculative insert abort case

2019-08-21 Thread Melanie Plageman
On Wed, Aug 7, 2019 at 1:47 PM Melanie Plageman wrote: > > > On Wed, Jul 24, 2019 at 11:48 AM Andres Freund wrote: > >> > diff --git a/src/test/isolation/specs/insert-conflict-specconflict.spec >> b/src/test/isolation/specs/insert-conflict-specconflict.spec >&g

Re: Cleanup isolation specs from unused steps

2019-08-22 Thread Melanie Plageman
On Wed, Aug 21, 2019 at 12:16 PM Alvaro Herrera wrote: > On 2019-Aug-21, Melanie Plageman wrote: > > > In Greenplum, we mainly add new tests to a separate isolation > > framework (called isolation2) which uses a completely different > > syntax. It doesn't use

Re: Cleanup isolation specs from unused steps

2019-08-22 Thread Melanie Plageman
On Thu, Aug 22, 2019 at 6:53 PM Michael Paquier wrote: > On Thu, Aug 22, 2019 at 10:20:48AM -0700, Melanie Plageman wrote: > > So, there is some historical context as to why it is a separate test > suite. > > And some of the differences are specific to Greenplum -- e.g. needing

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Melanie Plageman
h join details. > > This made me think about other explain wishlist items. For parallel hashjoin, I would find it useful to know which batches each worker participated in (maybe just probing to start with, but loading would be great too). I'm not sure anyone else (especially users) would care about this, though. -- Melanie Plageman

Re: Memory-Bounded Hash Aggregation

2019-12-04 Thread Melanie Plageman
On Thu, Nov 28, 2019 at 9:47 AM Tomas Vondra wrote: > On Wed, Nov 27, 2019 at 02:58:04PM -0800, Jeff Davis wrote: > >On Wed, 2019-08-28 at 12:52 -0700, Taylor Vesely wrote: > >> Right now the patch always initializes 32 spill partitions. Have you > >> given > >> any thought into how to intelligen

Re: Extracting only the columns needed for a query

2020-01-02 Thread Melanie Plageman
check IS_DUMMY_REL in > extract_used_columns? > > I am wondering, when IS_DUMMY_REL is true for a relation, do we reference the associated RTE later? It seems like if it is a dummy rel, we wouldn't scan it. It still makes sense to add it to extract_used_columns(), I think, to avoid any wasted loops through the rel's expressions. Thanks for the idea! -- Melanie Plageman v2-0001-Plan-time-extraction-of-scan-cols.patch Description: Binary data

Re: accounting for memory used for BufFile during hash joins

2020-01-03 Thread Melanie Plageman
e prototype. [1] https://www.postgresql.org/message-id/CAAKRu_YsWm7gc_b2nBGWFPE6wuhdOLfc1LBZ786DUzaCPUDXCA%40mail.gmail.com -- Melanie Plageman

Re: Extracting only the columns needed for a query

2020-01-07 Thread Melanie Plageman
I just wanted to address a question we got about making scanCols instead of using RelOptInfo->attr_needed. David Rowley had asked: > For planning, isn't this information already available via either > targetlists or from the RelOptInfo->attr_needed array combined with > min_attr and max_attr? at

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2020-01-09 Thread Melanie Plageman
Blocks (from > https://en.wikipedia.org/wiki/Block_nested_loop, though, no, strike > that, blocks are also super overloaded). > Hmmm. I think loop is kinda confusing. "fragment" has potential. I also thought of "piece". That is actually where I am leaning now. What do you think? [1] https://www.postgresql.org/message-id/flat/CA%2BhUKG%2BA6ftXPz4oe92%2Bx8Er%2BxpGZqto70-Q_ERwRaSyA%3DafNg%40mail.gmail.com -- Melanie Plageman

Re: Parallel leader process info in EXPLAIN

2020-01-17 Thread Melanie Plageman
oice (even with xxx the actual numbers, I would have thought that there would be an EXPLAIN VERBOSE with leader participation somewhere in regress). -- Melanie Plageman 0001-Show-parallel-leader-stats-in-EXPLAIN-output-v4.patch Description: Binary data

Re: Parallel leader process info in EXPLAIN

2020-01-24 Thread Melanie Plageman
So, I think from a code review perspective the code in the patches LGTM. As for the EXPLAIN ANALYZE tests--I don't see that many of them in regress, so maybe that's because they aren't normally very useful. In this case, it would only be to protect against regressions in printing the leader instru

Calling PrepareTempTablespaces in BufFileCreateTemp

2019-04-22 Thread Melanie Plageman
temp_tablespaces GUC to ensure I create the spill files there if it is set. Thanks, Melanie Plageman

Re: Calling PrepareTempTablespaces in BufFileCreateTemp

2019-04-24 Thread Melanie Plageman
w callers of the > second and third that I'm not sure that'd be better. Thoughts? > > I think an assertion is sufficiently clear for GetNextTempTableSpace based on what it does and its current callers. The same is probably true for GetTempTableSpaces. -- Melanie Plageman

Re: Calling PrepareTempTablespaces in BufFileCreateTemp

2019-04-25 Thread Melanie Plageman
the two. > > Would an existing test cover the code after moving PrepareTempTablespaces into OpenTemporaryFile? -- Melanie Plageman

Re: Calling PrepareTempTablespaces in BufFileCreateTemp

2019-04-30 Thread Melanie Plageman
descriptor API should do catalog lookups. Is this correct? [1] https://www.postgresql.org/message-id/11777.1556133426%40sss.pgh.pa.us -- Melanie Plageman

Adding a test for speculative insert abort case

2019-04-30 Thread Melanie Plageman
Today, while poking around the table_complete_speculative code which Ashwin mentioned in [1], we were trying to understand when exactly we would complete a speculative insert by aborting. We added a logging message to heapam_tuple_complete_speculative before it calls heap_abort_speculative and ran

Re: Adding a test for speculative insert abort case

2019-04-30 Thread Melanie Plageman
, I did notice that, in the initial check of the index by s2, XactLockTableWait is called with reason_wait as XLTW_InsertIndex (even though we are just trying to check it, so maybe it knows our intentions:)) Is there something I can do in the test to allow my check to go through but the insert to have to wait? -- Melanie Plageman

Re: Adding a test for speculative insert abort case

2019-05-01 Thread Melanie Plageman
about how to use it [1]. The Greenplum implementation is not documented particularly well in the code, but, it is something that folks working on Greenplum have talked about modifying and proposing to Postgres. [1] http://engineering.pivotal.io/post/testing_greenplum_database_using_fault_injection/ -- Melanie Plageman

Re: accounting for memory used for BufFile during hash joins

2019-05-06 Thread Melanie Plageman
king a look at the patches is that I prefer the first approach--increasing the resize threshhold. The second patch, the per-slice-overflow patch, adds a major new mechanic to hashjoin in order to address what is, based on my understanding, an edge case. -- Melanie Plageman

Re: accounting for memory used for BufFile during hash joins

2019-05-07 Thread Melanie Plageman
sed. hashtable->nbatch_inmemory was 2 for me, thus, nbatch_tmp was 2 so, I didn't meet this condition if (nbatch_tmp > hashtable->nbatch_inmemory) since I just set nbatch_tmp using hashtable->nbatch_inmemory So, I didn't increase the number of slices, which is what I was expecting. What happens when hashtable->nbatch_inmemory is equal to nbatch_tmp? -- Melanie Plageman

Re: accounting for memory used for BufFile during hash joins

2019-05-07 Thread Melanie Plageman
ou switched to NLJ on a batch-by-batch basis and did it before starting execution of the join but after building the inner side of the hash table. That way, no tuples will have been sent to other nodes yet. -- Melanie Plageman

Re: Adding a test for speculative insert abort case

2019-05-15 Thread Melanie Plageman
how a successful UPSERT "controller_show" I was also wondering: Is it possible that one of the "controller_unlock_*" functions will get called before the session with the upsert has had a chance to move forward in its progress and be waiting on that lock? That is, given that we don't check that the sessions are waiting on the locks before unlocking them, is there a race condition? I noticed that there is not a test case which would cover the speculative wait codepath. This seems much more challenging, however, it does seem like a worthwhile test to have. -- Melanie Plageman

Re: Adding a test for speculative insert abort case

2019-05-15 Thread Melanie Plageman
quot; "s1_upsert" "s2_upsert" "controller_show" "controller_unlock_1_1" "controller_unlock_2_1" "controller_unlock_1_3" "controller_unlock_2_3" "controller_unlock_1_2" "s1_magically_pause_before_complete_speculative" # put s2 in speculative wait "controller_unlock_2_2" "s1_magically_unpause_before_complete_speculative" So, how would another lock on another index keep s1 from clearing the speculative token after it has updated the index? -- Melanie Plageman specconflict_permutation_comment_update.patch Description: Binary data

Re: Adding a test for speculative insert abort case

2019-05-16 Thread Melanie Plageman
had for the wording of the comments overall in the above email I sent). -- Melanie Plageman 0002-Add-test-to-validate-speculative-wait-is-performed.patch Description: Binary data

Re: Adding a test for speculative insert abort case

2019-05-16 Thread Melanie Plageman
On Thu, May 16, 2019 at 2:03 PM Andres Freund wrote: > Hi, > > On 2019-05-16 13:59:47 -0700, Melanie Plageman wrote: > > On Wed, May 15, 2019 at 10:32 PM Ashwin Agrawal > wrote: > > > > > > > > The second index would help to hold the session after inser

Re: Adding a test for speculative insert abort case

2019-05-17 Thread Melanie Plageman
On Thu, May 16, 2019 at 8:46 PM Melanie Plageman wrote: > > I squashed the changes I suggested in previous emails, Ashwin's patch, my > suggested updates to that patch, and the index order check all into one > updated > patch attached. > > I realized that the num

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-17 Thread Melanie Plageman
e_NLJ and when making batches for the outer side, make one "batch" that has all the tuples from the outer side which the inner side batch which was flagged will do NLJ with. -- Melanie Plageman

describe working as intended?

2019-05-17 Thread Melanie Plageman
nspname | relname ---+---+- 23609 | public| t1 23612 | pg_temp_4 | t1 (2 rows) So, without much more digging, is the current behavior of describe intended? I couldn't find an email thread discussing this with the search terms I tried. (I noticed it on master and checked 11 as well and got the sam

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-20 Thread Melanie Plageman
On Sun, May 19, 2019 at 4:07 PM Thomas Munro wrote: > On Sat, May 18, 2019 at 12:15 PM Melanie Plageman > wrote: > > On Thu, May 16, 2019 at 3:22 PM Thomas Munro > wrote: > >> Admittedly I don't have a patch, just a bunch of handwaving. One > >> rea

Re: describe working as intended?

2019-05-21 Thread Melanie Plageman
ntax to display tables with such name in all schemas. > > regards, Sergei > Thanks! I suppose it would behoove me to check the documentation before resorting to looking at the source code :) -- Melanie Plageman

Re: accounting for memory used for BufFile during hash joins

2019-05-21 Thread Melanie Plageman
On Wed, May 8, 2019 at 8:08 AM Tomas Vondra wrote: > On Tue, May 07, 2019 at 05:30:27PM -0700, Melanie Plageman wrote: > > One thing I was a little confused by was the nbatch_inmemory member > > of the hashtable. The comment in ExecChooseHashTableSize says that > >

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-03 Thread Melanie Plageman
On Sun, May 19, 2019 at 4:07 PM Thomas Munro wrote: > On Sat, May 18, 2019 at 12:15 PM Melanie Plageman > wrote: > > On Thu, May 16, 2019 at 3:22 PM Thomas Munro > wrote: > >> Admittedly I don't have a patch, just a bunch of handwaving. One > >> rea

Re: [HACKERS] [PATCH v2] Add and report the new "session_read_only" GUC pseudo-variable.

2018-11-11 Thread Melanie Plageman
On Mon, Sep 24, 2018 at 9:28 AM Elvis Pranskevichus wrote: > On Thursday, March 1, 2018 4:25:16 AM EDT Andres Freund wrote: > > A CF entry for this patch has been created yesterday, without any > > changes having happend since the above status update. Given that > > there's been no progress for

Re: BUG #15160: planner overestimates number of rows in join when there are more than 200 rows coming from CTE

2018-11-15 Thread Melanie Plageman
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested This patch applies cleanly and works for the case described in the or

Re: BUG #15160: planner overestimates number of rows in join when there are more than 200 rows coming from CTE

2018-11-16 Thread Melanie Plageman
Thanks for the quick responses. I've put some inline follow-up questions. On a separate note, I had one additional code clarity feedback. I felt that eqjoinsel could be reorganized a bit for readability/clarity for the reader. For example, eqjoinsel_inner uses only the AttStatsSlots up until here

Re: BUG #15160: planner overestimates number of rows in join when there are more than 200 rows coming from CTE

2018-11-20 Thread Melanie Plageman
Given that you have addressed all of my feedback and that it's a pretty low-risk change, I will change the status to "ready for committer". There are a couple of minor follow-up clarifications inline that relate mostly to the questions that I asked in previous emails. I did have one other questio

Re: Removing useless DISTINCT clauses

2018-03-20 Thread Melanie Plageman
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed This is a review of the patch to remove useless DISTINCT colu

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-10 Thread Melanie Plageman
On Wed, Jan 10, 2024 at 3:54 PM Robert Haas wrote: > > On Tue, Jan 9, 2024 at 5:42 PM Melanie Plageman > wrote: > > Done in attached v6. > > Don't kill me, but: > > + /* > +* For now, pass no_indexes == f

Re: Show WAL write and fsync stats in pg_stat_io

2024-01-10 Thread Melanie Plageman
I have code review feedback as well, but I've saved that for my next email. On Wed, Jan 3, 2024 at 8:11 AM Nazir Bilal Yavuz wrote: > > On Sun, 31 Dec 2023 at 03:58, Michael Paquier wrote: > > > > On Tue, Dec 26, 2023 at 03:35:52PM +0300, Nazir Bilal Yavuz wrote: > > > On Tue, 26 Dec 2023 at 13:

  1   2   3   4   5   6   7   8   9   10   >