acros - all they do is to
make it harder to understand the code.
From 236580a0dd381e245a459d0e8769bd30ba2d79e3 Mon Sep 17 00:00:00 2001
From: Reid Thompson
Date: Tue, 6 Feb 2024 09:17:28 -0500
Subject: [PATCH] Remove Start* macros in postmaster.c to ease understanding of
code. Per comment in thread he
On Thu, 2024-01-25 at 01:51 +0200, Heikki Linnakangas wrote:
>
> And here we go. BackendID is now a 1-based index directly into the
> PGPROC array.
>
Would it be worthwhile to also note in this comment FIRST_AUX_PROC's
and IsAuxProcess()'s dependency on B_ARCHIVER and it's location in the
enum
On Fri, 2024-01-26 at 13:51 +0900, Masahiko Sawada wrote:
> On Fri, Jan 26, 2024 at 1:24 PM wrote:
> >
> > On Thu, 2024-01-25 at 16:11 -0500, reid.thomp...@crunchydata.com wrote:
> > >
> > > I walked through v6 and didn't note any issues.
>
> Thank you for reviewing the patch!
>
Happy to.
>
>
On Thu, 2024-01-25 at 16:11 -0500, reid.thomp...@crunchydata.com wrote:
>
> I walked through v6 and didn't note any issues.
>
> I do want to ask, the patch alters ReorderBufferReturnTupleBuf() and
> drops the unused parameter ReorderBuffer *rb. It seems that
> ReorderBufferFreeSnap(), ReorderBuff
On Thu, 2024-01-25 at 16:16 +0300, Aleksander Alekseev wrote:
> Hi,
>
> > > Here is the corrected patch.
> >
> > Thank you for updating the patch! I have some comments:
>
> Thanks for the review.
>
> > -tuple = (ReorderBufferTupleBuf *)
> > +tuple = (HeapTuple)
> >
On Fri, 2024-01-19 at 17:37 -0500, reid.thomp...@crunchydata.com wrote:
> On Sat, 2024-01-13 at 10:31 -0700, Roberto Mello wrote:
> >
> > I can add a suggestion for the user to consider increasing
> > shared_buffers in accordance with higher max_connections, but it
> > would be better if there was
On Sat, 2024-01-13 at 10:31 -0700, Roberto Mello wrote:
> On Fri, Jan 12, 2024 at 3:15 PM Cary Huang
> wrote:
> > I think it is good to warn the user about the increased allocation
> > of memory for certain parameters so that they do not abuse it by
> > setting it to a huge number without knowing
On Sun, 2023-07-09 at 13:24 -0400, Joseph Koshakow wrote:
>
> I've broken up this patch into three logical commits and attached
> them.
> None of the actual code has changed.
>
> Thanks,
> Joe Koshakow
I made a another pass through the separated patches, it looks good to
me.
Thanks,
Reid
On Sat, 2023-07-08 at 13:18 -0400, Joseph Koshakow wrote:
> Jacob Champion writes:
> > Hi Joe, here's a partial review:
>
> Thanks so much for the review!
>
> > I'm new to this code, but I agree that the use of `type` and the
> > lookahead are not particularly obvious/intuitive. At the very
> >
On Mon, 2023-05-22 at 08:42 -0400, reid.thomp...@crunchydata.com wrote:
More followup to the above.
>
> I experimented on my system regarding
> "The simple query select * from generate_series(0, 1000) shows roughly
> 18.9 % degradation on my test server."
>
> My laptop:
> 32GB ram
> 11th G
-
+ t| t| t| t
+(1 row)
+
+-- ensure that pg_stat_memory_allocation view exists
+SELECT
+pid > 0, allocated_bytes >= 0, aset_allocated_bytes >= 0, dsm_allocated_bytes >= 0, generation_allocated_bytes >= 0, slab_allocated_bytes >= 0
+FROM
+pg_stat_m
On Wed, 2023-05-17 at 23:07 -0400, reid.thomp...@crunchydata.com wrote:
> Thanks for the feedback.
>
> I'm plannning to look at this.
>
> Is your benchmark something that I could utilize? I.E. is it a set of
> scripts or a standard test from somewhere that I can duplicate?
>
> Thanks,
> Reid
>
On Wed, 2023-04-19 at 23:28 +, Arne Roland wrote:
> > Thank you! I just tried our benchmark and got a performance
> > degration > of around 28 %, which is way better than the last
> > patch.
> >
> > The simple query select * from generate_series(0, 1000) shows >
> > roughly 18.9 % degradat
34514ae2bebe5e3ab2a0b5b680d3932b5e7706ee Mon Sep 17 00:00:00 2001
From: Reid Thompson
Date: Sat, 4 Jun 2022 22:23:59 -0400
Subject: [PATCH 2/2] Add the ability to limit the amount of memory that can be
allocated to backends.
This builds on the work that adds backend memory allocated tracking.
Add
On Thu, 2023-03-30 at 16:11 +, John Morris wrote:
> Hi Reid!
> Some thoughts
> I was looking at lmgr/proc.c, and I see a potential integer overflow
> - bothmax_total_bkend_mem and result are declared as “int”, so the
> expression “max_total_bkend_mem * 1024 * 1024 - result * 1024 * 1024”
> cou
From 4dd47f04764b5df9c3962d9fdb4096398bf85dfd Mon Sep 17 00:00:00 2001
From: Reid Thompson
Date: Sat, 4 Jun 2022 22:23:59 -0400
Subject: [PATCH 2/2] Add the ability to limit the amount of memory that can be
allocated to backends.
This builds on the work that adds backend memory allocated tracking.
Add GUC var
memory is accumulated and
returned to the shared counter upon meeting a threshold and/or upon process
exit. At this point arbitrarily picked 1MB as the initial allowance and
return threshold.
Thanks,
Reid
From e044bacedab503d1cd732146e1b9947406191bb6 Mon Sep 17 00:00:00 2001
From: Reid Th
Regarding the shared counter noted here,
> What you could do is to have a single, imprecise, shared counter for the total
> memory allocation, and have a backend-local "allowance". When the allowance is
> used up, refill it from the shared counter (a single atomic op).
Is there a preferred or sug
On Mon, 2023-01-23 at 12:31 -0800, Andres Freund wrote:
> Hi,
>
> I think it's basically still waiting on author, until the O(N) cost is gone
> from the overflow limit check.
>
> Greetings,
>
> Andres Freund
Yes, just a rebase. There is still work to be done per earlier in the
thread.
I do wan
On Thu, 2023-01-19 at 16:50 +0530, vignesh C wrote:
>
> The patch does not apply on top of HEAD as in [1], please post a rebased
> patch:
>
> Regards,
> Vignesh
rebased patch attached
Thanks,
Reid
From b32a346d6e0e00c568e9a285ad15fc2703998c26 Mon Sep 17 00:00:00 2001
F
On Mon, 2023-01-09 at 18:31 -0800, Andres Freund wrote:
> Hi,
>
> On 2023-01-05 13:44:20 -0500, Reid Thompson wrote:
> > This new field displays the current bytes of memory allocated to the
> > backend process. It is updated as memory for the process is
> > malloc&#x
.ca...@crunchydata.com
Thanks,
Reid
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp...@crunchydata.com
www.crunchydata.com
On Thu, 8 Dec 2022 at 19:44, Reid Thompson
wrote:
>
> On Sun, 2022-11-27 at 09:40 -0600, Justin Pryzby wrote:
> > > ...
> > > I still wonder whether there needs to be a separate CF entry for
> > > the 0001 patch. One issue is that there's two different li
On Tue, 2023-01-03 at 16:22 +0530, vignesh C wrote:
>
> The patch does not apply on top of HEAD as in [1], please post a
> rebased patch:
> ...
> Regards,
> Vignesh
>
Attached is rebased patch, with some updates related to committed changes.
Thanks,
Reid
--
Reid Tho
On Tue, 2022-12-06 at 10:32 -0800, Andres Freund wrote:
> Hi,
>
> On 2022-11-26 22:22:15 -0500, Reid Thompson wrote:
> > rebased/patched to current master && current pg-stat-activity-
> > backend-memory-allocated
>
> This version fails to build with msvc,
ead if everyone else
is. If there's no argument against, then patches after today will go
to the "Add the ability to limit the amount of memory that can be
allocated to backends" thread
https://www.postgresql.org/message-id/bd57d9a4c219cc1392665fd5fba61dde8027b3da.ca...@crunchy
On Fri, 2022-12-02 at 09:18 -0800, Andres Freund wrote:
> Hi,
>
> On 2022-12-02 11:09:30 -0500, Reid Thompson wrote:
> > It appears to me that Postmaster populates the local variable that
> > *my_allocated_bytes points to. That allocation is passed to forked
> > chil
On Mon, 2022-11-28 at 10:59 -0800, Andres Freund wrote:
> On 2022-11-26 22:10:06 -0500, Reid Thompson wrote:
> > - zero allocated bytes after fork to avoid double counting
> > postmaster allocations
>
> I still don't understand this - postmaster shouldn't be coun
On Sat, 2022-11-26 at 22:10 -0500, Reid Thompson wrote:
> Code rebased to current master.
> Updated to incorporate additional recommendations from the the list
> - add units to variables in view
> - remove 'backend_' prefix from variables/functions
> - update
On Thu, 2022-11-03 at 11:48 -0400, Reid Thompson wrote:
> On Tue, 2022-10-25 at 11:49 -0400, Reid Thompson wrote:
>
> Rebased to current. Add a couple changes per conversation with D
> Christensen (include units in field name, group field with
> backend_xid
> and back
On Wed, 2022-11-09 at 08:54 -0500, Reid Thompson wrote:
> Hi Andres,
> Thanks for looking at this and for the feedback. Responses inline
> below.
>
>> On Fri, 2022-11-04 at 19:41 -0700, Andres Freund wrote:
> > Hi,
> >
> On 2022-11-04 08:56:13 -0400, Reid
On Wed, 2022-11-09 at 09:23 -0500, Reid Thompson wrote:
> Hi Melanie,
> Thank you for looking at this and for the feedback. Responses inline
> below.
>
> On Mon, 2022-11-07 at 16:17 -0500, Melanie Plageman wrote:
> >
> > It doesn't seem like you need the b
reporting to reduce number of functions and
branches/reduce performance hit
- zero allocated bytes after fork to avoid double counting postmaster
allocations
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp...@crunchydata.com
www.crunchydata
Hi Melanie,
Thank you for looking at this and for the feedback. Responses inline
below.
On Mon, 2022-11-07 at 16:17 -0500, Melanie Plageman wrote:
> On Fri, Nov 04, 2022 at 08:56:13AM -0400, Reid Thompson wrote:
> > From a8de5d29c0c6f10962181926a49ad4fec1e52bd1 Mon Sep 17 00:00:00
Hi Andres,
Thanks for looking at this and for the feedback. Responses inline below.
On Fri, 2022-11-04 at 19:41 -0700, Andres Freund wrote:
> Hi,
>
> On 2022-11-04 08:56:13 -0400, Reid Thompson wrote:
>
> I'm not convinced that counting DSM values this way is quite right.
&g
On Fri, 2022-11-04 at 11:06 +0100, Drouvot, Bertrand wrote:
> Hi,
>
> It looks like the patch does not apply anymore since b1099eca8f.
>
> Regards,
>
Thanks,
rebased to current master attached.
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp.
On Tue, 2022-10-25 at 11:49 -0400, Reid Thompson wrote:
> Hi Arne,
>
> On Mon, 2022-10-24 at 15:27 +, Arne Roland wrote:
> > Hello Reid,
> >
> > could you rebase the patch again? It doesn't apply currently
> > (http://cfbot.cputube.org/patch_40_3
On Tue, 2022-10-25 at 14:51 -0400, Reid Thompson wrote:
> patch rebased to current master
>
actually attach the patch
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp...@crunchydata.com
www.crunchydata.com
From ab654a48ec7bfbc3bc377c5757a04f1756e72e79 Mon Sep 17 00
patch rebased to current master
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp...@crunchydata.com
www.crunchydata.com
; Arne
From ab654a48ec7bfbc3bc377c5757a04f1756e72e79 Mon Sep 17 00:00:00 2001
From: Reid Thompson
Date: Thu, 11 Aug 2022 12:01:25 -0400
Subject: [PATCH 1/2] Add tracking of backend memory allocated to
pg_stat_activity
This new field displays the current bytes of memory allocated to the
backe
gt;
> patching file src/backend/utils/misc/postgresql.conf.sample
>
rebased patches attached.
Thanks,
Reid
From 0e7010c53508d5a396edd16fd9166abe431f5dbe Mon Sep 17 00:00:00 2001
From: Reid Thompson
Date: Thu, 11 Aug 2022 12:01:25 -0400
Subject: [PATCH 1/2] Add tracking of backend memory
On Fri, 2022-09-09 at 12:14 -0500, Justin Pryzby wrote:
> On Sat, Sep 03, 2022 at 11:40:03PM -0400, Reid Thompson wrote:
> > > > + 0, 0, INT_MAX,
> > > > + NULL, NULL, NULL
> > > I think this needs a maximum like INT_MAX/1024/1024
>
nd on this idea.
Another thought is, rather than just failing the query/transaction we
have the affected backend do a clean exit, freeing all it's resources.
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp...@crunchydata.com
www.crunchydata.com
On Thu, 2022-09-01 at 11:48 +0900, Kyotaro Horiguchi wrote:
> >
> > The patch seems to limit both of memory-context allocations and DSM
> > allocations happen on a specific process by the same budget. In the
> > fist place I don't think it's sensible to cap the amount of DSM
> > allocations by per
On Thu, 2022-09-01 at 13:43 +0900, Kyotaro Horiguchi wrote:
>
> > @@ -916,6 +930,7 @@ AllocSetAlloc(MemoryContext context, Size size)
> > return NULL;
> >
> > context->mem_allocated += blksize;
> > + pgstat_report_backend_mem_allocated_increase(blksi
>
d for, or otherwise Assert() that the value is not
> negative. I don't know how expensive it'd be to have conditionals
> for
> each decrement, but maybe the value would only be decremented at
> strategic times, like at transaction commit or backend shutdown.
>
--
Reid Th
ed
> > here asit's
>
> as it's
Corrected
> errmsg() doesn't require the outside paranthesis since a couple years
> ago.
Corrected
> > > mem_allocated cast to
> add a comma before "cast" ?
Corrected
Patch with the corrections attached
open for discussion as to what should/should not be
included. Backend memory allocations are displayed in the
pg_stat_activity view.
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp...@crunchydata.com
www.crunchydata.com
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml
;
context rather than the TopMemoryContext. Prior to the change,
leaving an idle laptop PG instance running just shy of 3 days saw the
autovacuum launcher process grow to 42MB with most of that growth in
TopMemoryContext due to the palloc allocations issued with autovacuum
worker startup.
--
Reid
aned up on postmaster startup. This may result in decreasing the sum
without a prior increment. We limit the floor of backend_mem_allocated
to zero.
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp...@crunchydata.com
www.crunchydata.com
diff --git a/doc/src/sgml/monitoring.
ory for these
two contexts until it was needed?
> regards.
>
> --
> Kyotaro Horiguchi
> NTT Open Source Software Center
--
Reid Thompson
Senior Software Engineer
Crunchy Data, Inc.
reid.thomp...@crunchydata.com
www.crunchydata.com
diff --git a/src/backend/utils/activity/pg
name);
return (MemoryContext) slab;
}
--
Reid Thompson
Senior Software Engineer
Crunchy Data
Hi,
There are instances where pgstat_setup_memcxt() and
pgstat_prep_pending_entry() are invoked before the CacheMemoryContext
has been created. This results in PgStat* contexts being created
without a parent context. Most easily reproduced/seen in autovacuum
worker via pgstat_setup_memcxt().
A
On Wed, 2019-03-27 at 15:07 +0100, Andreas Karlsson wrote:
> [EXTERNAL SOURCE]
>
>
>
> On 3/27/19 2:51 PM, Tomas Vondra wrote:
> > I think the consensus in this thread (and the previous ancient ones) is
> > that it's not worth it. It's one thing to introduce new commands with the
> > pg_ prefix,
54 matches
Mail list logo
