Re: Add the ability to limit the amount of memory that can be allocated to backends.

2025-01-08 Thread James Hunter
On Mon, Jan 6, 2025 at 1:07 PM Jim Nasby wrote: > > I’ve been saying “workload management” for lack of a better term, but my > initial suggestion upthread was to simply stop allowing new transactions to > start if global work_mem consumption exceeded some threshold. That’s > simplistic enough t

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2025-01-06 Thread Tomas Vondra
On 1/6/25 22:07, Jim Nasby wrote: > On Jan 2, 2025, at 9:21 PM, Tomas Vondra wrote: >> >>> That said, I do think a workload manager would be more effective than >>> trying to limit total connections. >> >> The "workload management" concept is so abstract I find it very >> difficult to discuss with

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2025-01-06 Thread Jim Nasby
On Jan 2, 2025, at 9:21 PM, Tomas Vondra wrote: > >> That said, I do think a workload manager would be more effective than >> trying to limit total connections. > > The "workload management" concept is so abstract I find it very > difficult to discuss without much more detail about how would it

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2025-01-03 Thread James Hunter
On Thu, Jan 2, 2025 at 7:21 PM Tomas Vondra wrote: > > I'm not opposed to having a some sort of "workload management" (similar > to what's available in some databases), but my guess is that's (at > least) an order of magnitude more complex than introducing the memory > limit discussed here. I can

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2025-01-03 Thread James Hunter
I think this discussion is getting away from a manageable scope of work... On Thu, Jan 2, 2025 at 1:09 PM Jim Nasby wrote: > That’s basically my argument for having workload management. If a system > becomes loaded enough for the global limit to start kicking in it’s likely > that query respon

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2025-01-02 Thread Tomas Vondra
On 1/2/25 22:09, Jim Nasby wrote: > >> On Dec 31, 2024, at 5:41 PM, Tomas Vondra wrote: >> >> On 12/31/24 21:46, Jim Nasby wrote: >>> On Dec 30, 2024, at 7:05 PM, James Hunter >>> wrote: On Sat, Dec 28, 2024 at 11:24 PM Jim Nasby wrote: > > IMHO none of this will be very sane

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2025-01-02 Thread Jim Nasby
> On Dec 31, 2024, at 5:41 PM, Tomas Vondra wrote: > > On 12/31/24 21:46, Jim Nasby wrote: >> On Dec 30, 2024, at 7:05 PM, James Hunter wrote: >>> >>> On Sat, Dec 28, 2024 at 11:24 PM Jim Nasby wrote: IMHO none of this will be very sane until we actually have cluster-level li

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-31 Thread Tomas Vondra
On 12/31/24 21:46, Jim Nasby wrote: > On Dec 30, 2024, at 7:05 PM, James Hunter wrote: >> >> On Sat, Dec 28, 2024 at 11:24 PM Jim Nasby wrote: >>> >>> IMHO none of this will be very sane until we actually have cluster-level >>> limits. One sudden burst in active connections and you still OOM the

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-31 Thread Jim Nasby
On Dec 30, 2024, at 7:05 PM, James Hunter wrote: > > On Sat, Dec 28, 2024 at 11:24 PM Jim Nasby wrote: >> >> IMHO none of this will be very sane until we actually have cluster-level >> limits. One sudden burst in active connections and you still OOM the >> instance. > > Fwiw, PG does support

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-30 Thread James Hunter
On Mon, Dec 30, 2024 at 3:12 PM David Rowley wrote: > > On Sat, 28 Dec 2024 at 08:14, James Hunter wrote: > > 2. We use this backend_work_mem to "adjust" work_mem values used by > > the executor. (I don't care about the optimizer right now -- optimizer > > just does its best to predict what will

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-30 Thread James Hunter
On Sat, Dec 28, 2024 at 11:24 PM Jim Nasby wrote: > > IMHO none of this will be very sane until we actually have cluster-level > limits. One sudden burst in active connections and you still OOM the instance. Fwiw, PG does support "max_connections" GUC, so a backend/connection - level limit, time

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-30 Thread James Hunter
On Sat, Dec 28, 2024 at 10:26 AM Jeremy Schneider wrote: > Thanks for the feedback, Jeremy! > While I don't have a detailed design in mind, I'd like to add a strong > +1 on the general idea that work_mem is hard to effectively use because > queries can vary so widely in how many nodes might need

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-30 Thread James Hunter
On Sat, Dec 28, 2024 at 6:57 AM Tomas Vondra wrote: > > On 12/28/24 13:36, Anton A. Melnikov wrote: > > > > ... In more details let me suggest > > the following steps or parts: > > 1) realize memory limitation for a separate backend independent from the > > work_mem GUC; > > 2) add workmem-like li

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-30 Thread James Hunter
On Mon, Dec 30, 2024 at 2:56 PM David Rowley wrote: > > On Tue, 31 Dec 2024 at 10:11, James Hunter wrote: > > Does PostgreSQL currently rescan Hash Joins when they are "no longer > > needed," to free work_mem early? If so, then I would try to reuse this > > existing logic to decide which nodes ne

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-30 Thread David Rowley
On Sat, 28 Dec 2024 at 08:14, James Hunter wrote: > 2. We use this backend_work_mem to "adjust" work_mem values used by > the executor. (I don't care about the optimizer right now -- optimizer > just does its best to predict what will happen at runtime.) While I do want to see improvements in thi

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-30 Thread David Rowley
On Tue, 31 Dec 2024 at 10:11, James Hunter wrote: > Does PostgreSQL currently rescan Hash Joins when they are "no longer > needed," to free work_mem early? If so, then I would try to reuse this > existing logic to decide which nodes need work_mem concurrently. > > If not, then all nodes that use w

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-30 Thread James Hunter
On Fri, Dec 27, 2024 at 5:48 PM Tomas Vondra wrote: > Whenever I've been thinking about this in the past, it wasn't clear to > me how would we know when to start adjusting work_mem, because we don't > know which nodes will actually use work_mem concurrently. You certainly know the PostgreSQL sou

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-28 Thread Jim Nasby
On Dec 28, 2024, at 12:26 PM, Jeremy Schneider wrote: > > While I don't have a detailed design in mind, I'd like to add a strong > +1 on the general idea that work_mem is hard to effectively use because > queries can vary so widely in how many nodes might need work memory. > > I'd almost like to

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-28 Thread Jeremy Schneider
On Sat, 28 Dec 2024 15:57:44 +0100 Tomas Vondra wrote: > > Not sure a simple memory limit like in the patch (which just adds > memory accounting + OOM after hitting the limit) can work as anything > but a the last safety measure. It seems to me the limit would have to > be set to a value that's m

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-28 Thread Tomas Vondra
On 12/28/24 13:36, Anton A. Melnikov wrote: > Hi! > > On 28.12.2024 04:48, Tomas Vondra wrote: >> On 12/27/24 20:14, James Hunter wrote: >>> Reviving this thread, because I am thinking about something related -- >>> please ignore the "On Fri, Dec 27, 2024" date, this seems to be an >>> artifact

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-28 Thread Anton A. Melnikov
Hi! On 28.12.2024 04:48, Tomas Vondra wrote: On 12/27/24 20:14, James Hunter wrote: Reviving this thread, because I am thinking about something related -- please ignore the "On Fri, Dec 27, 2024" date, this seems to be an artifact of me re-sending the message, from the list archive. The origina

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-27 Thread Tomas Vondra
On 12/27/24 20:14, James Hunter wrote: > Reviving this thread, because I am thinking about something related -- > please ignore the "On Fri, Dec 27, 2024" date, this seems to be an > artifact of me re-sending the message, from the list archive. The > original message was from January 28, 2024. >

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-27 Thread James Hunter
Reviving this thread, because I am thinking about something related -- please ignore the "On Fri, Dec 27, 2024" date, this seems to be an artifact of me re-sending the message, from the list archive. The original message was from January 28, 2024. On Fri, Dec 27, 2024 at 11:02 AM Tomas Vondra wro

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-03-15 Thread Alexander Lakhin
Hello Anton, 14.03.2024 23:36, Anton A. Melnikov wrote: On 13.03.2024 10:41, Anton A. Melnikov wrote: Here is a version updated for the current master. During patch updating i mistakenly added double counting of deallocatated blocks. That's why the tests in the patch tester failed. Fixed i

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-03-14 Thread Anton A. Melnikov
On 13.03.2024 10:41, Anton A. Melnikov wrote: Here is a version updated for the current master. During patch updating i mistakenly added double counting of deallocatated blocks. That's why the tests in the patch tester failed. Fixed it and squashed fix 0002 with 0001. Here is fixed version.

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-03-13 Thread Anton A. Melnikov
On 12.03.2024 16:30, Aleksander Alekseev wrote: Just wanted to let you know that v20231226 doesn't apply. The patch needs love from somebody interested in it. Thanks for pointing to this! Here is a version updated for the current master. With the best regards, -- Anton A. Melnikov Postgres P

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-03-12 Thread Aleksander Alekseev
Hi, > I took a closer look at this patch over the last couple days, and I did > a bunch of benchmarks to see how expensive the accounting is. The good > news is that I haven't observed any overhead - I did two simple tests, > that I think would be particularly sensitive to this: > > [...] Just wa

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-12-26 Thread Tomas Vondra
Hi, I wanted to take a look at the patch, and I noticed it's broken since 3d51cb5197 renamed a couple pgstat functions in August. I plan to maybe do some benchmarks etc. preferably on current master, so here's a version fixing that minor bitrot. As for the patch, I only skimmed through the thread

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-12-26 Thread Tomas Vondra
On 12/26/23 11:49, Anton A. Melnikov wrote: > Hello! > > Earlier in this thread, the pgbench results were published, where with a > strong memory limit of 100MB > a significant, about 10%, decrease in TPS was observed [1]. > > Using dedicated server with 12GB RAM and methodology described in [3],

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-10 Thread jian he
hi. +static void checkAllocations(); should be "static void checkAllocations(void);" ? PgStatShared_Memtrack there is a lock, but seems not initialized, and not used. Can you expand on it? So in view pg_stat_global_memory_tracking, column "total_memory_reserved" is a point of time, total memory

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-08 Thread Andres Freund
Hi, On 2023-11-07 15:55:48 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2023-11-06 13:02:50 -0500, Stephen Frost wrote: > > > > > The max_total_memory limit is checked whenever the global counters are > > > > > updated. There is no special error handling if a me

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-07 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2023-11-06 13:02:50 -0500, Stephen Frost wrote: > > > > The max_total_memory limit is checked whenever the global counters are > > > > updated. There is no special error handling if a memory allocation > > > > exceeds > > > > the global

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-07 Thread Andres Freund
Hi, On 2023-11-06 13:02:50 -0500, Stephen Frost wrote: > > > The max_total_memory limit is checked whenever the global counters are > > > updated. There is no special error handling if a memory allocation exceeds > > > the global limit. That allocation returns a NULL for malloc style > > > allocat

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2023-10-31 17:11:26 +, John Morris wrote: > > Postgres memory reservations come from multiple sources. > > > > * Malloc calls made by the Postgres memory allocators. > > * Static shared memory created by the postmaster at ser

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-03 Thread Andres Freund
Hi, On 2023-10-31 17:11:26 +, John Morris wrote: > Postgres memory reservations come from multiple sources. > > * Malloc calls made by the Postgres memory allocators. > * Static shared memory created by the postmaster at server startup, > * Dynamic shared memory created by the bac

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-23 Thread Andres Freund
Hi, On 2023-10-24 09:39:42 +0700, Andrei Lepikhov wrote: > On 20/10/2023 19:39, Stephen Frost wrote: > Greetings, > > * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: > > > The only issue I worry about is the uncertainty and clutter that can be > > > created by this feature. In the worst case,

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-23 Thread Andrei Lepikhov
On 20/10/2023 19:39, Stephen Frost wrote: Greetings, * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: The only issue I worry about is the uncertainty and clutter that can be created by this feature. In the worst case, when we have a complex error stack (including the extension's CATCH sectio

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-20 Thread Stephen Frost
Greetings, * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: > The only issue I worry about is the uncertainty and clutter that can be > created by this feature. In the worst case, when we have a complex error > stack (including the extension's CATCH sections, exceptions in stored > procedures,

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-19 Thread Andrei Lepikhov
On 20/10/2023 05:06, Stephen Frost wrote: Greetings, * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: On 19/10/2023 02:00, Stephen Frost wrote: * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: On 29/9/2023 09:52, Andrei Lepikhov wrote: On 22/5/2023 22:59, reid.thomp...@crunchydata.co

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-19 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2023-10-19 18:06:10 -0400, Stephen Frost wrote: > > Ignoring such would defeat much of the point of this effort- which is to > > get to a position where we can say with some confidence that we're not > > going to go over some limit that t

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-19 Thread Andres Freund
Hi, On 2023-10-19 18:06:10 -0400, Stephen Frost wrote: > Ignoring such would defeat much of the point of this effort- which is to > get to a position where we can say with some confidence that we're not > going to go over some limit that the user has set and therefore not > allow ourselves to end

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-19 Thread Stephen Frost
Greetings, * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: > On 19/10/2023 02:00, Stephen Frost wrote: > > * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: > > > On 29/9/2023 09:52, Andrei Lepikhov wrote: > > > > On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote: > > > > > Attach pa

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-18 Thread Andrei Lepikhov
On 19/10/2023 02:00, Stephen Frost wrote: Greetings, * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: On 29/9/2023 09:52, Andrei Lepikhov wrote: On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote: Attach patches updated to master. Pulled from patch 2 back to patch 1 a change that was

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-18 Thread Stephen Frost
Greetings, * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote: > On 29/9/2023 09:52, Andrei Lepikhov wrote: > > On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote: > > > Attach patches updated to master. > > > Pulled from patch 2 back to patch 1 a change that was also pertinent > > > to patc

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-10-03 Thread Andrei Lepikhov
On 29/9/2023 09:52, Andrei Lepikhov wrote: On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote: Attach patches updated to master. Pulled from patch 2 back to patch 1 a change that was also pertinent to patch 1. +1 to the idea, have doubts on the implementation. I have a question. I see th

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-09-28 Thread Andrei Lepikhov
On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote: Attach patches updated to master. Pulled from patch 2 back to patch 1 a change that was also pertinent to patch 1. +1 to the idea, have doubts on the implementation. I have a question. I see the feature triggers ERROR on the exceeding of

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-06-05 Thread reid . thompson
On Mon, 2023-05-22 at 08:42 -0400, reid.thomp...@crunchydata.com wrote: More followup to the above. > > I experimented on my system regarding > "The simple query select * from generate_series(0, 1000) shows roughly > 18.9 % degradation on my test server." > > My laptop: > 32GB ram > 11th G

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-05-22 Thread reid . thompson
On Mon, 2023-05-22 at 08:42 -0400, reid.thomp...@crunchydata.com wrote: > On Wed, 2023-05-17 at 23:07 -0400, reid.thomp...@crunchydata.com wrote: > > Thanks for the feedback. > > > > I'm plannning to look at this. > > > > Is your benchmark something that I could utilize? I.E. is it a set of > >

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-05-22 Thread reid . thompson
On Wed, 2023-05-17 at 23:07 -0400, reid.thomp...@crunchydata.com wrote: > Thanks for the feedback. > > I'm plannning to look at this. > > Is your benchmark something that I could utilize? I.E. is it a set of > scripts or a standard test from somewhere that I can duplicate? > > Thanks, > Reid >

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-05-17 Thread reid . thompson
On Wed, 2023-04-19 at 23:28 +, Arne Roland wrote: > > Thank you! I just tried our benchmark and got a performance > > degration > of around 28 %, which is way better than the last > > patch. > > > > The simple query select * from generate_series(0, 1000) shows > > > roughly 18.9 % degradat

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-04-19 Thread Arne Roland
Thank you! I just tried our benchmark and got a performance degration of around 28 %, which is way better than the last patch. The simple query select * from generate_series(0, 1000) shows roughly 18.9 % degradation on my test server. By raising initial_allocation_allowance and allocation_a

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-04-06 Thread reid . thompson
Updated patches attached. Rebased to current master. Added additional columns to pg_stat_global_memory_allocation to summarize backend allocations by type. Updated documentation. Corrected some issues noted in review by John Morris. Added code re EXEC_BACKEND for dev-max-memory branch. From 3451

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-03-24 Thread reid . thompson
Updated patches attached. pg-stat-activity-backend-memory-allocated DSM allocations created by a process and not destroyed prior to it's exit are considered l

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-03-02 Thread Andres Freund
On 2023-03-02 14:41:26 -0500, reid.thomp...@crunchydata.com wrote: > Patch has been rebased to master. Quite a few prior review comments seem to not have been addressed. There's not much point in posting new versions without that. I think there's zero chance 0002 can make it into 16. If 0001 is c

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-03-02 Thread reid . thompson
On Mon, 2023-02-13 at 16:26 -0800, Andres Freund wrote: > Hi, > > The tests recently started to fail: > > https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3867 > > I marked this as waiting on author. > > Greetings, > > Andres Freund Patch has been rebased to master.

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-02-13 Thread Andres Freund
Hi, On 2023-01-26 15:27:20 -0500, Reid Thompson wrote: > Yes, just a rebase. There is still work to be done per earlier in the > thread. The tests recently started to fail: https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3867 > I do want to follow up and note re pall

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-02-02 Thread Reid Thompson
Regarding the shared counter noted here, > What you could do is to have a single, imprecise, shared counter for the total > memory allocation, and have a backend-local "allowance". When the allowance is > used up, refill it from the shared counter (a single atomic op). Is there a preferred or sug

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-26 Thread Reid Thompson
On Mon, 2023-01-23 at 12:31 -0800, Andres Freund wrote: > Hi, > > I think it's basically still waiting on author, until the O(N) cost is gone > from the overflow limit check. > > Greetings, > > Andres Freund Yes, just a rebase. There is still work to be done per earlier in the thread. I do wan

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-23 Thread Andres Freund
Hi, On 2023-01-23 10:48:38 -0500, Reid Thompson wrote: > On Thu, 2023-01-19 at 16:50 +0530, vignesh C wrote: > > > > The patch does not apply on top of HEAD as in [1], please post a rebased > > patch: > > > > Regards, > > Vignesh > > rebased patch attached I think it's basically still waiting

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-23 Thread Reid Thompson
On Thu, 2023-01-19 at 16:50 +0530, vignesh C wrote: > > The patch does not apply on top of HEAD as in [1], please post a rebased > patch: > > Regards, > Vignesh rebased patch attached Thanks, Reid From b32a346d6e0e00c568e9a285ad15fc2703998c26 Mon Sep 17 00:00:00 2001 From: Reid Thompson Date

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-19 Thread vignesh C
On Fri, 6 Jan 2023 at 00:19, Reid Thompson wrote: > > On Tue, 2023-01-03 at 16:22 +0530, vignesh C wrote: > > > > The patch does not apply on top of HEAD as in [1], please post a > > rebased patch: > > ... > > Regards, > > Vignesh > > > > Attached is rebased patch, with some updates related t

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-13 Thread Andres Freund
Hi, On 2023-01-13 09:15:10 -0500, Reid Thompson wrote: > On Mon, 2023-01-09 at 18:31 -0800, Andres Freund wrote: > > > Dynamic shared memory allocations are included only in the value displayed > > > for the backend that created them, they are not included in the value for > > > backends that are

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-13 Thread Reid Thompson
On Mon, 2023-01-09 at 18:31 -0800, Andres Freund wrote: > Hi, > > On 2023-01-05 13:44:20 -0500, Reid Thompson wrote: > > This new field displays the current bytes of memory allocated to the > > backend process. It is updated as memory for the process is > > malloc'd/free'd. Memory allocated to ite

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-09 Thread Andres Freund
Hi, On 2023-01-05 13:44:20 -0500, Reid Thompson wrote: > From 0a6b152e0559a25033bd7d43eb0959432e0d Mon Sep 17 00:00:00 2001 > From: Reid Thompson > Date: Thu, 11 Aug 2022 12:01:25 -0400 > Subject: [PATCH 1/2] Add tracking of backend memory allocated to > pg_stat_activity > > This new field

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-05 Thread Reid Thompson
On Tue, 2023-01-03 at 16:22 +0530, vignesh C wrote: > > The patch does not apply on top of HEAD as in [1], please post a > rebased patch: > ... > Regards, > Vignesh > Attached is rebased patch, with some updates related to committed changes. Thanks, Reid -- Reid Thompson Senior Software En

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-03 Thread vignesh C
On Fri, 9 Dec 2022 at 20:41, Reid Thompson wrote: > > On Tue, 2022-12-06 at 10:32 -0800, Andres Freund wrote: > > Hi, > > > > On 2022-11-26 22:22:15 -0500, Reid Thompson wrote: > > > rebased/patched to current master && current pg-stat-activity- > > > backend-memory-allocated > > > > This version

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-12-09 Thread Reid Thompson
On Tue, 2022-12-06 at 10:32 -0800, Andres Freund wrote: > Hi, > > On 2022-11-26 22:22:15 -0500, Reid Thompson wrote: > > rebased/patched to current master && current pg-stat-activity- > > backend-memory-allocated > > This version fails to build with msvc, and builds with warnings on > other > pla

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-12-06 Thread Andres Freund
Hi, On 2022-11-26 22:22:15 -0500, Reid Thompson wrote: > rebased/patched to current master && current > pg-stat-activity-backend-memory-allocated This version fails to build with msvc, and builds with warnings on other platforms. https://cirrus-ci.com/build/5410696721072128 msvc: [20:26:51.286]

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-11-26 Thread Reid Thompson
On Thu, 2022-11-03 at 11:48 -0400, Reid Thompson wrote: > On Tue, 2022-10-25 at 11:49 -0400, Reid Thompson wrote: > > Rebased to current. Add a couple changes per conversation with D > Christensen (include units in field name, group field with > backend_xid > and backend_xmin fields in pg_stat_acti

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-11-03 Thread Reid Thompson
On Tue, 2022-10-25 at 11:49 -0400, Reid Thompson wrote: > Hi Arne, > > On Mon, 2022-10-24 at 15:27 +, Arne Roland wrote: > > Hello Reid, > > > > could you rebase the patch again? It doesn't apply currently > > (http://cfbot.cputube.org/patch_40_3867.log). Thanks! > > rebased patches attached

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-10-25 Thread Reid Thompson
Hi Arne, On Mon, 2022-10-24 at 15:27 +, Arne Roland wrote: > Hello Reid, > > could you rebase the patch again? It doesn't apply currently > (http://cfbot.cputube.org/patch_40_3867.log). Thanks! rebased patches attached. > You mention, that you want to prevent the compiler from getting > cut

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-10-24 Thread Arne Roland
Regards Arne From: Reid Thompson Sent: Thursday, September 15, 2022 4:58:19 PM To: Ibrar Ahmed; pgsql-hackers@lists.postgresql.org Cc: reid.thomp...@crunchydata.com; Justin Pryzby Subject: Re: Add the ability to limit the amount of memory that can be allocated t

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-15 Thread Reid Thompson
On Thu, 2022-09-15 at 12:07 +0400, Ibrar Ahmed wrote: > > The patch does not apply; please rebase the patch. > > patching file src/backend/utils/misc/guc.c > Hunk #1 FAILED at 3664. > 1 out of 1 hunk FAILED -- saving rejects to file > src/backend/utils/misc/guc.c.rej  > > patching file src/backe

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-15 Thread Ibrar Ahmed
On Mon, Sep 12, 2022 at 8:30 PM Reid Thompson wrote: > On Fri, 2022-09-09 at 12:14 -0500, Justin Pryzby wrote: > > On Sat, Sep 03, 2022 at 11:40:03PM -0400, Reid Thompson wrote: > > > > > + 0, 0, INT_MAX, > > > > > + NULL, NULL, NULL > > > > I think this needs a maximu

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-12 Thread Reid Thompson
On Fri, 2022-09-09 at 12:14 -0500, Justin Pryzby wrote: > On Sat, Sep 03, 2022 at 11:40:03PM -0400, Reid Thompson wrote: > > > > +   0, 0, INT_MAX, > > > > +   NULL, NULL, NULL > > > I think this needs a maximum like INT_MAX/1024/1024 > > > > Is this noting that we'd set a

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-09 Thread Justin Pryzby
On Sat, Sep 03, 2022 at 11:40:03PM -0400, Reid Thompson wrote: > > > +   0, 0, INT_MAX, > > > +   NULL, NULL, NULL > > I think this needs a maximum like INT_MAX/1024/1024 > > Is this noting that we'd set a ceiling of 2048MB? The reason is that you're later multiplying it b

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-09 Thread Stephen Frost
Greetings, * David Rowley (dgrowle...@gmail.com) wrote: > On Thu, 1 Sept 2022 at 04:52, Reid Thompson > wrote: > > Add the ability to limit the amount of memory that can be allocated to > > backends. > > Are you aware that relcache entries are stored in backend local memory > and that once we've

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-06 Thread Reid Thompson
On Fri, 2022-09-02 at 09:30 +0200, Drouvot, Bertrand wrote: > Hi, > > I'm not sure we are choosing the right victims here (aka the ones > that are doing the request that will push the total over the limit). > > Imagine an extreme case where a single backend consumes say 99% of > the limit, shouldn

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-06 Thread Reid Thompson
On Thu, 2022-09-01 at 11:48 +0900, Kyotaro Horiguchi wrote: > > > > The patch seems to limit both of memory-context allocations and DSM > > allocations happen on a specific process by the same budget. In the > > fist place I don't think it's sensible to cap the amount of DSM > > allocations by per

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-03 Thread Reid Thompson
On Wed, 2022-08-31 at 12:34 -0500, Justin Pryzby wrote: > You should name the patches with different prefixes, like > 001,002,003  Otherwise, cfbot may try to apply them in the wrong > order. > git format-patch is the usual tool for that. Thanks for the pointer. My experience with git in the past

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-09-02 Thread David Rowley
On Thu, 1 Sept 2022 at 04:52, Reid Thompson wrote: > Add the ability to limit the amount of memory that can be allocated to > backends. Are you aware that relcache entries are stored in backend local memory and that once we've added a relcache entry for a relation that we have no current code whi

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-08-31 Thread Kyotaro Horiguchi
At Wed, 31 Aug 2022 12:50:19 -0400, Reid Thompson wrote in > Hi Hackers, > > Add the ability to limit the amount of memory that can be allocated to > backends. The patch seems to limit both of memory-context allocations and DSM allocations happen on a specific process by the same budget. In th

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2022-08-31 Thread Justin Pryzby
On Wed, Aug 31, 2022 at 12:50:19PM -0400, Reid Thompson wrote: > Hi Hackers, > > Add the ability to limit the amount of memory that can be allocated to > backends. > > This builds on the work that adds backend memory allocated to > pg_stat_activity > https://www.postgresql.org/message-id/67bb5c15