Re: Backend memory dump analysis

2018-04-06 Thread Peter Eisentraut
On 3/27/18 19:55, Tom Lane wrote: > Seems reasonable, although I think if you were to delay setting the > name till the end of that function, you could point to portal->name > and avoid the extra pstrdup. Maybe that's useless microoptimization. done >> The term CopySetIdentifier has confused me

Re: Backend memory dump analysis

2018-03-27 Thread Tom Lane
Peter Eisentraut writes: > How about this one as well: > portal->portalContext = AllocSetContextCreate(TopPortalContext, > "PortalContext", > ALLOCSET_SMALL_SIZES); > + MemoryContextCopySetId

Re: Backend memory dump analysis

2018-03-27 Thread Peter Eisentraut
On 3/27/18 12:47, Tom Lane wrote: > Here's an updated patch that adjusts the output format per discussion: > > - context identifier at the end of the line, so it's easier to see the > numbers > > - identifiers truncated at 100 bytes, control characters replaced by > spaces > > Also, I hacked

Re: Backend memory dump analysis

2018-03-27 Thread Tom Lane
Alvaro Herrera writes: > My only gripe is the pattern where the identifier needs to be > re-installed when resetting the context. I don't think we need to hold > push for that reason alone, but I bet we'll be revisiting that. Yeah, that's slightly annoying; if I'd found more than one case of tha

Re: Backend memory dump analysis

2018-03-27 Thread Alvaro Herrera
Great stuff. My only gripe is the pattern where the identifier needs to be re-installed when resetting the context. I don't think we need to hold push for that reason alone, but I bet we'll be revisiting that. I suppose this infrastructure can be used to implement the idea in https://www.postgre

Re: Backend memory dump analysis

2018-03-26 Thread Tom Lane
Vladimir Sitnikov writes: > It takes just a single pass to compute "total" (and it takes no memory), so > it would be much better if "TopMemoryContext: ..." was replaced with > "Total memory used by all contexts is XXX bytes" > Current TopMemoryContext row is extremely misleading. This may or may

Re: Backend memory dump analysis

2018-03-25 Thread Craig Ringer
On 24 March 2018 at 03:01, Andres Freund wrote: > Hi, > > On 2018-03-23 14:33:25 -0400, Tom Lane wrote: > > func_cxt = AllocSetContextCreate(TopMemoryContext, > > "PL/pgSQL function context", > > ALLOCSET_DEFAULT_SIZES)

Re: Backend memory dump analysis

2018-03-25 Thread Craig Ringer
On 24 March 2018 at 02:33, Tom Lane wrote: > Andres Freund writes: > > On 2018-03-23 18:05:38 +, Vladimir Sitnikov wrote: > >> For instance, I assume statament cache is stored in some sort of a hash > >> table, so there should be a way to enumerate it in a programmatic way. > Of > >> course

Re: Backend memory dump analysis

2018-03-25 Thread Vladimir Sitnikov
Tom>One concrete objection to the above is it'd obscure hierarchical relationships in the context tree, What is the problem with relationships? Context names are aligned as well provided 8192 is justified to 6-7-8-9 (you pick) characters. Tom>But given the lack of previous complaints 1) Here it

Re: Backend memory dump analysis

2018-03-25 Thread Tom Lane
Vladimir Sitnikov writes: > Tom>Well, as I said, you can do anything you want now in an extension. > That is true. However it basically means "everybody who cares to > troubleshoot the memory use of a production system should install an > extension". If you're interested in capturing memory usag

Re: Backend memory dump analysis

2018-03-25 Thread Vladimir Sitnikov
Tom>Well, as I said, you can do anything you want now in an extension. That is true. However it basically means "everybody who cares to troubleshoot the memory use of a production system should install an extension". Should https://wiki.postgresql.org/wiki/Developer_FAQ#Examining_backend_memory_us

Re: Backend memory dump analysis

2018-03-25 Thread Tom Lane
Vladimir Sitnikov writes: >> While I didn't do anything about it here, I think it'd likely be a >> good idea for MemoryContextStats printout to truncate the context ID >> strings at 100 characters or so > It would be great if there was an option to show full sql. Well, as I said, you can do anyt

Re: Backend memory dump analysis

2018-03-25 Thread Vladimir Sitnikov
It looks much better. >While I didn't do anything about it here, I think it'd likely be a >good idea for MemoryContextStats printout to truncate the context ID >strings at 100 characters or so It would be great if there was an option to show full sql. For instance, current statistics is not sorte

Re: Backend memory dump analysis

2018-03-23 Thread Teodor Sigaev
Hi! Some help you could get from https://github.com/postgrespro/memstat Vladimir Sitnikov wrote: Hi, I investigate an out of memory-related case for PostgreSQL 9.6.5, and it looks like MemoryContextStatsDetail + gdb are the only friends there. -- Teodor Sigaev E-mail: t

Re: Backend memory dump analysis

2018-03-23 Thread Tom Lane
Andres Freund writes: > On 2018-03-23 15:12:43 -0400, Tom Lane wrote: >> Well, in the cases I'm thinking of at the moment, there's no handy Node >> to point at, just module-private structs like PLpgSQL_function. > Well, the cases Vladimir were concerned about seem less clear > though. It'd be nic

Re: Backend memory dump analysis

2018-03-23 Thread Andres Freund
On 2018-03-23 15:12:43 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-03-23 14:33:25 -0400, Tom Lane wrote: > >> + MemoryContextSetIdentifier(func_cxt, function->fn_signature); > >> > >> This would cost an extra char * field in struct MemoryContextData, > >> which is slightly annoyi

Re: Backend memory dump analysis

2018-03-23 Thread Tom Lane
Andres Freund writes: > On 2018-03-23 14:33:25 -0400, Tom Lane wrote: >> + MemoryContextSetIdentifier(func_cxt, function->fn_signature); >> >> This would cost an extra char * field in struct MemoryContextData, >> which is slightly annoying but it doesn't exactly seem like a killer. >> Then the

Re: Backend memory dump analysis

2018-03-23 Thread Andres Freund
Hi, On 2018-03-23 14:33:25 -0400, Tom Lane wrote: > func_cxt = AllocSetContextCreate(TopMemoryContext, > "PL/pgSQL function context", > ALLOCSET_DEFAULT_SIZES); > plpgsql_compile_tmp_cxt = MemoryContextSwitchTo(func_

Re: Backend memory dump analysis

2018-03-23 Thread Tom Lane
Andres Freund writes: > On 2018-03-23 18:05:38 +, Vladimir Sitnikov wrote: >> For instance, I assume statament cache is stored in some sort of a hash >> table, so there should be a way to enumerate it in a programmatic way. Of >> course it would take time, however I do not think it creates cpu

Re: Backend memory dump analysis

2018-03-23 Thread Andres Freund
On 2018-03-23 18:05:38 +, Vladimir Sitnikov wrote: > Andres>The overhead required for it (in cycles, in higher memory usage due > to > additional bookeeping > > Does that mean the memory contexts are unparseable? (there's not enough > information to enumerate contents) You can enumerate them

Re: Backend memory dump analysis

2018-03-23 Thread Vladimir Sitnikov
Andres>The overhead required for it (in cycles, in higher memory usage due to additional bookeeping Does that mean the memory contexts are unparseable? (there's not enough information to enumerate contents) What if memory dump is produced by walking the C structures? For instance, I assume statam

Re: Backend memory dump analysis

2018-03-23 Thread Andres Freund
Hi, On 2018-03-23 16:18:52 +, Vladimir Sitnikov wrote: > Hi, > > I investigate an out of memory-related case for PostgreSQL 9.6.5, and it > looks like MemoryContextStatsDetail + gdb are the only friends there. > > MemoryContextStatsDetail does print some info, however it is rarely > possible

Backend memory dump analysis

2018-03-23 Thread Vladimir Sitnikov
Hi, I investigate an out of memory-related case for PostgreSQL 9.6.5, and it looks like MemoryContextStatsDetail + gdb are the only friends there. MemoryContextStatsDetail does print some info, however it is rarely possible to associate the used memory with business cases. For insance: CachedP