Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-08-10 Thread Matt Magoffin
> These numbers don't even have any demonstrable connection to Postgres, > let alone to an xpath-related memory leak. You're going to need to come > up with a concrete test case if you want anyone to investigate. > > regards, tom lane As I said in the start of this thread, t

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-08-10 Thread Tom Lane
"Matt Magoffin" <[EMAIL PROTECTED]> writes: > I'm following up on this thread from a month ago on PG 8.3 memory use. I'm > afraid even after updating to 8.3.3 + this patch, I still see the same > overall memory trend. You can see what I'm looking at here with a couple > of memory graphs. These num

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-08-10 Thread Matt Magoffin
>> Gregory Stark <[EMAIL PROTECTED]> writes: >>> That's just a special case of what would be expected to happen with >>> memory >>> allocation anyways though. Few allocators return memory to the OS >>> anyways. >> >> Well, that does happen on Linux for instance. Since Matt knew in his >> original

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-06 Thread Matt Magoffin
> Gregory Stark <[EMAIL PROTECTED]> writes: >> That's just a special case of what would be expected to happen with >> memory >> allocation anyways though. Few allocators return memory to the OS >> anyways. > > Well, that does happen on Linux for instance. Since Matt knew in his > original report t

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-03 Thread Tom Lane
Gregory Stark <[EMAIL PROTECTED]> writes: > That's just a special case of what would be expected to happen with memory > allocation anyways though. Few allocators return memory to the OS anyways. Well, that does happen on Linux for instance. Since Matt knew in his original report that the xpath l

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-03 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > Well, you tell me --- *you* reported a behavior that isn't obviously > explained by the bug we found. In case it wasn't clear, the bug found was a intra-transaction memory leak. When the transaction ended the memory would be reclaimed. That doesn't seem to

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Matt Magoffin
> Probably the right thing for you to do now is just to install the known > fix, and keep an eye on your server for awhile to see if you still see > any indication of the long-term leak behavior. Certainly, that is my plan. Once I can get the patch rolled out to these systems, I should be able to

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Tom Lane
"Matt Magoffin" <[EMAIL PROTECTED]> writes: >> So there may be a second issue remaining to be found. Can you put >> together a test case for the long-term small leak? > Hmm, I'm not sure what else to add to this test case. This test case was a > good example of what our database is doing with xpa

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Matt Magoffin
> This part seems to match the bug though --- the leak is approximately > the same size as all the text returned by xpath() within the current > transaction. > > So there may be a second issue remaining to be found. Can you put > together a test case for the long-term small leak? > >

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Tom Lane
"Matt Magoffin" <[EMAIL PROTECTED]> writes: > Later, I added a large set of plpgsql trigger functions that operate on > that new xml column data, using the xpath() function to extract bits of > XML and populate them into normal tables. The server has been running in > this fashion for many months n

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Tom Lane
"Matt Magoffin" <[EMAIL PROTECTED]> writes: >>> I think this should fix it. >>> Kris Jurka Confirmed, that makes it go away nicely here: LibxmlContext: 57344 total in 3 blocks; 55720 free (202 chunks); 1624 used >> It looks like xml.c source has changed considerably since 8.3 (looking at >> re

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Matt Magoffin
>> I think this should fix it. >> >> Kris Jurka > > It looks like xml.c source has changed considerably since 8.3 (looking at > revision 1.68.2.2 from the 8.3.3. release). Do you know where/if this > patch would apply to the 8.3 branch? I diff'ed 1.74 and 1.68.2.2, and I'm guessing this new line c

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Tom Lane
"Matt Magoffin" <[EMAIL PROTECTED]> writes: > It looks like xml.c source has changed considerably since 8.3 No, hardly at all actually, but this patch happens to be right next door to one of the lines that did change. cstring_to_text() replaces some grottier stuff that used to be used for the sam

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Matt Magoffin
>> I'm able to duplicate the memory leak in this function with the current >> Fedora 8 libxml2 (2.6.32). The leak is definitely inside libxml2 >> itself, because the bloat shows up here: >> > > I think this should fix it. > > Kris Jurka It looks like xml.c source has changed considerably since 8.

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Kris Jurka
On Wed, 2 Jul 2008, Tom Lane wrote: "Matt Magoffin" <[EMAIL PROTECTED]> writes: Below is a test case that simulates the use of xpath() within a plpgsql function in my application. I'm able to duplicate the memory leak in this function with the current Fedora 8 libxml2 (2.6.32). The leak is

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Matt Magoffin
> I looked through the libxml2 sources a little bit but couldn't > immediately find the problem. I'm fairly confident though that > this could be reproduced outside Postgres, by replicating the sequence > of libxml2 calls we make in xpath(). The next step should probably be > to build a reproduce

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Tom Lane
"Matt Magoffin" <[EMAIL PROTECTED]> writes: > Below is a test case that simulates the use of xpath() within a plpgsql > function in my application. I'm able to duplicate the memory leak in this function with the current Fedora 8 libxml2 (2.6.32). The leak is definitely inside libxml2 itself, beca

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-02 Thread Matt Magoffin
>> OK, I'll try to come up with something. Do you have a recommended way of >> capturing the amount memory being used by Postgres related to this? I >> was >> thinking I would have a plpgsql function that loops a large number of >> times, calling a few xpath() calls, > > Yeah, that's what I'd try f

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-01 Thread Tom Lane
"Matt Magoffin" <[EMAIL PROTECTED]> writes: >> Ugh. Sounds like "small memory leak inside libxml2" --- probably not >> going to be easy to find. Can you put together a self-contained test >> case? > OK, I'll try to come up with something. Do you have a recommended way of > capturing the amount m

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-01 Thread Matt Magoffin
> Ugh. Sounds like "small memory leak inside libxml2" --- probably not > going to be easy to find. Can you put together a self-contained test > case? OK, I'll try to come up with something. Do you have a recommended way of capturing the amount memory being used by Postgres related to this? I was

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-07-01 Thread Tom Lane
"Matt Magoffin" <[EMAIL PROTECTED]> writes: > Later, I added a large set of plpgsql trigger functions that operate on > that new xml column data, using the xpath() function to extract bits of > XML and populate them into normal tables. The server has been running in > this fashion for many months n