Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-27 Thread Craig Ringer
On 07/27/2012 07:52 PM, luben karavelov wrote: It's good that the bug is not in HEAD. I was testing on 9.1.4. So was I, and while I thought I'd reproduced it I now suspect I was just seeing shared_buffers touching. Are you able to produce a self-contained SQL test that demonstrates the leak?

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-27 Thread luben karavelov
On Jul 27, 2012, at 8:47 AM, Tom Lane wrote: > Craig Ringer writes: >> OK, it's certainly leaking, but not in the same drastic way I was able >> to reproduce manually a couple of times earlier. Self-contained test >> case attached. > > Using HEAD with stock parameters, I don't see any signifi

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Craig Ringer
On 07/27/2012 10:31 AM, Craig Ringer wrote: Ouch. Sure looks like a leak to me, yeah. ... but it turns out I'm not thinking very straight. I forgot to check the size of your `shared_buffers' or `work_mem' and forgot to get you to report `free -m' output after each run to measure _real_ memory

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Craig Ringer
On 07/27/2012 01:47 PM, Tom Lane wrote: Craig Ringer writes: OK, it's certainly leaking, but not in the same drastic way I was able to reproduce manually a couple of times earlier. Self-contained test case attached. Using HEAD with stock parameters, I don't see any significant change in alloca

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Tom Lane
Craig Ringer writes: > OK, it's certainly leaking, but not in the same drastic way I was able > to reproduce manually a couple of times earlier. Self-contained test > case attached. Using HEAD with stock parameters, I don't see any significant change in allocated address space (VSZ): it sits ri

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Craig Ringer
OK, it's certainly leaking, but not in the same drastic way I was able to reproduce manually a couple of times earlier. Self-contained test case attached. $ bash hstore-leak-demo.sh NOTICE: extension "hstore" already exists, skipping CREATE EXTENSION DROP TABLE CREATE TABLE CREATE TABLE INSERT

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Craig Ringer
On 07/27/2012 10:30 AM, Craig Ringer wrote: Hi all Here's a fully self contained, automated test case that demonstrates the leak. Gah. Except it doesn't now, as shown by the text I pasted. WTF? I was *definitely* seeing this on my system. What's changed? Will follow up. -- Sent via pgsql-b

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Craig Ringer
On 07/26/2012 09:55 PM, luben karavelov wrote: Ok, I will do the procedure again with taking notes on each step. Thankyou for taking the time to do this in detail. First, here are the results of the queries you asked: count | to_char -+--- 1257262 | 2.26 pg_s

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Craig Ringer
Hi all Here's a fully self contained, automated test case that demonstrates the leak. Example output on my system, pasted as quote to stop Thunderbird mangling it: $ ./hstore-leak-demo.sh NOTICE: extension "hstore" already exists, skipping CREATE EXTENSION DROP TABLE CREATE TABLE CREATE T

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Craig Ringer
Woah. Your email client did something insane, and I cannot read your message. See below: On 07/26/2012 09:37 PM, karave...@mail.bg wrote: - Craig Ringer (ring...@ringerc.id.au), на 26.07.2012 в 11:17 - On 07/26/2012 09:32 AM, karave...@mail.bg wrote: >> Finally I have managed to migra

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread luben karavelov
On Jul 26, 2012, at 11:17 AM, Craig Ringer wrote: > On 07/26/2012 09:32 AM, karave...@mail.bg wrote: >> Finally I have managed to migrate it in batches of 100-200k user ids and >> disconnecting after each query in order to free the backend and leaked >> memory. > If you do it in batches, but you

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread karavelov
- Craig Ringer (ring...@ringerc.id.au), на 26.07.2012 в 11:17 - > On 07/26/2012 09:32 AM, karave...@mail.bg wrote: >> Finally I have managed to migrate it in batches of 100-200k user ids and >> disconnecting after each query in order to free the backend and leaked >> memory. > If you do it

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-26 Thread Craig Ringer
On 07/26/2012 09:32 AM, karave...@mail.bg wrote: Finally I have managed to migrate it in batches of 100-200k user ids and disconnecting after each query in order to free the backend and leaked memory. If you do it in batches, but you do NOT disconnect and reconnect, does the backend continue to

Re: [BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-25 Thread Tom Lane
karave...@mail.bg writes: > The query I have tried to use is: > INSERT INTO preferences > SELECT user_id,hstore(array_agg(name), array_agg(value)) > FROM old_prefs > GROUP BY user_id; > But the postgres backend consumed all the available memory (6G free + 4G > swap) and finally was killed by t

[BUGS] BUG #6763: Severe memory leak with arrays and hstore

2012-07-25 Thread karavelov
The following bug has been logged on the website: Bug reference: 6763 Logged by: Luben Karavelov Email address: karave...@mail.bg PostgreSQL version: 9.1.4 Operating system: Debian Linux Description: I was trying to migrate a big ER table (user preferences) into a new