Re: [HACKERS] In-place upgrade: catalog side

2008-12-03 Thread Greg Smith
On Wed, 3 Dec 2008, Zdenek Kotala wrote: It works fine for 8.3->8.4 too, but I'm working on cleanup and fixing bugs. I hope that I will send updated version to community today. That would be great. It didn't feel like you were quite done with it yet. I'll be glad to help test it out, just di

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Heikki Linnakangas
Alvaro Herrera wrote: Gregory Stark escribió: "Joshua D. Drake" <[EMAIL PROTECTED]> writes: I don't think at any time I have said to my self, I am going to set this parameter low so I don't fill up my disk. If I am saying that to myself I have either greatly underestimated the hardware for th

Re: [HACKERS] In-place upgrade: catalog side

2008-12-03 Thread Zdenek Kotala
Heikki Linnakangas napsal(a): Zdenek Kotala wrote: Greg Smith napsal(a): -There are 10 TODO items listed for the pg_migrator project, most or all of which look like should be squashed before this is really complete. Any chance somebody (Korry?) has an improved version of this floating around

Re: [HACKERS] Sync Rep: First Thoughts on Code

2008-12-03 Thread Fujii Masao
Hi, On Wed, Dec 3, 2008 at 11:33 PM, Simon Riggs <[EMAIL PROTECTED]> wrote: > I'm patient, I know it takes time. Happy to spend hours on the review, > but I want to do that knowing I agree with the higher level features and > architecture first. I wrote the features and restrictions of Synch Rep.

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Greg Smith
On Wed, 3 Dec 2008, Mark Wong wrote: http://207.173.203.223/~markwkm/pgsql/default_statistics_target/q2.png http://207.173.203.223/~markwkm/pgsql/default_statistics_target/q9.png http://207.173.203.223/~markwkm/pgsql/default_statistics_target/q17.png http://207.173.203.223/~markwkm/pgsql/default

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
>> I think the tests you could consider next is to graph the target going from >> 10 to 100 in steps of 10 just for those 5 queries. If it gradually >> degrades, that's interesting but hard to nail down. But if there's a sharp >> transition, getting an explain plan for the two sides of that shoul

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Greg Smith
On Thu, 4 Dec 2008, Gregory Stark wrote: My point was more that you could have a data warehouse on a non-dedicated machine, you could have a web server on a non-dedicated machine, or you could have a mixed server on a non-dedicated machine. I should just finish the documentation, where there

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
> If we do though, it shouldn't default one way and then get randomly flipped by > a tool that has the same information to make its decision on. What I'm saying > is that "mixed" is the same information that initdb had about the workload. +1. > If we do change this then I wonder if we need the pa

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Mark Wong
On Mon, Dec 1, 2008 at 9:32 PM, Greg Smith <[EMAIL PROTECTED]> wrote: > On Mon, 1 Dec 2008, Mark Wong wrote: > >> So then I attempted to see if there might have been difference between the >> executing time of each individual query with the above parameters. The >> queries that don't seem to be eff

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
Greg Smith <[EMAIL PROTECTED]> writes: > On Thu, 4 Dec 2008, Gregory Stark wrote: > >> Greg Smith <[EMAIL PROTECTED]> writes: >> >>> Is it worse to suffer from additional query overhead if you're sloppy with >>> the tuning tool, or to discover addition partitions didn't work as you >>> expected? >

Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1268)

2008-12-03 Thread KaiGai Kohei
Bruce Momjian wrote: KaiGai Kohei wrote: I updated the patch set of SE-PostgreSQL (revision 1268). [1/6] http://sepgsql.googlecode.com/files/sepostgresql-sepgsql-8.4devel-3-r1268.patch [2/6] http://sepgsql.googlecode.com/files/sepostgresql-pg_dump-8.4devel-3-r1268.patch [3/6] http://sepgsql.

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Greg Smith
On Thu, 4 Dec 2008, Gregory Stark wrote: Greg Smith <[EMAIL PROTECTED]> writes: Is it worse to suffer from additional query overhead if you're sloppy with the tuning tool, or to discover addition partitions didn't work as you expected? Surely that's the same question we faced when deciding w

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
Greg Smith <[EMAIL PROTECTED]> writes: > On Thu, 4 Dec 2008, Gregory Stark wrote: > >> What I'm suggesting is that you shouldn't have to special case this. That you >> should expect whatever formulas you're using to produce the same values as >> initdb if they were run on the same machine initdb

Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1268)

2008-12-03 Thread Bruce Momjian
KaiGai Kohei wrote: > I updated the patch set of SE-PostgreSQL (revision 1268). > > [1/6] > http://sepgsql.googlecode.com/files/sepostgresql-sepgsql-8.4devel-3-r1268.patch > [2/6] > http://sepgsql.googlecode.com/files/sepostgresql-pg_dump-8.4devel-3-r1268.patch > [3/6] > http://sepgsql.googleco

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
> What fun. I'm beginning to remember why nobody has ever managed to deliver > a community tool that helps with this configuration task before. I have to say I really like this tool. It may not be perfect but it's a lot easier than trying to do this analysis from scratch. And we are really only

Re: [HACKERS] Re: pgsql: Properly unregister OpenSSL callbacks when libpq is done with

2008-12-03 Thread Bruce Momjian
Kris Jurka wrote: > Magnus Hagander wrote: > > Log Message: > > --- > > Properly unregister OpenSSL callbacks when libpq is done with > > it's connection. This is required for applications that unload > > the libpq library (such as PHP) in which case we'd otherwise > > have pointers to thes

[HACKERS] Re: pgsql: Properly unregister OpenSSL callbacks when libpq is done with

2008-12-03 Thread Kris Jurka
Magnus Hagander wrote: Log Message: --- Properly unregister OpenSSL callbacks when libpq is done with it's connection. This is required for applications that unload the libpq library (such as PHP) in which case we'd otherwise have pointers to these functions when they no longer exist. B

Re: [HACKERS] V2 of PITR performance improvement for 8.4

2008-12-03 Thread Koichi Suzuki
Agreed. I borrowed WAL parsing code from XLogdump and I think WAL parsing should be another candidate. 2008/12/3 Fujii Masao <[EMAIL PROTECTED]>: > Hi, > > On Thu, Nov 27, 2008 at 9:04 PM, Koichi Suzuki <[EMAIL PROTECTED]> wrote: >> Please find enclosed a revised version of pg_readahead and a pat

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
> The idea of the mixed mode is that you want to reduce the odds someone will > get a massively wrong configuration if they're not paying attention. Is it > worse to suffer from additional query overhead if you're sloppy with the > tuning tool, or to discover addition partitions didn't work as you

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Greg Smith
On Thu, 4 Dec 2008, Gregory Stark wrote: Right now, my program doesn't fiddle with any memory settings if you've got less than 256MB of RAM. What I'm suggesting is that you shouldn't have to special case this. That you should expect whatever formulas you're using to produce the same values as

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Joshua D. Drake
On Wed, 2008-12-03 at 22:17 -0300, Alvaro Herrera wrote: > Gregory Stark escribió: > > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > > > I don't think at any time I have said to my self, I am going to set this > > > parameter low so I don't fill up my disk. If I am saying that to myself > > >

Re: [HACKERS] Re: [COMMITTERS] pgsql: Explicitly bind gettext() to the UTF8 locale when in use.

2008-12-03 Thread Hiroshi Inoue
Magnus Hagander wrote: Hiroshi Inoue wrote: I think the thing us that as long as the encodings are compatible (latin1 with different names for example) it worked fine. In any case I think the problem is that gettext is looking at a setting that is not what we are looking at. Particularly wi

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > On Thu, 2008-12-04 at 00:11 +, Gregory Stark wrote: >> "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > >> I >> started to do this for you last week but got side-tracked. Do you have any >> time for this? > > I can do it if you have a script. > >

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Alvaro Herrera
Gregory Stark escribió: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > I don't think at any time I have said to my self, I am going to set this > > parameter low so I don't fill up my disk. If I am saying that to myself > > I have either greatly underestimated the hardware for the task. Consi

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > On Thu, 2008-12-04 at 00:11 +, Gregory Stark wrote: >> "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > >> I >> started to do this for you last week but got side-tracked. Do you have any >> time for this? > > I can do it if you have a script. W

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
Greg Smith <[EMAIL PROTECTED]> writes: > Is it worse to suffer from additional query overhead if you're sloppy with > the tuning tool, or to discover addition partitions didn't work as you > expected? Surely that's the same question we faced when deciding what the Postgres default should be? Th

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Greg Smith
On Wed, 3 Dec 2008, Robert Haas wrote: Then I tried "-T web" and got what seemed like a more reasonable set of values. But I wasn't sure I needed that many connections, so I added "-c 150" to see how much difference that made. Kaboom! That and the import errors fixed in the version attached

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
Greg Smith <[EMAIL PROTECTED]> writes: > On Wed, 3 Dec 2008, Gregory Stark wrote: > >> It sure seems strange to me to have initdb which presumably is targeting a >> "mixed" system -- where it doesn't know for sure what workload will be run -- >> produce a different set of values than the tuner on

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Greg Smith
On Wed, 3 Dec 2008, Guillaume Smet wrote: - it would be really nice to make it work with Python 2.4 as RHEL 5 is a Python 2.4 thing and it is a very widespread platform out there, The 2.5 stuff is only required in order to detect memory on Windows. My primary box is RHEL5 and runs 2.4, it wo

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Greg Smith
On Wed, 3 Dec 2008, Robert Haas wrote: I'm not sure if you've thought about this, but there is also a difference between max_connections and maximum LIKELY connections. It's actually an implicit assumption of the model Josh threw out if you stare at the numbers. The settings for work_mem are

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Joshua D. Drake
On Thu, 2008-12-04 at 00:11 +, Gregory Stark wrote: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > I > started to do this for you last week but got side-tracked. Do you have any > time for this? I can do it if you have a script. > So how big should a minimum postgres install be not inclu

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
"Kevin Grittner" <[EMAIL PROTECTED]> writes: > One more data point to try to help. > > While the jump from a default_statistics_target from 10 to 1000 > resulted in a plan time increase for a common query from 50 ms to 310 > ms, at a target of 50 the plan time was 53 ms. That sounds like it w

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Actually there are years worth of evidence in these archives. Not that > the 50 is the right number but that the current settings are definitely > wrong and that higher ones are needed. That people generally start > around 100 and go from there, exce

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Greg Smith
On Wed, 3 Dec 2008, Gregory Stark wrote: It sure seems strange to me to have initdb which presumably is targeting a "mixed" system -- where it doesn't know for sure what workload will be run -- produce a different set of values than the tuner on the same machine. It's been a long time since th

Re: [HACKERS] In-place upgrade: catalog side

2008-12-03 Thread Bruce Momjian
Zdenek Kotala wrote: > If you compare with pg_migrator, there is better handling of locale and I > think > vacuum freeze is used correctly. Also shuffling with tablespaces is little > bit > different (it should avoid to move data outside of mountpoint). But in > principal > the idea is same.

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Kevin Grittner
>>> "Robert Haas" <[EMAIL PROTECTED]> wrote: > On Wed, Dec 3, 2008 at 4:41 PM, Joshua D. Drake <[EMAIL PROTECTED]> wrote: >> If you are concerned about the analyze time between 10, 50 and 150, I >> would suggest that you are concerned about the wrong things. Remember > > I can't rule that out. W

Re: [HACKERS] cvs head initdb hangs on unixware

2008-12-03 Thread Bruce Momjian
Andrew Dunstan wrote: > > > [EMAIL PROTECTED] wrote: > >> > >> Looking at fsm_rebuild_page, I wonder if the compiler is treating > >> "int" as an unsigned integer? That would cause an infinite loop. > >> > >> > > No, a simple printf of nodeno shows it starting at 4096 all the way > > down to 0

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Joshua D. Drake
On Wed, 2008-12-03 at 17:33 -0500, Robert Haas wrote: > On Wed, Dec 3, 2008 at 4:41 PM, Joshua D. Drake <[EMAIL PROTECTED]> wrote: > > If you are concerned about the analyze time between 10, 50 and 150, I > > would suggest that you are concerned about the wrong things. Remember > > I can't rule th

Re: [HACKERS] Visibility map, partial vacuums

2008-12-03 Thread Gregory Stark
Gregory Stark <[EMAIL PROTECTED]> writes: > Heikki Linnakangas <[EMAIL PROTECTED]> writes: > >> Gregory Stark wrote: >>> 1) Raise autovacuum_max_freeze_age to 400M or 800M. Having it at 200M just >>>means unnecessary full table vacuums long before they accomplish >>> anything. >> >> It allows

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
On Wed, Dec 3, 2008 at 4:41 PM, Joshua D. Drake <[EMAIL PROTECTED]> wrote: > If you are concerned about the analyze time between 10, 50 and 150, I > would suggest that you are concerned about the wrong things. Remember I can't rule that out. What things do you think I should be concerned about?

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > It also seems unlikely that you would hit 256MB of checkpoint segments > on a 100MB database before checkpoint_timeout and if you did, you > certainly did need them. > > Remember postgresql only creates the segments when it needs them. Should we ch

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Joshua D. Drake
On Wed, 2008-12-03 at 16:37 -0500, Robert Haas wrote: > > I can see an argument about constraint_exclusion but > > default_statistics_target I don't. > > Why not? I don't want to accept a big increase in ANALYZE times (or > planning times, though I'm really not seeing that at this point) > withou

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
> I can see an argument about constraint_exclusion but > default_statistics_target I don't. Why not? I don't want to accept a big increase in ANALYZE times (or planning times, though I'm really not seeing that at this point) without some benefit. >> It seems unlikely that you would want 256 MB o

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
> Well did you have any response to what I posited before? I said "mixed" should > produce the same settings that the default initdb settings produce. At least > on a moderately low-memory machine that initdb targets. I'm actually really skeptical of this whole idea of modes. The main thing mode

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Joshua D. Drake
On Wed, 2008-12-03 at 15:21 -0500, Robert Haas wrote: > >> I'm not sure what "mixed" mode is supposed to be, but based on what > >> I've seen so far, I'm a skeptical of the idea that encouraging people > >> to raise default_statistics_target to 50 and turn on > >> constraint_exclusion is reasonable

Re: [HACKERS] snapshot leak and core dump with serializable transactions

2008-12-03 Thread Alvaro Herrera
Heikki Linnakangas escribió: > I'm surprised you implemented RegisterSnapshotOnOwner by switching > CurrentResourceOwner and calling RegisterSnapshot, rather than > implementing RegisterSnapshot by calling RegisterSnapshotOnOwner(..., > CurrentResourceOwner). Yeah, that was plenty silly. U

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > On Wed, 2008-12-03 at 13:30 -0500, Robert Haas wrote: >> I'm not sure what "mixed" mode is supposed to be, but based on what >> I've seen so far, I'm a skeptical of the idea that encouraging people >> to raise default_statistics_target to 50 and turn

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
>> I'm not sure what "mixed" mode is supposed to be, but based on what >> I've seen so far, I'm a skeptical of the idea that encouraging people >> to raise default_statistics_target to 50 and turn on >> constraint_exclusion is reasonable. > > Why? Because both of those settings are strictly worse

Re: [HACKERS] snapshot leak and core dump with serializable transactions

2008-12-03 Thread Heikki Linnakangas
Alvaro Herrera wrote: Alvaro Herrera escribió: Yeah, we need two "at-commit" routines, one of which needs to be called early. I'm prepping a patch. Here it is ... the large object patch is also included. I've created new functions to specify the resource owner to register a snapshot in; now

Re: [HACKERS] Re: [BUGS] libpq does not manage SSL callbacks properly when other libraries are involved.

2008-12-03 Thread Magnus Hagander
Magnus Hagander wrote: > Bruce Momjian wrote: >> Bruce Momjian wrote: >>> Thanks for the review, Magnus. I have adjusted the patch to use the >>> same mutex every time the counter is accessed, and adjusted the >>> pqsecure_destroy() call to properly decrement in the right place. >>> >>> Also, I re

Re: [HACKERS] snapshot leak and core dump with serializable transactions

2008-12-03 Thread Alvaro Herrera
Alvaro Herrera escribió: > Yeah, we need two "at-commit" routines, one of which needs to be called > early. I'm prepping a patch. Here it is ... the large object patch is also included. I've created new functions to specify the resource owner to register a snapshot in; now that there are two ca

Re: [HACKERS] snapshot leak and core dump with serializable transactions

2008-12-03 Thread Alvaro Herrera
Pavan Deolasee escribió: > On Wed, Dec 3, 2008 at 7:42 PM, Tom Lane <[EMAIL PROTECTED]> wrote: > > > That's absolutely wrong. It'll complain about whatever snapshots the > > owners still hold. > > You must be right; I don't understand that code much. But don't we expect > the snapshots to be clea

Re: [HACKERS] Transactions and temp tables

2008-12-03 Thread Emmanuel Cecchet
I would really like to have support for temp tables at least for the case where the table is created and dropped in the same transaction. But I guess that the other limitations on index, sequences and views would still hold, right? manu Heikki Linnakangas wrote: Emmanuel Cecchet wrote: There

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Joshua D. Drake
On Wed, 2008-12-03 at 13:30 -0500, Robert Haas wrote: > > Looks like I need to add Python 2.5+Linux to my testing set. I did not > > expect that the UNIX distributions of Python 2.5 would ship with wintypes.py > > at all. I think I can fix this on the spot though. On line 40, you'll find > > thi

Re: [HACKERS] Visibility map, partial vacuums

2008-12-03 Thread Heikki Linnakangas
Gregory Stark wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: Gregory Stark wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: Hmm. It just occurred to me that I think this circumvented the anti-wraparound vacuuming: a normal vacuum doesn't advance relfrozenxid anymore. We'll need to

Re: [HACKERS] Simple postgresql.conf wizard

2008-12-03 Thread Robert Haas
> Looks like I need to add Python 2.5+Linux to my testing set. I did not > expect that the UNIX distributions of Python 2.5 would ship with wintypes.py > at all. I think I can fix this on the spot though. On line 40, you'll find > this bit: > > except ImportError: > > Change that to the followin

Re: [HACKERS] cvs head initdb hangs on unixware

2008-12-03 Thread Heikki Linnakangas
[EMAIL PROTECTED] wrote: On Tue, 2 Dec 2008, Heikki Linnakangas wrote: Date: Tue, 02 Dec 2008 20:47:19 +0200 From: Heikki Linnakangas <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Cc: Zdenek Kotala <[EMAIL PROTECTED]>, pgsql-hackers list Subject: Re: [HACKERS] cvs head initdb hangs on unixware

Re: [HACKERS] cvs head initdb hangs on unixware

2008-12-03 Thread Andrew Dunstan
[EMAIL PROTECTED] wrote: Looking at fsm_rebuild_page, I wonder if the compiler is treating "int" as an unsigned integer? That would cause an infinite loop. No, a simple printf of nodeno shows it starting at 4096 all the way down to 0, starting back at 4096... I wonder if leftchild/righ

Re: [HACKERS] Visibility map, partial vacuums

2008-12-03 Thread Gregory Stark
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > Gregory Stark wrote: >> Heikki Linnakangas <[EMAIL PROTECTED]> writes: >> >>> Hmm. It just occurred to me that I think this circumvented the >>> anti-wraparound >>> vacuuming: a normal vacuum doesn't advance relfrozenxid anymore. We'll need >>> to

Re: [HACKERS] snapshot leak and core dump with serializable transactions

2008-12-03 Thread Pavan Deolasee
On Wed, Dec 3, 2008 at 7:42 PM, Tom Lane <[EMAIL PROTECTED]> wrote: > > > That's absolutely wrong. It'll complain about whatever snapshots the > owners still hold. > > You must be right; I don't understand that code much. But don't we expect the snapshots to be cleanly released at that point and

Re: [HACKERS] Visibility map, partial vacuums

2008-12-03 Thread Heikki Linnakangas
Gregory Stark wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: Hmm. It just occurred to me that I think this circumvented the anti-wraparound vacuuming: a normal vacuum doesn't advance relfrozenxid anymore. We'll need to disable the skipping when autovacuum is triggered to prevent wraparou

Re: [HACKERS] [PATCHES] GIN improvements

2008-12-03 Thread Heikki Linnakangas
Tom Lane wrote: Greg Stark <[EMAIL PROTECTED]> writes: If we do this though it would be really nice to do it at a higher level than the indexam. If we could do it for any indexam that provides a kind of bulk insert method that would be great. I'm just not sure how to support all the indexab

Re: [HACKERS] tuplestore potential performance problem

2008-12-03 Thread Hitoshi Harada
2008/12/3 Tom Lane <[EMAIL PROTECTED]>: > If this means a lot of contortion/complication in the upper-level code, > seems like it'd be better to address the performance issue within > tuplestore/buffile. We could keep separate buffers for write and read > perhaps. But do you have real evidence of

Re: [HACKERS] tuplestore potential performance problem

2008-12-03 Thread Hitoshi Harada
> I don't have real evidence but reasoned it. No strace was done. So it > may not be cased by flushing out but this commit gets performance > quite better, to earlier patch performance, around 44sec from around > 76sec. > Oh, I mean, 116sec to 44sec. -- Hitoshi Harada -- Sent via pgsql-hacker

Re: [HACKERS] [PATCHES] GIN improvements

2008-12-03 Thread Tom Lane
Greg Stark <[EMAIL PROTECTED]> writes: > If we do this though it would be really nice to do it at a higher > level than the indexam. If we could do it for any indexam that > provides a kind of bulk insert method that would be great. > I'm just not sure how to support all the indexable operator

Re: [HACKERS] Sync Rep: First Thoughts on Code

2008-12-03 Thread Simon Riggs
On Wed, 2008-12-03 at 21:37 +0900, Fujii Masao wrote: > Since I thought that the figure was more intelligible for some people > than my poor English, I illustrated the architecture first. > http://wiki.postgresql.org/wiki/NTT%27s_Development_Projects#Detailed_Design > > Are there any other parts

Re: [HACKERS] tuplestore potential performance problem

2008-12-03 Thread Tom Lane
"Hitoshi Harada" <[EMAIL PROTECTED]> writes: > While attacking this issue(*1), I found that tuplestore that is on the > file status has potential performance problem. > The performance problem introduced by Heikki's new approach was caused > by BufFile's frequent flush out in such cases like you p

[HACKERS] tuplestore potential performance problem

2008-12-03 Thread Hitoshi Harada
While attacking this issue(*1), I found that tuplestore that is on the file status has potential performance problem. The performance problem introduced by Heikki's new approach was caused by BufFile's frequent flush out in such cases like you put a new row into it and read middle row of it then p

Re: [HACKERS] snapshot leak and core dump with serializable transactions

2008-12-03 Thread Tom Lane
"Pavan Deolasee" <[EMAIL PROTECTED]> writes: > 2. In CommitTransaction(), I think we should call AtEOXact_Snapshot *before* > releasing the resource owners. That's absolutely wrong. It'll complain about whatever snapshots the owners still hold. regards, tom lane -- Sent

Re: [HACKERS] Visibility map, partial vacuums

2008-12-03 Thread Magnus Hagander
Gregory Stark wrote: > Heikki Linnakangas <[EMAIL PROTECTED]> writes: > >> Hmm. It just occurred to me that I think this circumvented the >> anti-wraparound >> vacuuming: a normal vacuum doesn't advance relfrozenxid anymore. We'll need >> to >> disable the skipping when autovacuum is triggered t

Re: [HACKERS] Visibility map, partial vacuums

2008-12-03 Thread Alvaro Herrera
Heikki Linnakangas wrote: > Hmm. It just occurred to me that I think this circumvented the > anti-wraparound vacuuming: a normal vacuum doesn't advance relfrozenxid > anymore. We'll need to disable the skipping when autovacuum is triggered > to prevent wraparound. VACUUM FREEZE does that alr

Re: [HACKERS] Visibility map, partial vacuums

2008-12-03 Thread Gregory Stark
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > Hmm. It just occurred to me that I think this circumvented the anti-wraparound > vacuuming: a normal vacuum doesn't advance relfrozenxid anymore. We'll need to > disable the skipping when autovacuum is triggered to prevent wraparound. > VACUUM > FR

Re: [HACKERS] Re: [BUGS] libpq does not manage SSL callbacks properly when other libraries are involved.

2008-12-03 Thread Magnus Hagander
Bruce Momjian wrote: > Bruce Momjian wrote: >> Thanks for the review, Magnus. I have adjusted the patch to use the >> same mutex every time the counter is accessed, and adjusted the >> pqsecure_destroy() call to properly decrement in the right place. >> >> Also, I renamed the libpq global destroy

Re: [HACKERS] Visibility map, partial vacuums

2008-12-03 Thread Heikki Linnakangas
Heikki Linnakangas wrote: Here's an updated version, with a lot of smaller cleanups, and using relcache invalidation to notify other backends when the visibility map fork is extended. I already committed the change to FSM to do the same. I'm feeling quite satisfied to commit this patch early ne

Re: [HACKERS] cvs head initdb hangs on unixware

2008-12-03 Thread ohp
On Tue, 2 Dec 2008, Heikki Linnakangas wrote: Date: Tue, 02 Dec 2008 20:47:19 +0200 From: Heikki Linnakangas <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Cc: Zdenek Kotala <[EMAIL PROTECTED]>, pgsql-hackers list Subject: Re: [HACKERS] cvs head initdb hangs on unixware [EMAIL PROTECTED] wrote:

Re: [HACKERS] maintenance memory vs autovac

2008-12-03 Thread Alvaro Herrera
Magnus Hagander wrote: > Tom Lane wrote: > > It seems like mostly a confusion-generator to me. Is there any actual > > evidence that autovac should use a different maintenance_work_mem than > > other processes? > > The use-case that made me think of that is one with lots of autovac > workers in

Re: [HACKERS] snapshot leak and core dump with serializable transactions

2008-12-03 Thread Alvaro Herrera
Pavan Deolasee escribió: > 2. In CommitTransaction(), I think we should call AtEOXact_Snapshot *before* > releasing the resource owners. Otherwise, ResourceOwnerReleaseInternal > complains about snapshot leak and then forcefully unregisters the snapshot. > Later when AtEOXact_Snapshot is called, i

[HACKERS] pg_get_keywords descriptions

2008-12-03 Thread Peter Eisentraut
=> select distinct catcode, catdesc from pg_get_keywords(); catcode |catdesc -+--- C | Column name T | Type or function name R | Reserved U | Unreserved I find the descriptions of C and T quite confusing. For example, saying that

Re: [HACKERS] Sync Rep: First Thoughts on Code

2008-12-03 Thread Fujii Masao
Hi, On Tue, Dec 2, 2008 at 10:09 PM, Simon Riggs <[EMAIL PROTECTED]> wrote: > > On Tue, 2008-12-02 at 21:37 +0900, Fujii Masao wrote: > >> Thanks for taking many hours to review the code!! >> >> On Mon, Dec 1, 2008 at 8:42 PM, Simon Riggs <[EMAIL PROTECTED]> wrote: >> > Can you confirm that all th

Re: [HACKERS] maintenance memory vs autovac

2008-12-03 Thread Magnus Hagander
Guillaume Smet wrote: > On Wed, Dec 3, 2008 at 10:49 AM, Magnus Hagander <[EMAIL PROTECTED]> wrote: >>> The autovacuum workers change that and make it a default behaviour (as >>> we can have 3*maintenance_work_mem by default). >> It's still one per process, it's just that autovac uses more than one

[HACKERS] snapshot leak and core dump with serializable transactions

2008-12-03 Thread Pavan Deolasee
The following test flashes snapshot leak warning and subsequently dumps core. Though this looks very similar to other bug report, this is a different issue. postgres=# BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE ; BEGIN postgres=# SAVEPOINT A; SAVEPOINT postgres=# SELECT count(*) from pg_class

Re: [HACKERS] maintenance memory vs autovac

2008-12-03 Thread Magnus Hagander
Gregory Stark wrote: > "Guillaume Smet" <[EMAIL PROTECTED]> writes: > >> On Wed, Dec 3, 2008 at 10:49 AM, Magnus Hagander <[EMAIL PROTECTED]> wrote: >>> It's probably worthwhile to add a note about the effects of >>> autovacuum around the documentation of maintenance_work_mem, though. >> +1 >> A l

Re: [HACKERS] maintenance memory vs autovac

2008-12-03 Thread Gregory Stark
"Guillaume Smet" <[EMAIL PROTECTED]> writes: > On Wed, Dec 3, 2008 at 10:49 AM, Magnus Hagander <[EMAIL PROTECTED]> wrote: >> It's probably worthwhile to add a note about the effects of >> autovacuum around the documentation of maintenance_work_mem, though. > > +1 > A lot of people set maintenance

Re: [HACKERS] maintenance memory vs autovac

2008-12-03 Thread Guillaume Smet
On Wed, Dec 3, 2008 at 10:49 AM, Magnus Hagander <[EMAIL PROTECTED]> wrote: >> The autovacuum workers change that and make it a default behaviour (as >> we can have 3*maintenance_work_mem by default). > > It's still one per process, it's just that autovac uses more than one > process. I agree. Wha

Re: [HACKERS] Re: [COMMITTERS] pgsql: Explicitly bind gettext() to the UTF8 locale when in use.

2008-12-03 Thread Magnus Hagander
Hiroshi Inoue wrote: >> I think the thing us that as long as the encodings are compatible >> (latin1 with different names for example) it worked fine. >> >>> In any case I think the problem is that gettext is >>> looking at a setting that is not what we are looking at. Particularly >>> with the

Re: [HACKERS] Erroring out on parser conflicts

2008-12-03 Thread Greg Stark
On 3 Dec 2008, at 03:32 AM, Tom Lane <[EMAIL PROTECTED]> wrote: Bruce Momjian <[EMAIL PROTECTED]> writes: FYI, this is going to make it hard for developers to test CVS changes until they get their grammar cleaned up; perhaps add a comment on how to disable the check? Well, the point is

Re: [HACKERS] maintenance memory vs autovac

2008-12-03 Thread Magnus Hagander
Guillaume Smet wrote: > On Wed, Dec 3, 2008 at 2:00 AM, Tom Lane <[EMAIL PROTECTED]> wrote: >> It seems like mostly a confusion-generator to me. Is there any actual >> evidence that autovac should use a different maintenance_work_mem than >> other processes? > > IMHO, the point is that we were us

Re: [HACKERS] maintenance memory vs autovac

2008-12-03 Thread Magnus Hagander
Tom Lane wrote: > Magnus Hagander <[EMAIL PROTECTED]> writes: >> Greg Stark wrote: >>> One concern I have about this is people asking "how come when I >>> runvacuum manually it takes x minutes but when autovacuum runs it it >>> tale 5x minutes?" > >> As long as the default is the same, people woul

Re: [HACKERS] pg_stat_all_tables vs NULLs

2008-12-03 Thread Magnus Hagander
Tom Lane wrote: > Magnus Hagander <[EMAIL PROTECTED]> writes: >> I've noticed that pg_stat_all_tables returns NULL for idx_scan and >> idx_tup_fetch if there are no indexes present on a table. > >> Is this actually intended, or is that something that should be fixed? > > Hmm. I suspect it's an i

Re: [HACKERS] [PATCHES] GIN improvements

2008-12-03 Thread Greg Stark
On 3 Dec 2008, at 06:57 AM, Heikki Linnakangas <[EMAIL PROTECTED] > wrote: Tom Lane wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: If *that* is a use case we're interested in, the incoming tuples could be accumulated in backend-private memory, and inserted into the index at commi

Re: [HACKERS] pg_stop_backup wait bug fix

2008-12-03 Thread Ibrar Ahmed
Hi, I have looked at the patch and it looks OK to me. BTW I am not too much familiar with this area of code, so I am not at the position to argue that patch -:) . I haven't found an easy way to test the patch. On Wed, Dec 3, 2008 at 1:24 PM, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: > Fujii M

Re: [HACKERS] Transactions and temp tables

2008-12-03 Thread Heikki Linnakangas
Emmanuel Cecchet wrote: There is a problem with temp tables with on delete rows that are created inside a transaction. Take the 2pc_on_delete_rows_transaction.sql test case and change the creation statement, instead of create temp table foo(x int) on commit delete rows; try create temp table fo

Re: [HACKERS] Transactions and temp tables

2008-12-03 Thread Heikki Linnakangas
Emmanuel Cecchet wrote: I think that the Assert in is_temp_rel(Oid) in tablecmds.c should be replaced by if (on_commits == NULL) return false; As the use case below shows, a regular table can be created and hold a LOCKTAG_RELATION lock that will trigger the call to is_temp_rel in is_preparable_

Re: [HACKERS] pg_stop_backup wait bug fix

2008-12-03 Thread Heikki Linnakangas
Fujii Masao wrote: On Wed, Dec 3, 2008 at 5:13 AM, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: Agreed, should use XLByteToPrevSeg. But I wonder if we can just replace the current XLByteToSeg call with XLByteToPrevSeg? That would offset the return value of the function by one byte as well, as w