Re: [PERFORM] Volunteer to build a configuration tool

2007-06-25 Thread Jim Nasby
On Jun 23, 2007, at 2:28 PM, Greg Smith wrote: On Thu, 21 Jun 2007, Campbell, Lance wrote: I have a PostgreSQL database that runs on a dedicated server. The server has 24Gig of memory. What would be the max size I would ever want to set the shared_buffers to if I where to relying on the OS for

Re: [PERFORM] Database-wide VACUUM ANALYZE

2007-06-25 Thread Jim Nasby
On Jun 21, 2007, at 3:37 PM, Steven Flatt wrote: Thanks everyone. It appears that we had hacked the 502.pgsql script for our 8.1 build to disable the daily vacuum. I was not aware of this when building and upgrading to 8.2. Much better to change stuff in a config file than to hack installe

Re: [PERFORM]

2007-06-25 Thread Tom Lane
Ed Tyrrill <[EMAIL PROTECTED]> writes: > It seems to me that the first plan is the optimal one for this case, but > when the planner has more information about the table it chooses not to > use it. Do you think that if work_mem were higher it might choose the > first plan again? It's worth foolin

Re: [PERFORM]

2007-06-25 Thread Stephen Frost
* Ed Tyrrill ([EMAIL PROTECTED]) wrote: > It seems to me that the first plan is the optimal one for this case, but > when the planner has more information about the table it chooses not to > use it. Do you think that if work_mem were higher it might choose the > first plan again? Seems likely to

Re: [PERFORM]

2007-06-25 Thread Ed Tyrrill
On Mon, 2007-06-25 at 18:10 -0400, Tom Lane wrote: > "Tyrrill, Ed" <[EMAIL PROTECTED]> writes: > > ... With 8.2.4.1 I get the same plan and performance with > > default_statistics_target set to either 10 or 100: > > There's something fishy about that, because AFAICS from the CVS logs, > there are

Re: [PERFORM]

2007-06-25 Thread Stephen Frost
* Ed Tyrrill ([EMAIL PROTECTED]) wrote: > Yes, work_mem was set to 128MB for all runs. All settings were the same > except for the change to default_statistics_target. I'm certainly > memory constrained, but giving 2GB to one one session doesn't allow > other sessions to do anything. Possibly wh

Re: [PERFORM]

2007-06-25 Thread Ed Tyrrill
On Mon, 2007-06-25 at 17:56 -0400, Stephen Frost wrote: > Was the 'work_mem' set to the same thing on all these runs? Also, you > might try increasing the 'work_mem' under 8.2.4, at least for this query > (you can set it by just doing: set work_mem = '2GB'; or similar in psql, > or you can change

Re: [PERFORM] startup caching suggestions

2007-06-25 Thread Bryan Murphy
No, but I was just informed of that trick earlier and intend to try it soon. Sometimes, the solution is so simple it's TOO obvious... :) Bryan On 6/25/07, Oleg Bartunov <[EMAIL PROTECTED]> wrote: On Mon, 25 Jun 2007, Bryan Murphy wrote: > We have a search facility in our database that uses f

Re: [PERFORM] startup caching suggestions

2007-06-25 Thread Oleg Bartunov
On Mon, 25 Jun 2007, Bryan Murphy wrote: We have a search facility in our database that uses full text indexing to search about 300,000 records spread across 2 tables. Nothing fancy there. The problem is, whenever we restart the database (system crash, lost connectivity to SAN, upgrade, config

Re: [PERFORM]

2007-06-25 Thread Tom Lane
"Tyrrill, Ed" <[EMAIL PROTECTED]> writes: > ... With 8.2.4.1 I get the same plan and performance with > default_statistics_target set to either 10 or 100: There's something fishy about that, because AFAICS from the CVS logs, there are no relevant planner changes between 8.2.3 and 8.2.4. You shoul

Re: [PERFORM]

2007-06-25 Thread Stephen Frost
* Tyrrill, Ed ([EMAIL PROTECTED]) wrote: > Based on all this we will be going with 8.2.4.1, but it seems like > currently the query planner isn't choosing the best plan for this case. Was the 'work_mem' set to the same thing on all these runs? Also, you might try increasing the 'work_mem' under 8

[PERFORM] Non-optimal query plan with 8.2

2007-06-25 Thread Tyrrill, Ed
Sorry to repost this, but I forgot the subject the first time around. Hey All, I am testing upgrading our database from version 8.1 to 8.2. I ran our worst performing query on this table, an outer join with an "is null" condition, and I was happy to see it ran over four times faster. I also not

[PERFORM]

2007-06-25 Thread Tyrrill, Ed
Hey All, I am testing upgrading our database from version 8.1 to 8.2. I ran our worst performing query on this table, an outer join with an "is null" condition, and I was happy to see it ran over four times faster. I also noticed the explain analyze showed the planner chose to do sequential scan

[PERFORM] startup caching suggestions

2007-06-25 Thread Bryan Murphy
We have a search facility in our database that uses full text indexing to search about 300,000 records spread across 2 tables. Nothing fancy there. The problem is, whenever we restart the database (system crash, lost connectivity to SAN, upgrade, configuration change, etc.) our data is not cache

Re: [PERFORM] PITR Backups

2007-06-25 Thread Tom Lane
Dan Gorman <[EMAIL PROTECTED]> writes: > I didn't however keep the snapshots around. I could try and re-set > this scenario up. I was in the middle of doing some data migration > with Netapp and wanted to just 'test' it to make sure it was sane. > If you guys would like me to try to 'break' it

Re: [PERFORM] PITR Backups

2007-06-25 Thread Dan Gorman
No, however, I will attach the postgreql.conf so everyone can look at other settings just in case. postgresql.conf Description: Binary data Regards, Dan Gorman On Jun 25, 2007, at 10:07 AM, Gregory Stark wrote: "Simon Riggs" <[EMAIL PROTECTED]> writes: WARNING: page 28900 of relation 166

Re: [PERFORM] PITR Backups

2007-06-25 Thread Tom Lane
"Simon Riggs" <[EMAIL PROTECTED]> writes: >> Reformatting and sorting, we have >> >> WARNING: page 28900 of relation 1663/16384/76718 was uninitialized >> WARNING: page 28902 of relation 1663/16384/76718 was uninitialized >> WARNING: page 26706 of relation 1663/16384/76719 was uninitialized >> WA

Re: [PERFORM] PITR Backups

2007-06-25 Thread Gregory Stark
"Simon Riggs" <[EMAIL PROTECTED]> writes: >> WARNING: page 28900 of relation 1663/16384/76718 was uninitialized >> WARNING: page 28902 of relation 1663/16384/76718 was uninitialized > >> WARNING: page 26706 of relation 1663/16384/76719 was uninitialized >> WARNING: page 26708 of relation 1663/1638

Re: [PERFORM] PITR Backups

2007-06-25 Thread Dan Gorman
Thanks for the pointers to a) make it readable and b) log min messages I didn't however keep the snapshots around. I could try and re-set this scenario up. I was in the middle of doing some data migration with Netapp and wanted to just 'test' it to make sure it was sane. If you guys would l

Re: [PERFORM] PITR Backups

2007-06-25 Thread Simon Riggs
On Mon, 2007-06-25 at 12:34 -0400, Tom Lane wrote: > Dan Gorman <[EMAIL PROTECTED]> writes: > > Jun 21 00:39:43 sfmedstorageha001 postgres[3506]: [9-1] 2007-06-21 > > 00:39:43 PDTLOG: redo done at 71/99870670 This is mid-way through an xlog file. > > Jun 21 00:39:43 sfmedstorageha001 postgres[

Re: [PERFORM] PITR Backups

2007-06-25 Thread Tom Lane
Dan Gorman <[EMAIL PROTECTED]> writes: > Jun 21 00:39:43 sfmedstorageha001 postgres[3506]: [9-1] 2007-06-21 > 00:39:43 PDTLOG: redo done at 71/99870670 > Jun 21 00:39:43 sfmedstorageha001 postgres[3506]: [10-1] 2007-06-21 > 00:39:43 PDTWARNING: page 28905 of relation 1663/16384/76718 was >

Re: [PERFORM] PITR Backups

2007-06-25 Thread Simon Riggs
On Mon, 2007-06-25 at 08:28 -0700, Dan Gorman wrote: > I took several snapshots. In all cases the FS was fine. In one case > the db looked like on recovery it thought there were outstanding > pages to be written to disk as seen below and the db wouldn't start. > > Jun 21 00:39:43 sfmedstorageh

Re: [PERFORM] PITR Backups

2007-06-25 Thread Dan Gorman
Greg, PG 8.2.4 Regards, Dan Gorman On Jun 25, 2007, at 9:02 AM, Gregory Stark wrote: "Dan Gorman" <[EMAIL PROTECTED]> writes: I took several snapshots. In all cases the FS was fine. In one case the db looked like on recovery it thought there were outstanding pages to be written to disk

Re: [PERFORM] PITR Backups

2007-06-25 Thread Gregory Stark
"Dan Gorman" <[EMAIL PROTECTED]> writes: > I took several snapshots. In all cases the FS was fine. In one case the db > looked like on recovery it thought there were outstanding pages to be written > to disk as seen below and the db wouldn't start. > > Jun 21 00:39:43 sfmedstorageha001 postgres[3

Re: [PERFORM] PITR Backups

2007-06-25 Thread Dan Gorman
I took several snapshots. In all cases the FS was fine. In one case the db looked like on recovery it thought there were outstanding pages to be written to disk as seen below and the db wouldn't start. Jun 21 00:39:43 sfmedstorageha001 postgres[3506]: [9-1] 2007-06-21 00:39:43 PDTLOG: redo

Re: [PERFORM] PITR Backups

2007-06-25 Thread Dan Gorman
It's the latter, is snapshot of the durable state of the storage system (e.g. it will never be corrupted) Regards, Dan Gorman On Jun 22, 2007, at 11:02 AM, Tom Lane wrote: "Simon Riggs" <[EMAIL PROTECTED]> writes: On Fri, 2007-06-22 at 13:12 -0400, Tom Lane wrote: If you saw a problem I'd

Re: [PERFORM] PITR Backups

2007-06-25 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > AFAIK, actually workable methods of this type depend on filesystem > cooperation, and are able to produce coherent snapshots of the logical > (not necessarily physical) filesystem content at a specific instant. I think you need filesystem cooperation in o

Re: [PERFORM] PITR Backups

2007-06-25 Thread Tom Lane
Koichi Suzuki <[EMAIL PROTECTED]> writes: > Year, I agree we should carefully follow how Done really did a backup. > My point is PostgreSQL may have to extend the file during the hot backup > to write to the new block. It is slightly different from Oracle's case. > Oracle allocates all the da

Re: [PERFORM] Slow indexscan

2007-06-25 Thread Tom Lane
"Mikko Partio" <[EMAIL PROTECTED]> writes: > This is already a great improvement compared to the previous 8 seconds. Our > app developers claim though that previously the same queries have run in > less than 1 second. The database had a mysterious crash a few months ago > (some tables lost their co

Re: [PERFORM] PITR Backups

2007-06-25 Thread Simon Riggs
On Mon, 2007-06-25 at 19:06 +0900, Koichi Suzuki wrote: > Year, I agree we should carefully follow how Done really did a backup. > My point is PostgreSQL may have to extend the file during the hot backup > to write to the new block. If the snapshot is a consistent, point-in-time copy then I d

Re: [PERFORM] Performance query about large tables, lots of concurrent access

2007-06-25 Thread Jean-David Beyer
Michael Stone wrote: > On Thu, Jun 21, 2007 at 12:29:49PM -0400, Karl Wright wrote: >> I checked the disk picture - this is a RAID disk array with 6 drives, >> with a bit more than 1Tbyte total storage. 15,000 RPM. It would be >> hard to get more/faster disk than that. > > Well, it's not hard

Re: [PERFORM] PostgreSQL Configuration Tool for Dummies - feedback adjustable control

2007-06-25 Thread Sabin Coanda
<[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > On Fri, 22 Jun 2007, Sabin Coanda wrote: > >> Instead of (or in addition to) configure dozens of settings, what do you >> say about a feedback adjustable control based on the existing system >> statistics and parsing logs (e.g >> ht

Re: [PERFORM] Performance query about large tables, lots of concurrent access

2007-06-25 Thread Michael Stone
On Thu, Jun 21, 2007 at 12:29:49PM -0400, Karl Wright wrote: I checked the disk picture - this is a RAID disk array with 6 drives, with a bit more than 1Tbyte total storage. 15,000 RPM. It would be hard to get more/faster disk than that. Well, it's not hard to more disk than that, but you'd

Re: [PERFORM] PITR Backups

2007-06-25 Thread Koichi Suzuki
Hi, Year, I agree we should carefully follow how Done really did a backup. My point is PostgreSQL may have to extend the file during the hot backup to write to the new block. It is slightly different from Oracle's case. Oracle allocates all the database space in advance so that there could

[PERFORM] Is AIX Concurrent IO safe with PostgreSQL?

2007-06-25 Thread Dawid Kuroczko
Hello, I am wondering if it is safe to assume that specifying cio mount option is safe with PostgreSQL. As far as I understand the CIO (AIX Concurrent I/O) means that filesystem does not serialize access to file blocks. In other words multiple threads can simultaneously read and write the file b

Re: [PERFORM] Slow indexscan

2007-06-25 Thread Mikko Partio
On 6/20/07, Tom Lane <[EMAIL PROTECTED]> wrote: There's no obvious reason for the previous query to be so slow, unless you've got horrendously slow or overloaded disk hardware. What sort of machine is this anyway, and was it doing any other work at the time? Granted it is doing other work