Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Tom Lane
Peter Eisentraut writes: > Greg Stark wrote: >> It would be perfectly reasonable to add an amisrecoverable like Simon >> described. It could automatically set indisvalid to false after a crash >> and treat the index as if indisvalid is false during recovery. That >> would be a lot smoother and

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Peter Eisentraut
Greg Stark wrote: It would be perfectly reasonable to add an amisrecoverable like Simon described. It could automatically set indisvalid to false after a crash and treat the index as if indisvalid is false during recovery. That would be a lot smoother and safer than what we have now. It might

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Greg Stark
It would be perfectly reasonable to add an amisrecoverable like Simon described. It could automatically set indisvalid to false after a crash and treat the index as if indisvalid is false during recovery. That would be a lot smoother and safer than what we have now. It might even be possibl

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Heikki Linnakangas
Pavan Deolasee wrote: BTW, if there is no proven case where hash index works significantly better than btree (that's what the doc says), why not just completely abandon it ? That has been considered many times, see archives. I believe the changes done in 8.4 actually made it faster for some ca

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Pavan Deolasee
On Thu, Dec 18, 2008 at 6:02 PM, Heikki Linnakangas wrote: > > In short, don't use hash index, unless you're prepared to run REINDEX > manually after every crash. > I think that should be mentioned in *bold* letters in the documentation. The doc currently has the following: "so hash indexes migh

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Heikki Linnakangas
Pavan Deolasee wrote: Just curious, how do we handle the case of corrupted hash index today? We don't. If we can detect that the index is corrupt because of bad page headers etc, then its still OK; we can throw an error. But what if the hash index is used after recovery and it returns wrong t

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Pavan Deolasee
On Thu, Dec 18, 2008 at 11:59 AM, Heikki Linnakangas wrote: > > Right, this is certainly not a new problem. It's not even a new problem in > the context of replication or hot standby, because we already have the > problem with PITR and file-based log shipping. > > Also, it's not just a problem *d

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Simon Riggs
On Thu, 2008-12-18 at 13:52 +0200, Peter Eisentraut wrote: > Simon Riggs wrote: > >> I think we should just leave it alone for 8.4, and fix it properly in a > >> future relase by implementing WAL-logging for hash indexes. > > > > You really think we should just leave it alone? That gives me less

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Peter Eisentraut
Simon Riggs wrote: I think we should just leave it alone for 8.4, and fix it properly in a future relase by implementing WAL-logging for hash indexes. You really think we should just leave it alone? That gives me less work, so I will accept that if you think so. Gives me the shivers though. A

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-18 Thread Simon Riggs
On Thu, 2008-12-18 at 08:29 +0200, Heikki Linnakangas wrote: > Right, this is certainly not a new problem. It's not even a new problem > in the context of replication or hot standby, because we already have > the problem with PITR and file-based log shipping. > > Also, it's not just a problem

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Heikki Linnakangas
Jeff Davis wrote: On Wed, 2008-12-17 at 17:10 -0600, Kenneth Marshall wrote: On Wed, Dec 17, 2008 at 06:07:41PM -0500, Jaime Casanova wrote: On Wed, Dec 17, 2008 at 5:47 PM, Kenneth Marshall wrote: Rebuilding a hash index for the case for which it is preferred (large, large tables) would be e

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Simon Riggs
On Wed, 2008-12-17 at 23:28 -0300, Alvaro Herrera wrote: > Hmm, it doesn't seem like you understood my suggestion ... basically I'm > saying that a hash index insert/delete should put out this WAL record: > > HEAP update address-of-pg_index-tuple set indisvalid=false > > (I'm just guessing at i

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Alvaro Herrera
Simon Riggs wrote: > > On Wed, 2008-12-17 at 21:23 -0300, Alvaro Herrera wrote: > > Maybe we should add a WAL record that's the physical representation for > > "mark this index invalid", and have any transaction that modifies a hash > > index write that to WAL. It should be simple code to write,

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Jeff Davis
On Wed, 2008-12-17 at 18:20 -0800, Jeff Davis wrote: > On Wed, 2008-12-17 at 17:10 -0600, Kenneth Marshall wrote: > > On Wed, Dec 17, 2008 at 06:07:41PM -0500, Jaime Casanova wrote: > > > On Wed, Dec 17, 2008 at 5:47 PM, Kenneth Marshall wrote: > > > > Rebuilding a hash index for the case > > > >

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Simon Riggs
On Wed, 2008-12-17 at 21:26 +, Simon Riggs wrote: > Hot Standby won't work with hash indexes because they are > non-recoverable. > > We have a number of ways of dealing with this: > 2. Specific Solution: make hashcostestimate() massively increase cost of > scans during recovery so that they

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Simon Riggs
On Wed, 2008-12-17 at 18:09 -0800, Josh Berkus wrote: > Are we really releasing an index type without recoverability for 8.4? > Will this be in /contrib? Worse than that, I'm talking about hash indexes (which are already here, without WAL) I would not contemplate a new index type without WAL.

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Jeff Davis
On Wed, 2008-12-17 at 17:10 -0600, Kenneth Marshall wrote: > On Wed, Dec 17, 2008 at 06:07:41PM -0500, Jaime Casanova wrote: > > On Wed, Dec 17, 2008 at 5:47 PM, Kenneth Marshall wrote: > > > Rebuilding a hash index for the case > > > for which it is preferred (large, large tables) would be excrut

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Josh Berkus
All, Are we really releasing an index type without recoverability for 8.4? Will this be in /contrib? --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Simon Riggs
On Wed, 2008-12-17 at 21:23 -0300, Alvaro Herrera wrote: > Simon Riggs wrote: > > > > On Wed, 2008-12-17 at 18:01 -0500, Jaime Casanova wrote: > > > On Wed, Dec 17, 2008 at 5:54 PM, Simon Riggs > > > wrote: > > > > > > > We don't yet have a mechanism for an > > > > index AM to say "damn, this

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Alvaro Herrera
Simon Riggs wrote: > > On Wed, 2008-12-17 at 18:01 -0500, Jaime Casanova wrote: > > On Wed, Dec 17, 2008 at 5:54 PM, Simon Riggs wrote: > > > > > We don't yet have a mechanism for an > > > index AM to say "damn, this index is screwed up, don't use it". > > > > mark pg_index.indisvalid and/or pg

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Simon Riggs
On Wed, 2008-12-17 at 18:01 -0500, Jaime Casanova wrote: > On Wed, Dec 17, 2008 at 5:54 PM, Simon Riggs wrote: > > > We don't yet have a mechanism for an > > index AM to say "damn, this index is screwed up, don't use it". > > > > mark pg_index.indisvalid and/or pg_index.indisready to false in t

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Kenneth Marshall
On Wed, Dec 17, 2008 at 05:10:40PM -0600, Kenneth Marshall wrote: > On Wed, Dec 17, 2008 at 06:07:41PM -0500, Jaime Casanova wrote: > > On Wed, Dec 17, 2008 at 5:47 PM, Kenneth Marshall wrote: > > > Rebuilding a hash index for the case > > > for which it is preferred (large, large tables) would be

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Kenneth Marshall
On Wed, Dec 17, 2008 at 06:07:41PM -0500, Jaime Casanova wrote: > On Wed, Dec 17, 2008 at 5:47 PM, Kenneth Marshall wrote: > > Rebuilding a hash index for the case > > for which it is preferred (large, large tables) would be excrutiating. > > > > there's such a situation? > As of 8.4, yes. Ken

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Jaime Casanova
On Wed, Dec 17, 2008 at 5:47 PM, Kenneth Marshall wrote: > Rebuilding a hash index for the case > for which it is preferred (large, large tables) would be excrutiating. > there's such a situation? -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sist

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Kenneth Marshall
On Wed, Dec 17, 2008 at 10:58:11PM +, Simon Riggs wrote: > > On Wed, 2008-12-17 at 16:47 -0600, Kenneth Marshall wrote: > > > I think having your index survive a server power outage or other > > crash is a very good thing. Rebuilding a hash index for the case > > for which it is preferred (la

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Jaime Casanova
On Wed, Dec 17, 2008 at 5:54 PM, Simon Riggs wrote: > We don't yet have a mechanism for an > index AM to say "damn, this index is screwed up, don't use it". > mark pg_index.indisvalid and/or pg_index.indisready to false in the hot standby node? -- Atentamente, Jaime Casanova Soporte y capacita

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Simon Riggs
On Wed, 2008-12-17 at 16:47 -0600, Kenneth Marshall wrote: > I think having your index survive a server power outage or other > crash is a very good thing. Rebuilding a hash index for the case > for which it is preferred (large, large tables) would be excrutiating. Completely agree. We may be o

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Simon Riggs
On Wed, 2008-12-17 at 17:42 -0500, Jaime Casanova wrote: > On Wed, Dec 17, 2008 at 4:26 PM, Simon Riggs wrote: > > Hot Standby won't work with hash indexes because they are > > non-recoverable. > > > > We have a number of ways of dealing with this: > > > > i don't see a reason for inventing the

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Kenneth Marshall
On Wed, Dec 17, 2008 at 05:42:41PM -0500, Jaime Casanova wrote: > On Wed, Dec 17, 2008 at 4:26 PM, Simon Riggs wrote: > > Hot Standby won't work with hash indexes because they are > > non-recoverable. > > > > We have a number of ways of dealing with this: > > > > i don't see a reason for inventin

Re: [HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Jaime Casanova
On Wed, Dec 17, 2008 at 4:26 PM, Simon Riggs wrote: > Hot Standby won't work with hash indexes because they are > non-recoverable. > > We have a number of ways of dealing with this: > i don't see a reason for inventing the wheel, we don't have wal for hash indexes because makes those more slow wi

[HACKERS] Preventing index scans for non-recoverable index AMs

2008-12-17 Thread Simon Riggs
Hot Standby won't work with hash indexes because they are non-recoverable. We have a number of ways of dealing with this: 1. Workaround: Implement WAL for hash indexes 2. Specific Solution: make hashcostestimate() massively increase cost of scans during recovery so that they will very seldom be