Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-14 Thread Bruce Momjian
On Tue, Dec 11, 2012 at 03:12:37PM -0500, Bruce Momjian wrote: > On Fri, Dec 7, 2012 at 04:49:19PM -0500, Bruce Momjian wrote: > > On Fri, Dec 7, 2012 at 10:38:39PM +0100, Andres Freund wrote: > > > On 2012-12-07 16:30:36 -0500, Bruce Momjian wrote: > > > > On Fri, Dec 7, 2012 at 04:21:48PM -050

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-11 Thread Bruce Momjian
On Tue, Dec 11, 2012 at 09:19:34PM +0100, Andres Freund wrote: > > > > Only 9.2 :(. Before that there was no DROP INDEX CONCURRENTLY and in 9.3 > > > > there's an actual indislive field and indisready is always set to false > > > > there if indislive is false. > > > > > > > > But I see no problem u

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-11 Thread Andres Freund
On 2012-12-11 15:12:37 -0500, Bruce Momjian wrote: > On Fri, Dec 7, 2012 at 04:49:19PM -0500, Bruce Momjian wrote: > > On Fri, Dec 7, 2012 at 10:38:39PM +0100, Andres Freund wrote: > > > On 2012-12-07 16:30:36 -0500, Bruce Momjian wrote: > > > > On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-11 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 04:49:19PM -0500, Bruce Momjian wrote: > On Fri, Dec 7, 2012 at 10:38:39PM +0100, Andres Freund wrote: > > On 2012-12-07 16:30:36 -0500, Bruce Momjian wrote: > > > On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: > > > > Andres Freund writes: > > > > > On 2012-12-

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 10:38:39PM +0100, Andres Freund wrote: > On 2012-12-07 16:30:36 -0500, Bruce Momjian wrote: > > On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: > > > Andres Freund writes: > > > > On 2012-12-07 13:59:41 -0500, Tom Lane wrote: > > > >> indisvalid should be sufficie

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 16:30:36 -0500, Bruce Momjian wrote: > On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: > > Andres Freund writes: > > > On 2012-12-07 13:59:41 -0500, Tom Lane wrote: > > >> indisvalid should be sufficient. If you try to test more than that > > >> you're going to make the co

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Bruce Momjian writes: > On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: >> Andres Freund writes: >>> Doesn't the check need to be at least indisvalid && indisready? Given >>> that 9.2 represents !indislive as indisvalid && !indisready? >> Um, good point. It's annoying that we had to d

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: > Andres Freund writes: > > On 2012-12-07 13:59:41 -0500, Tom Lane wrote: > >> indisvalid should be sufficient. If you try to test more than that > >> you're going to make the code more version-specific, without actually > >> buying much.

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Andres Freund writes: > On 2012-12-07 13:59:41 -0500, Tom Lane wrote: >> indisvalid should be sufficient. If you try to test more than that >> you're going to make the code more version-specific, without actually >> buying much. > Doesn't the check need to be at least indisvalid && indisready? G

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 11:57:34AM -0500, Bruce Momjian wrote: > On Fri, Dec 7, 2012 at 11:46:51AM -0500, Tom Lane wrote: > > Bruce Momjian writes: > > > On Fri, Dec 7, 2012 at 10:29:22AM -0500, Tom Lane wrote: > > >> On balance I am coming around to support the "just throw an error if > > >> th

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 13:59:41 -0500, Tom Lane wrote: > Bruce Momjian writes: > > On Fri, Dec 7, 2012 at 07:49:14PM +0100, Andres Freund wrote: > >> !indislive indexes are created during DROP INDEX CONCURRENTLY. Thats a > >> different case than CREATE INDEX CONCURRENTLY. Accessing their > >> definition i

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 13:54:44 -0500, Tom Lane wrote: > Andres Freund writes: > > !indislive indexes are created during DROP INDEX CONCURRENTLY. Thats a > > different case than CREATE INDEX CONCURRENTLY. Accessing their > > definition is actually problematic because i can vanish while youre > > examing it

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Bruce Momjian writes: > On Fri, Dec 7, 2012 at 07:49:14PM +0100, Andres Freund wrote: >> !indislive indexes are created during DROP INDEX CONCURRENTLY. Thats a >> different case than CREATE INDEX CONCURRENTLY. Accessing their >> definition is actually problematic because i can vanish while youre

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 07:49:14PM +0100, Andres Freund wrote: > On 2012-12-07 10:44:53 -0800, Josh Berkus wrote: > > > > > I wonder though if we shouldn't ignore !indislive indexes in pg_dump > > > (and the respective bw-compat hack). > > Obviously I wanted to ask whether we *should* ignore them

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Andres Freund writes: > !indislive indexes are created during DROP INDEX CONCURRENTLY. Thats a > different case than CREATE INDEX CONCURRENTLY. Accessing their > definition is actually problematic because i can vanish while youre > examing it which could cause errors both in the backend and in pg_

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 10:44:53 -0800, Josh Berkus wrote: > > > I wonder though if we shouldn't ignore !indislive indexes in pg_dump > > (and the respective bw-compat hack). Obviously I wanted to ask whether we *should* ignore them in the future. > Quite likely we shouldn't. However, that is why it wasn'

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Josh Berkus
> I wonder though if we shouldn't ignore !indislive indexes in pg_dump > (and the respective bw-compat hack). Quite likely we shouldn't. However, that is why it wasn't considered a problem. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 10:22:12 -0800, Josh Berkus wrote: > > > Yes, I thought of not dumping it. The problem is that we don't delete > > the index when it fails, so I assumed we didn't want to lose the index > > creation information. I need to understand why we did that. Why do we > > have pg_dump dump

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Josh Berkus
> Yes, I thought of not dumping it. The problem is that we don't delete > the index when it fails, so I assumed we didn't want to lose the index > creation information. I need to understand why we did that. Why do we > have pg_dump dump the index then? Because pg_restore will recreate the inde

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 11:46:51AM -0500, Tom Lane wrote: > Bruce Momjian writes: > > On Fri, Dec 7, 2012 at 10:29:22AM -0500, Tom Lane wrote: > >> On balance I am coming around to support the "just throw an error if > >> there are any invalid indexes" position. Adding extra complication in > >>

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Bruce Momjian writes: > On Fri, Dec 7, 2012 at 10:29:22AM -0500, Tom Lane wrote: >> On balance I am coming around to support the "just throw an error if >> there are any invalid indexes" position. Adding extra complication in >> pg_dump and pg_upgrade to handle ignoring them doesn't seem like a

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 10:29:22AM -0500, Tom Lane wrote: > Bruce Momjian writes: > > On Fri, Dec 7, 2012 at 03:32:51PM +0100, Andres Freund wrote: > >> ISTM that anybody who does DDL during or after pg_upgrade --check > >> deserves any pain. > >> > >> So throwing an error in both seems perfectl

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Bruce Momjian writes: > On Fri, Dec 7, 2012 at 03:32:51PM +0100, Andres Freund wrote: >> ISTM that anybody who does DDL during or after pg_upgrade --check >> deserves any pain. >> >> So throwing an error in both seems perfectly fine for me. I agree with Andres on this. > I am just saying that

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 03:32:51PM +0100, Andres Freund wrote: > > Well, it is a CONCURRENT index creation, so locking would be minimal. > > I wouldn't call a ShareUpdateExclusiveLock minimal... > > > Do we want pg_upgrade to be groveling over the lock view to look for > > locks? I don't think s

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 09:27:39 -0500, Bruce Momjian wrote: > On Fri, Dec 7, 2012 at 03:24:46PM +0100, Andres Freund wrote: > > On 2012-12-07 09:21:58 -0500, Bruce Momjian wrote: > > > On Thu, Dec 6, 2012 at 10:27:21PM -0500, Stephen Frost wrote: > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > >

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 03:24:46PM +0100, Andres Freund wrote: > On 2012-12-07 09:21:58 -0500, Bruce Momjian wrote: > > On Thu, Dec 6, 2012 at 10:27:21PM -0500, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > On Thu, Dec 6, 2012 at 09:45:11PM -0500, Stephen Frost wrot

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 09:21:58 -0500, Bruce Momjian wrote: > On Thu, Dec 6, 2012 at 10:27:21PM -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > On Thu, Dec 6, 2012 at 09:45:11PM -0500, Stephen Frost wrote: > > > > Or preserve it as-is. I don't really like the 'make them fi

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 09:16:56AM +, Simon Riggs wrote: > On 7 December 2012 04:02, Alvaro Herrera wrote: > > Andrew Dunstan wrote: > >> > >> On 12/06/2012 09:23 PM, Bruce Momjian wrote: > > > >> >As soon as pg_dump stopped dumping the CREATE INDEX, pg_upgrade would > >> >stop creating creati

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 10:27:21PM -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > On Thu, Dec 6, 2012 at 09:45:11PM -0500, Stephen Frost wrote: > > > Or preserve it as-is. I don't really like the 'make them fix it' > > > option, as a user could run into that in the mi

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Alvaro Herrera
Bruce Momjian wrote: > On Thu, Dec 6, 2012 at 09:23:14PM -0500, Bruce Momjian wrote: > > On Thu, Dec 6, 2012 at 09:10:21PM -0500, Tom Lane wrote: > > > Bruce Momjian writes: > > > > On Thu, Dec 6, 2012 at 07:53:57PM -0500, Tom Lane wrote: > > > >> Because CREATE INDEX CONCURRENTLY can't drop th

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 09:23:14PM -0500, Bruce Momjian wrote: > On Thu, Dec 6, 2012 at 09:10:21PM -0500, Tom Lane wrote: > > Bruce Momjian writes: > > > On Thu, Dec 6, 2012 at 07:53:57PM -0500, Tom Lane wrote: > > >> Because CREATE INDEX CONCURRENTLY can't drop the index if it's already > > >>

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Simon Riggs
On 7 December 2012 04:02, Alvaro Herrera wrote: > Andrew Dunstan wrote: >> >> On 12/06/2012 09:23 PM, Bruce Momjian wrote: > >> >As soon as pg_dump stopped dumping the CREATE INDEX, pg_upgrade would >> >stop creating creating it in the new cluster, and not transfer the index >> >files. >> >> So we

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Alvaro Herrera
Andrew Dunstan wrote: > > On 12/06/2012 09:23 PM, Bruce Momjian wrote: > >As soon as pg_dump stopped dumping the CREATE INDEX, pg_upgrade would > >stop creating creating it in the new cluster, and not transfer the index > >files. > > So we'll lose the index definition and leave some files behind

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Stephen Frost
* Bruce Momjian (br...@momjian.us) wrote: > I think we would have have pg_dump --binary-upgrade issue an UPDATE to > the system catalogs to mark it as invalid. That'd work for me too- I'm not particular on if it's done as a direct catalog update or some undocumented feature of CREATE INDEX.

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > Or preserve it as-is. > > To do that, we would have to add an option to CREATE INDEX to create it > in an invalid state. Which is stupid... Only in a binary-upgrade mode. Thanks, Stephen signature.asc

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Stephen Frost
* Bruce Momjian (br...@momjian.us) wrote: > On Thu, Dec 6, 2012 at 09:45:11PM -0500, Stephen Frost wrote: > > Or preserve it as-is. I don't really like the 'make them fix it' > > option, as a user could run into that in the middle of a planned upgrade > > that had been tested and never had that c

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 10:06:13PM -0500, Tom Lane wrote: > Stephen Frost writes: > > * Andrew Dunstan (and...@dunslane.net) wrote: > >> Making the user fix it seems much more sensible to me. Otherwise I > >> suspect we'll find users who get strangely surprised when they can > >> no longer find an

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 09:45:11PM -0500, Stephen Frost wrote: > * Andrew Dunstan (and...@dunslane.net) wrote: > > So we'll lose the index definition and leave some files behind? This > > sounds a bit messy to say the least. > > Agreed. > > > Making the user fix it seems much more sensible to me.

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 09:41:00PM -0500, Andrew Dunstan wrote: > > On 12/06/2012 09:23 PM, Bruce Momjian wrote: > >On Thu, Dec 6, 2012 at 09:10:21PM -0500, Tom Lane wrote: > >>Bruce Momjian writes: > >>>On Thu, Dec 6, 2012 at 07:53:57PM -0500, Tom Lane wrote: > Because CREATE INDEX CONCURR

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Tom Lane
Stephen Frost writes: > * Andrew Dunstan (and...@dunslane.net) wrote: >> Making the user fix it seems much more sensible to me. Otherwise I >> suspect we'll find users who get strangely surprised when they can >> no longer find any trace of an expected index in their upgraded >> database. > Or pr

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Stephen Frost
* Andrew Dunstan (and...@dunslane.net) wrote: > So we'll lose the index definition and leave some files behind? This > sounds a bit messy to say the least. Agreed. > Making the user fix it seems much more sensible to me. Otherwise I > suspect we'll find users who get strangely surprised when they

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Andrew Dunstan
On 12/06/2012 09:23 PM, Bruce Momjian wrote: On Thu, Dec 6, 2012 at 09:10:21PM -0500, Tom Lane wrote: Bruce Momjian writes: On Thu, Dec 6, 2012 at 07:53:57PM -0500, Tom Lane wrote: Because CREATE INDEX CONCURRENTLY can't drop the index if it's already failed. It's not because we want to d

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 09:10:21PM -0500, Tom Lane wrote: > Bruce Momjian writes: > > On Thu, Dec 6, 2012 at 07:53:57PM -0500, Tom Lane wrote: > >> Because CREATE INDEX CONCURRENTLY can't drop the index if it's already > >> failed. It's not because we want to do that, it's an implementation > >>

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Tom Lane
Bruce Momjian writes: > On Thu, Dec 6, 2012 at 07:53:57PM -0500, Tom Lane wrote: >> Because CREATE INDEX CONCURRENTLY can't drop the index if it's already >> failed. It's not because we want to do that, it's an implementation >> restriction of the horrid kluge that is CREATE/DROP INDEX CONCURREN

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Jeff Davis
On Thu, 2012-12-06 at 20:05 -0500, Andrew Dunstan wrote: > I think I prefer the first suggestion. If they are trying to upgrade > when there's an invalid index presumably they aren't aware of the > invalidity (or they would have done something about it). It would be > better to fail and make the

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 07:53:57PM -0500, Tom Lane wrote: > Bruce Momjian writes: > > Yes, I thought of not dumping it. The problem is that we don't delete > > the index when it fails, so I assumed we didn't want to lose the index > > creation information. I need to understand why we did that. >

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Andrew Dunstan
On 12/06/2012 07:58 PM, Jeff Davis wrote: On Thu, 2012-12-06 at 16:31 -0800, Josh Berkus wrote: There are a few possible fixes. The first would be to have pg_upgrade throw an error on any invalid index in the old cluster. Another option would be to preserve the invalid state in pg_dump --bina

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Jeff Davis
On Thu, 2012-12-06 at 16:31 -0800, Josh Berkus wrote: > > There are a few possible fixes. The first would be to have pg_upgrade > > throw an error on any invalid index in the old cluster. Another option > > would be to preserve the invalid state in pg_dump --binary-upgrade. > > Or to not dump in

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Tom Lane
Bruce Momjian writes: > Yes, I thought of not dumping it. The problem is that we don't delete > the index when it fails, so I assumed we didn't want to lose the index > creation information. I need to understand why we did that. Because CREATE INDEX CONCURRENTLY can't drop the index if it's alr

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 09:35:19PM -0300, Alvaro Herrera wrote: > Bruce Momjian wrote: > > > There are a few possible fixes. The first would be to have pg_upgrade > > throw an error on any invalid index in the old cluster. Another option > > would be to preserve the invalid state in pg_dump --bi

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Alvaro Herrera
Bruce Momjian wrote: > There are a few possible fixes. The first would be to have pg_upgrade > throw an error on any invalid index in the old cluster. Another option > would be to preserve the invalid state in pg_dump --binary-upgrade. Yet another option would be for pg_dump --binary-upgrade to

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-06 Thread Josh Berkus
> There are a few possible fixes. The first would be to have pg_upgrade > throw an error on any invalid index in the old cluster. Another option > would be to preserve the invalid state in pg_dump --binary-upgrade. Or to not dump invalid indexes at all in --binary-upgrade mode. > I also need h