Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-05-10 Thread Tom Lane
I forgot to mention it in the commit message, but this patch http://archives.postgresql.org/pgsql-committers/2006-05/msg00069.php includes a fix to avoid "tuple concurrently updated" failures in CREATE INDEX's update of the parent relation's pg_class entry. I'm still not convinced that that's where

Re: Catalog Access (was: [HACKERS] [GENERAL] Concurrency problem building indexes)

2006-04-25 Thread Tom Lane
Martijn van Oosterhout writes: > I think the basic problem is that DDL can't really work within a > transaction. If I do an ALTER TABLE, some of these changes need to show > up to concurrent transactions (maybe creating a unique index?). The point is that DDL can't be MVCC. If for instance you a

Re: Catalog Access (was: [HACKERS] [GENERAL] Concurrency problem building indexes)

2006-04-25 Thread Martijn van Oosterhout
On Tue, Apr 25, 2006 at 12:25:35PM -0500, Jim C. Nasby wrote: > Is there anything in comments/docs/list archives about why catalog > access uses a bunch of 'magic' instead of treating catalog tables the > same as every other table? I realize that ultimately you have to > bootstrap somehow (kinda ha

Catalog Access (was: [HACKERS] [GENERAL] Concurrency problem building indexes)

2006-04-25 Thread Jim C. Nasby
On Tue, Apr 25, 2006 at 12:48:04PM -0400, Alvaro Herrera wrote: > I'm late to this thread, but maybe we can make the process of storing > the new data in pg_class take a lock using LockObject() or something > like that to serialize the access to the pg_class row. The idea would > be that this lock

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > I'm late to this thread, but maybe we can make the process of storing > the new data in pg_class take a lock using LockObject() or something > like that to serialize the access to the pg_class row. I'm inclined to think that the right solution is to fix

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Alvaro Herrera
I'm late to this thread, but maybe we can make the process of storing the new data in pg_class take a lock using LockObject() or something like that to serialize the access to the pg_class row. The idea would be that this lock doesn't conflict with a LockRelation(), but it would of course conflict

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
Wes <[EMAIL PROTECTED]> writes: > A while back, I saw a posting (I think from Tom) to the effect of index > creation converging due to disk caching. It was along the lines of the > slower index would be reading from disk data cached by the first index > creation's read. When the faster creation w

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Wes
On 4/25/06 9:58 AM, "Tom Lane" <[EMAIL PROTECTED]> wrote: > I can't think of any effect in the PG code that would make CREATE INDEX > commands tend to converge rather than diverge, so maybe there is some > external effect here. Wes, is your machine a dual processor by any > chance? Which OS X ve

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
"Zeugswetter Andreas DCP SD" <[EMAIL PROTECTED]> writes: > My answer to this would be to (have an option to) ommit this relpages > and reltuples update. Wouldn't help, unless you want to get rid of relhasindex too. > Wes, you could most likely solve your immediate problem if you did an > analyze

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Zeugswetter Andreas DCP SD
> > Wes, you could most likely solve your immediate problem if you did an > > analyze before creating the indexes. > > I can try that. Is that going to be a reasonable thing to do when there's > 100 million rows per table? I obviously want to minimize the number of > sequential passes through t

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Wes
On 4/25/06 2:18 AM, "Tom Lane" <[EMAIL PROTECTED]> wrote: > So invent some made-up data. I'd be seriously surprised if this > behavior has anything to do with the precise data being indexed. > Experiment around till you've got something you don't mind posting > that exhibits the behavior you see.

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Zeugswetter Andreas DCP SD
> > [ shrug... ] Worksforme. > > > > There is a short interval at the end of the first CREATE INDEX on the > > table where the problem would happen if another CREATE INDEX tries to > > modify the pg_class row before the first one's committed. > I did a pg_dumpall and removed the index creation

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
Wes <[EMAIL PROTECTED]> writes: > On 4/25/06 1:01 AM, "Tom Lane" <[EMAIL PROTECTED]> wrote: >>> Where do you want me to send it to? >> >> pgsql-bugs would be appropriate. > It's not that small that I would want to post it to a list. Also, I think > I've sanitized the data, but I wouldn't want to

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Wes
On 4/25/06 1:01 AM, "Tom Lane" <[EMAIL PROTECTED]> wrote: >> Where do you want me to send it to? > > pgsql-bugs would be appropriate. It's not that small that I would want to post it to a list. Also, I think I've sanitized the data, but I wouldn't want to post it on a public list. Or are you j

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Tom Lane
Wes <[EMAIL PROTECTED]> writes: > Where do you want me to send it to? pgsql-bugs would be appropriate. > Of course, since your hardware is different, it may not show up since it > appears to be a timing thing.. I'm on a PB G4 1Ghz. My G4 is in the shop at the moment, but Apple promised it back

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Wes
On 4/24/06 11:02 PM, "Tom Lane" <[EMAIL PROTECTED]> wrote: > [ shrug... ] Worksforme. > > There is a short interval at the end of the first CREATE INDEX on the > table where the problem would happen if another CREATE INDEX tries to > modify the pg_class row before the first one's committed. Tha

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Tom Lane
Wes <[EMAIL PROTECTED]> writes: > I can duplicate the error at will. [ shrug... ] Worksforme. There is a short interval at the end of the first CREATE INDEX on the table where the problem would happen if another CREATE INDEX tries to modify the pg_class row before the first one's committed. Tha

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Wes
>> Long ago. I just tested it again, and AFAICS you can create indexes >> concurrently so long as the underlying table isn't changing (ie, >> neither reltuples nor relpages changes). > > Hrm, the OP seemed to find a case that was having problems: > http://archives.postgresql.org/pgsql-general/200

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Wes
On 4/24/06 7:54 PM, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote: >> Long ago. I just tested it again, and AFAICS you can create indexes >> concurrently so long as the underlying table isn't changing (ie, >> neither reltuples nor relpages changes). > > Hrm, the OP seemed to find a case that was havi

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Jim C. Nasby
On Mon, Apr 24, 2006 at 08:42:41PM -0400, Tom Lane wrote: > "Jim C. Nasby" <[EMAIL PROTECTED]> writes: > > Was the code ever changed so that it won't update relpages if the number > > is the same? > > Long ago. I just tested it again, and AFAICS you can create indexes > concurrently so long as th

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Tom Lane
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > Was the code ever changed so that it won't update relpages if the number > is the same? Long ago. I just tested it again, and AFAICS you can create indexes concurrently so long as the underlying table isn't changing (ie, neither reltuples nor relpages

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Jim C. Nasby
On Mon, Apr 24, 2006 at 08:14:33PM -0400, Tom Lane wrote: > "Jim C. Nasby" <[EMAIL PROTECTED]> writes: > > Since this seems to only be an issue due to trying to update pg_class > > for the table, perhaps CREATE INDEX can just ignore errors there? > > Lessee, where would ignoring an error potential

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Tom Lane
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > Since this seems to only be an issue due to trying to update pg_class > for the table, perhaps CREATE INDEX can just ignore errors there? Lessee, where would ignoring an error potentially cause the greatest damage? I can hardly think of a less critical

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Jim C. Nasby
moving to -hackers On Sun, Apr 23, 2006 at 09:06:59AM -0500, Wes wrote: > Postgres: 8.1.3 > OS: Mac OS X 10.4.6 > > I've run into another concurrency issue - parallel building of indexes. > When I try to build multiple indexes at a time, I randomly get: > > ERROR: tuple concurrently updated >