Re: [HACKERS] parallel restore fixes

2009-03-11 Thread Andrew Dunstan
Josh Berkus wrote: Tom Lane wrote: Andrew Dunstan writes: OK, here 'tis. Looks fairly reasonable to me, but of course I haven't tested it. Well, I have to do a blitz of parallel restores next week, so hopefully that will hit any soft spots. I have a known outstanding bug to do with de

Re: [HACKERS] parallel restore fixes

2009-03-11 Thread Josh Berkus
Tom Lane wrote: Andrew Dunstan writes: OK, here 'tis. Looks fairly reasonable to me, but of course I haven't tested it. Well, I have to do a blitz of parallel restores next week, so hopefully that will hit any soft spots. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@post

Re: [HACKERS] parallel restore fixes

2009-03-10 Thread Tom Lane
Andrew Dunstan writes: > OK, here 'tis. Looks fairly reasonable to me, but of course I haven't tested it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgs

Re: [HACKERS] parallel restore fixes

2009-03-10 Thread Andrew Dunstan
Tom Lane wrote: How about this: by default, fmtId uses the same logic as now (one static PQExpBuffer). If told to by a call of init_parallel_dump_utils(), which need only be called by pg_restore during its startup, then it switches to using per-thread storage. init_parallel_dump_utils can be

Re: [HACKERS] parallel restore fixes

2009-03-09 Thread Tom Lane
Andrew Dunstan writes: > Tom Lane wrote: >> How early is early? The proposed call sites for init_dump_utils seem >> already long past the point where any libc-level infrastructure would >> think it is "initialization" time. > Well, I think at least it needs to be done where other threads won't

Re: [HACKERS] parallel restore fixes

2009-03-09 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan writes: Tom Lane wrote: Actually, why bother with init_dump_utils at all? Well, the Windows reference I have suggests TlsAlloc() needs to be called early in the initialisation process ... How early is early? The proposed call sites

Re: [HACKERS] parallel restore fixes

2009-03-09 Thread Tom Lane
Andrew Dunstan writes: > Tom Lane wrote: >> Actually, why bother with init_dump_utils at all? > Well, the Windows reference I have suggests TlsAlloc() needs to be > called early in the initialisation process ... How early is early? The proposed call sites for init_dump_utils seem already long

Re: [HACKERS] parallel restore fixes

2009-03-09 Thread Andrew Dunstan
Tom Lane wrote: It makes me a bit nervous because there are some other programs that are linking dumputils.c (psql and some in src/bin/scripts/) and even calling fmtId. Actually, why bother with init_dump_utils at all? fmtId could be made to initialize the ID variable for itself on firs

Re: [HACKERS] parallel restore fixes

2009-03-09 Thread Tom Lane
Alvaro Herrera writes: > Hmm, if pg_restore is the only program that's threaded, why are you > calling init_dump_utils on pg_dump and pg_dumpall? ... because fmtId will crash on *any* use without that. > It makes me a bit > nervous because there are some other programs that are linking > dumputi

Re: [HACKERS] parallel restore fixes

2009-03-09 Thread Tom Lane
Andrew Dunstan writes: > + void > + init_dump_utils() This should be > + void > + init_dump_utils(void) please. We don't do K&R C around here. I'd lose the added retval variable too; that's not contributing anything. > ! #endif; Semicolon is bogus here. Looks pretty sane otherwise.

Re: [HACKERS] parallel restore fixes

2009-03-09 Thread Alvaro Herrera
Andrew Dunstan wrote: > > The attached patch fixes two issues with parallel restore: > >* the static buffer problem in dumputils.c::fmtId() on Windows > (solution: use thread-local storage) >* ReopenPtr() is called too often Hmm, if pg_restore is the only program that's threaded, why

[HACKERS] parallel restore fixes

2009-03-09 Thread Andrew Dunstan
The attached patch fixes two issues with parallel restore: * the static buffer problem in dumputils.c::fmtId() on Windows (solution: use thread-local storage) * ReopenPtr() is called too often There is one remaining bug I know of that I can reproduce: we can get into deadlock when