I've been poking into bug #5626,
http://archives.postgresql.org/pgsql-bugs/2010-08/msg00291.php
What's basically going on here is:
1. User tried to suppress the public schema from the restore list.
2. Since almost everything in the dump depends on the public schema,
pg_restore skipped over most of
On Wed, Jun 23, 2010 at 2:26 AM, Tom Lane wrote:
> Uh, that doesn't fix anything: if you can't seek, a TOC at the end of
> the file is useless. And the cases where the writer can't seek are
> likely to be identically the ones where the reader can't seek, viz
> pg_dump piped to pg_restore (perhaps
On Wed, Jun 23, 2010 at 03:26, Tom Lane wrote:
> Andrew Dunstan writes:
>>> 4. Is there any value in back-porting the Windows FSEEKO support into
>>> 8.3 and 8.2? Arguably, not writing the data offsets is a performance
>>> bug. However a back-port won't do anything for people who are dumping
>>
Tom Lane wrote:
Another possibility is to just remove the inside-the-loop error test
altogether: make it just skip till it finds the desired item, and only
throw an error if it hits EOF without finding it. In the case that
the error test is trying to catch, this would mean significantly more
w
Andrew Dunstan writes:
> Tom Lane wrote:
>> In short, parallel pg_restore is guaranteed to fail on any input file
>> made with a pre-8.4 pg_dump on Windows.
> IIRC, you can reproduce this on Unix too by sending the output of
> pg_dump into a pipe. So it's not uniquely a Windows problem.
Right.
Tom Lane wrote:
In short, parallel pg_restore is guaranteed to fail on any input file
made with a pre-8.4 pg_dump on Windows. It may be that there's some
other mechanism involved in the reports we've gotten of parallel restore
failing only some of the time, but I'm thinking that the heretofore
On Tue, Jun 22, 2010 at 9:07 PM, Tom Lane wrote:
> 3. Perhaps pg_dump ought to emit a warning when it can't seek, instead
> of just silently not writing the data offsets. That behavior was okay
> before when lack of data offsets didn't really matter that much, but
> lack of data offsets is a seri
I've dug into the problem reported by Igor Neyman:
http://archives.postgresql.org/pgsql-admin/2010-06/msg00148.php
Unlike previous complainants, Igor was kind enough to supply a pg_dump
archive file that triggers the problem. What I find is that his dump
file contains no data offsets, ie, dataStat
Another 0.02c, bringing the grand total to 0.04c.
Andrew Dunstan wrote:
> First, we need a way to decide the boundary between the serially run
> "pre-data" section and the remainder of the items in the TOC.
> Currently the code uses the first TABLEDATA item as the boundary.
> That's not terribly
There are a couple of open questions for parallel pg_restore.
First, we need a way to decide the boundary between the serially run
"pre-data" section and the remainder of the items in the TOC. Currently
the code uses the first TABLEDATA item as the boundary. That's not
terribly robust (what i
Andrew Dunstan wrote:
Stefan Kaltenbrunner wrote:
Tom Lane wrote:
Andrew Dunstan <[EMAIL PROTECTED]> writes:
Tom Lane wrote:
Um, FKs could conflict with each other too, so that by itself isn't
gonna fix anything.
Good point. Looks like we'll need to make a list of "can't run in
parallel
Philip Warner wrote:
Andrew Dunstan wrote:
Unfortunately, it quite possibly would. You would not be able to build
two indexes on the same table in parallel, even though they wouldn't
have conflicting locks.
I suppose so, but:
1. By the same logic it might speed things up; it might bu
Andrew Dunstan wrote:
> Unfortunately, it quite possibly would. You would not be able to build
> two indexes on the same table in parallel, even though they wouldn't
> have conflicting locks.
I suppose so, but:
1. By the same logic it might speed things up; it might build two
completely separate i
Philip Warner wrote:
+ if (strcmp(te->desc,"CONSTRAINT") == 0
||
+ strcmp(te->desc,"FK
CONSTRAINT") == 0 ||
+ strcmp(te->desc,"CHECK
CONSTRAINT") == 0 ||
+
>
> + if (strcmp(te->desc,"CONSTRAINT") == 0
> ||
> + strcmp(te->desc,"FK
> CONSTRAINT") == 0 ||
> + strcmp(te->desc,"CHECK
> CONSTRAINT") == 0 ||
> +
Stefan Kaltenbrunner wrote:
Tom Lane wrote:
Andrew Dunstan <[EMAIL PROTECTED]> writes:
Tom Lane wrote:
Um, FKs could conflict with each other too, so that by itself isn't
gonna fix anything.
Good point. Looks like we'll need to make a list of "can't run in
parallel with" items as well as
Tom Lane wrote:
Andrew Dunstan <[EMAIL PROTECTED]> writes:
Tom Lane wrote:
Um, FKs could conflict with each other too, so that by itself isn't
gonna fix anything.
Good point. Looks like we'll need to make a list of "can't run in
parallel with" items as well as strict dependencies.
Yeah, I
Dimitri Fontaine wrote:
Le lundi 29 septembre 2008, Tom Lane a écrit :
* Extend the archive format to provide some indication that "restoring
this object requires exclusive access to these dependencies".
* Hardwire knowledge into pg_restore that certain types of objects
require exclusive a
Le lundi 29 septembre 2008, Tom Lane a écrit :
> * Extend the archive format to provide some indication that "restoring
> this object requires exclusive access to these dependencies".
>
> * Hardwire knowledge into pg_restore that certain types of objects
> require exclusive access to their dependen
Andrew Dunstan <[EMAIL PROTECTED]> writes:
> Tom Lane wrote:
>> Um, FKs could conflict with each other too, so that by itself isn't
>> gonna fix anything.
> Good point. Looks like we'll need to make a list of "can't run in
> parallel with" items as well as strict dependencies.
Yeah, I was just t
Tom Lane wrote:
Andrew Dunstan <[EMAIL PROTECTED]> writes:
Tom Lane wrote:
Hmm, I'll bet the restore code doesn't realize that this can't run in
parallel with index creation on either table ...
Yeah. Of course, it's never needed to bother with stuff like that till now.
Andrew Dunstan <[EMAIL PROTECTED]> writes:
> Tom Lane wrote:
>> Hmm, I'll bet the restore code doesn't realize that this can't run in
>> parallel with index creation on either table ...
> Yeah. Of course, it's never needed to bother with stuff like that till now.
> The very simple fix is probably
Tom Lane wrote:
Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes:
pg_restore: [archiver (db)] could not execute query: ERROR: deadlock
detected
DETAIL: Process 18100 waits for AccessExclusiveLock on relation
1460818342 of database 1460815284; blocked by process 18103.
Process 18103 waits
Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes:
> pg_restore: [archiver (db)] could not execute query: ERROR: deadlock
> detected
> DETAIL: Process 18100 waits for AccessExclusiveLock on relation
> 1460818342 of database 1460815284; blocked by process 18103.
> Process 18103 waits for AccessExc
Andrew Dunstan wrote:
Andrew Dunstan wrote:
this works better but there is something fishy still - using the same
dump file I get a proper restore using pg_restore normally. If I
however use -m for a parallel one I only get parts (in this case only
243 of the 709 tables) of the database
Andrew Dunstan wrote:
this works better but there is something fishy still - using the same
dump file I get a proper restore using pg_restore normally. If I
however use -m for a parallel one I only get parts (in this case only
243 of the 709 tables) of the database restored ...
Yes
Joshua Drake wrote:
On Fri, 26 Sep 2008 17:10:44 -0400
Andrew Dunstan <[EMAIL PROTECTED]> wrote:
Yes, there are several funny things going on, including some stuff
with dependencies. I'll have a new patch tomorrow with luck. Thanks
for testing.
O.k. I took at look at the patch itsel
On Fri, 26 Sep 2008 17:10:44 -0400
Andrew Dunstan <[EMAIL PROTECTED]> wrote:
> Yes, there are several funny things going on, including some stuff
> with dependencies. I'll have a new patch tomorrow with luck. Thanks
> for testing.
O.k. I took at look at the patch itself and although I don't under
Stefan Kaltenbrunner wrote:
Andrew Dunstan wrote:
This version of the patch should fix the "shared file descriptor" bug
Russell Smith noticed. It also disables the 1/2 second sleep between
forks, so the performance on a small db (regression) is vastly improved.
this works better but ther
Andrew Dunstan wrote:
This version of the patch should fix the "shared file descriptor" bug
Russell Smith noticed. It also disables the 1/2 second sleep between
forks, so the performance on a small db (regression) is vastly improved.
this works better but there is something fishy still - us
This version of the patch should fix the "shared file descriptor" bug
Russell Smith noticed. It also disables the 1/2 second sleep between
forks, so the performance on a small db (regression) is vastly improved.
cheers
andrew
Index: pg_backup.h
===
Russell Smith wrote:
I'm sorry, I meant processes there. I'm aware there are no threads.
But my feeling was that when you forked with open files you got all of
the open file properties, including positions, and as you dupped the
descriptor, you share all that it's pointing to with every other
Andrew Dunstan wrote:
>> Do we know why we experience "tuple concurrently updated" errors if we
>> spawn thread too fast?
>>
>
> No. That's an open item.
Okay, I'll see if I can have a little more of a look into it. No
promises as the restore the restore isn't playing nicely.
>
>>
>> the memor
Stefan Kaltenbrunner wrote:
Andrew Dunstan wrote:
Attached is my WIP patch for parallel pg_restore. It's still very
rough, but seems to work.
Anyone who can test this with highend equipment would be helping some.
tried playing with this(on a 22Gb compressed dump using 4 connections)
but
Russell Smith wrote:
Hi,
As I'm interested in this topic, I thought I'd take a look at the
patch. I have no capability to test it on high end hardware but did
some basic testing on my workstation and basic review of the patch.
I somehow had the impression that instead of creating a new conne
Hi,
As I'm interested in this topic, I thought I'd take a look at the
patch. I have no capability to test it on high end hardware but did
some basic testing on my workstation and basic review of the patch.
I somehow had the impression that instead of creating a new connection
for each restore it
Andrew Dunstan wrote:
Attached is my WIP patch for parallel pg_restore. It's still very rough,
but seems to work.
Anyone who can test this with highend equipment would be helping some.
tried playing with this(on a 22Gb compressed dump using 4 connections)
but it does not seem to work at al
Attached is my WIP patch for parallel pg_restore. It's still very rough,
but seems to work.
Anyone who can test this with highend equipment would be helping some.
cheers
andrew
Index: pg_backup.h
===
RCS file: /cvsroot/pgsql/src
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Le 24 sept. 08 à 18:56, Andrew Dunstan a écrit :
The is purely a patch to pg_restore. No backend changes at all (and
if I did it would not use anything that isn't in core anyway).
Ok, good.
I'm eager to see what -core hackers will want to do with
Dimitri Fontaine wrote:
Le mercredi 24 septembre 2008, Andrew Dunstan a écrit :
No. The proposal will perform exactly the same set of steps as
single-threaded pg_restore, but in parallel. The individual steps won't
be broken up.
Ok, good for a solid trustworthy parallelism restore. W
Le mercredi 24 septembre 2008, Andrew Dunstan a écrit :
> No. The proposal will perform exactly the same set of steps as
> single-threaded pg_restore, but in parallel. The individual steps won't
> be broken up.
Ok, good for a solid trustworthy parallelism restore. Which is exactly what we
want. J
On Wed, 24 Sep 2008 07:52:52 +0100
Simon Riggs <[EMAIL PROTECTED]> wrote:
> > Just as an FYI, by far the number one bottle neck on the multiple
> > work restores I was doing was CPU. RAM and IO were never the
> > problem.
>
> It would be useful to see a full breakdown of those results.
Its in th
Dimitri Fontaine wrote:
Hi,
Le mardi 23 septembre 2008, Andrew Dunstan a écrit :
In any case, my agenda goes something like this:
* get it working with a basic selection algorithm on Unix (nearly
done - keep your eyes open for a patch soon)
* start testing
* get it worki
Hi,
Le mardi 23 septembre 2008, Andrew Dunstan a écrit :
> In any case, my agenda goes something like this:
>
> * get it working with a basic selection algorithm on Unix (nearly
> done - keep your eyes open for a patch soon)
> * start testing
> * get it working on Windows
> *
On Tue, 2008-09-23 at 22:17 -0700, Joshua D. Drake wrote:
> Simon Riggs wrote:
> > On Tue, 2008-09-23 at 16:50 -0400, Andrew Dunstan wrote:
> >
> >> If we get all that done by November we'll have done well. And we know
> >> that in some cases just this much can lead to reductions in restore
> >>
Simon Riggs wrote:
On Tue, 2008-09-23 at 16:50 -0400, Andrew Dunstan wrote:
If we get all that done by November we'll have done well. And we know
that in some cases just this much can lead to reductions in restore
time
of the order of 80%.
Agreed. Go for it.
Just as an FYI, by far the num
On Tue, 23 Sep 2008 16:50:43 -0400
Andrew Dunstan <[EMAIL PROTECTED]> wrote:
>
>
> Simon Riggs wrote:
> > On Tue, 2008-09-23 at 12:43 -0700, Joshua Drake wrote:
> >
> >> On Tue, 23 Sep 2008 08:44:19 +0100
> >> Simon Riggs <[EMAIL PROTECTED]> wrote:
> >>
> >>
> >>> On Mon, 2008-09-22 at 1
On Tue, 2008-09-23 at 16:50 -0400, Andrew Dunstan wrote:
> If we get all that done by November we'll have done well. And we know
> that in some cases just this much can lead to reductions in restore
> time
> of the order of 80%.
Agreed. Go for it.
--
Simon Riggs www.2ndQuadrant.co
Simon Riggs wrote:
On Tue, 2008-09-23 at 12:43 -0700, Joshua Drake wrote:
On Tue, 23 Sep 2008 08:44:19 +0100
Simon Riggs <[EMAIL PROTECTED]> wrote:
On Mon, 2008-09-22 at 15:05 -0400, Andrew Dunstan wrote:
j and m happen to be two of those that are available.
I honestly don'
On Tue, 2008-09-23 at 12:43 -0700, Joshua Drake wrote:
> On Tue, 23 Sep 2008 08:44:19 +0100
> Simon Riggs <[EMAIL PROTECTED]> wrote:
>
> >
> > On Mon, 2008-09-22 at 15:05 -0400, Andrew Dunstan wrote:
> >
> > > j and m happen to be two of those that are available.
> > >
> > > I honestly don't h
On Tue, 23 Sep 2008 08:44:19 +0100
Simon Riggs <[EMAIL PROTECTED]> wrote:
>
> On Mon, 2008-09-22 at 15:05 -0400, Andrew Dunstan wrote:
>
> > j and m happen to be two of those that are available.
> >
> > I honestly don't have a terribly strong opinion about what it
> > should be called. I can li
On Tue, 23 Sep 2008 09:14:33 +0200
"Stephen R. van den Berg" <[EMAIL PROTECTED]> wrote:
> Joshua D. Drake wrote:
> >Andrew Dunstan wrote:
> >>There are in fact very few letters available, as we've been fairly
> >>profligate in our use of option letters in the pg_dump suite.
>
> >>j and m happen
On Mon, 2008-09-22 at 15:05 -0400, Andrew Dunstan wrote:
> j and m happen to be two of those that are available.
>
> I honestly don't have a terribly strong opinion about what it should be
> called. I can live with jobs or multi-threads.
Perhaps we can use -j for jobs and -m for memory, so we
Joshua D. Drake wrote:
>Andrew Dunstan wrote:
>>There are in fact very few letters available, as we've been fairly
>>profligate in our use of option letters in the pg_dump suite.
>>j and m happen to be two of those that are available.
>--max-workers
Perhaps, but please do not use that as justif
Andrew Dunstan wrote:
There are in fact very few letters available, as we've been fairly
profligate in our use of option letters in the pg_dump suite.
j and m happen to be two of those that are available.
--max-workers
Max makes sense because the number of workers won't be consistent, a
wo
Simon Riggs wrote:
I will not argue vehemently here but I will say that "jobs" doesn't
seem correct. The term "workers" seems more appropriate.
Agreed, but most utilities have "j" free but not w, p, t or other
letters that might be synonyms.
j is at least used for exactly this purpose i
On Mon, 2008-09-22 at 09:30 -0700, Joshua Drake wrote:
> On Mon, 22 Sep 2008 17:24:28 +0100
> Simon Riggs <[EMAIL PROTECTED]> wrote:
>
> > > More importantly, I'm not convinced it's a good idea. It seems more
> > > like a footgun that will potentially try to launch thousands of
> > > simultaneous
Le lundi 22 septembre 2008, Joshua Drake a écrit :
> I will not argue vehemently here but I will say that "jobs" doesn't
> seem correct. The term "workers" seems more appropriate.
Mmmm, it sounds like it depends on the implementation (and how all workers
will share the same serializable transacti
On Mon, 22 Sep 2008 17:24:28 +0100
Simon Riggs <[EMAIL PROTECTED]> wrote:
> > More importantly, I'm not convinced it's a good idea. It seems more
> > like a footgun that will potentially try to launch thousands of
> > simultaneous restore connections. I should have thought that
> > optimal perform
On Mon, 2008-09-22 at 11:38 -0400, Andrew Dunstan wrote:
>
> Simon Riggs wrote:
> > On Mon, 2008-09-22 at 09:53 +0200, Dimitri Fontaine wrote:
> >
> >
> >>> My intention is to have single-thread restore remain the default, at
> >>> least for this go round, and have the user be able to choose
>
Simon Riggs wrote:
On Mon, 2008-09-22 at 09:53 +0200, Dimitri Fontaine wrote:
My intention is to have single-thread restore remain the default, at
least for this go round, and have the user be able to choose
--multi-thread=nn to specify the number of concurrent connections to use.
On Sun, 2008-09-21 at 18:15 -0400, Tom Lane wrote:
> Andrew Dunstan <[EMAIL PROTECTED]> writes:
> > I am working on getting parallel pg_restore working. I'm currently
> > getting all the scaffolding working, and hope to have a naive prototype
> > posted within about a week.
>
> > The major ques
On Mon, 2008-09-22 at 09:53 +0200, Dimitri Fontaine wrote:
> > My intention is to have single-thread restore remain the default, at
> > least for this go round, and have the user be able to choose
> > --multi-thread=nn to specify the number of concurrent connections to use.
>
> What about the ma
Le lundi 22 septembre 2008, Andrew Dunstan a écrit :
> > You'd really want the latter anyway for some cases, ie, when you don't
> > want the restore trying to hog the machine. Maybe the right form for
> > the extra option is just a limit on how many connections to use. Set it
> > to one to force
Andrew Dunstan <[EMAIL PROTECTED]> writes:
> I am working on getting parallel pg_restore working. I'm currently
> getting all the scaffolding working, and hope to have a naive prototype
> posted within about a week.
> The major question is how to choose the restoration order so as to
> maximize
Tom Lane wrote:
Andrew Dunstan <[EMAIL PROTECTED]> writes:
I am working on getting parallel pg_restore working. I'm currently
getting all the scaffolding working, and hope to have a naive prototype
posted within about a week.
The major question is how to choose the restoration or
I am working on getting parallel pg_restore working. I'm currently
getting all the scaffolding working, and hope to have a naive prototype
posted within about a week.
The major question is how to choose the restoration order so as to
maximize efficiency both on the server and in reading the ar
67 matches
Mail list logo
