Re: Warn when parallel restoring a custom dump without data offsets

2020-07-17 Thread Tom Lane
I wrote: > Given the current state of affairs, I'm inclined to commit the > attached with no new test coverage, and then come back and look > at better testing after the other bugs are dealt with. Pushed back to v12. regards, tom lane

Re: Warn when parallel restoring a custom dump without data offsets

2020-07-15 Thread Tom Lane
I wrote: > The attached 0001 rewrites your 0001 as per the above ideas (dropping > the proposed doc change for now), and includes your 0004 for simplicity. > I'm including your 0002 verbatim just so the cfbot will be able to do a > meaningful test on 0001; but as stated, I don't really want to comm

Re: Warn when parallel restoring a custom dump without data offsets

2020-07-12 Thread Tom Lane
David Gilman writes: > On Thu, Jul 02, 2020 at 05:25:21PM -0400, Tom Lane wrote: >> I guess I'm completely confused about the purpose of these patches. >> Far from coping with the situation of an unseekable file, they appear >> to change pg_restore so that it fails altogether if it can't seek >> i

Re: Warn when parallel restoring a custom dump without data offsets

2020-07-07 Thread David Gilman
On Thu, Jul 02, 2020 at 05:25:21PM -0400, Tom Lane wrote: > I guess I'm completely confused about the purpose of these patches. > Far from coping with the situation of an unseekable file, they appear > to change pg_restore so that it fails altogether if it can't seek > its input file. Why would we

Re: Warn when parallel restoring a custom dump without data offsets

2020-07-02 Thread Tom Lane
David Gilman writes: > I've attached the latest patches after further review from Justin Pryzby. I guess I'm completely confused about the purpose of these patches. Far from coping with the situation of an unseekable file, they appear to change pg_restore so that it fails altogether if it can't s

Re: Warn when parallel restoring a custom dump without data offsets

2020-06-22 Thread Justin Pryzby
Adding Jim, since he ask about helping with perl. You can read the history of the patch here: https://commitfest.postgresql.org/28/2568/ https://www.postgresql.org/message-id/flat/CALBH9DDuJ+scZc4MEvw5uO-=vRyR2=QF9+Yh=3hpenkhwfs...@mail.gmail.com Some context: David is adding a test case for a

Re: Warn when parallel restoring a custom dump without data offsets

2020-06-13 Thread Justin Pryzby
On Mon, May 25, 2020 at 01:54:29PM -0500, David Gilman wrote: > > Is it possible to dump to stdout (or pipe to cat or dd) to avoid a new > > option ? > > The underlying IPC::Run code seems to support piping in a cross-platform > way. I am not a Perl master though and after spending an evening try

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-27 Thread David Gilman
I've attached the latest patches after further review from Justin Pryzby. -- David Gilman :DG< https://gilslotd.com >From 90e06cbb724f6f6a244dfc69f3d59ca2e7d29c01 Mon Sep 17 00:00:00 2001 From: David Gilman Date: Wed, 20 May 2020 22:49:28 -0400 Subject: [PATCH 1/4] Scan all TOCs when restoring

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-25 Thread David Gilman
The earlier patches weren't applying because I had "git config diff.noprefix true" set globally and that was messing up the git format-patch output. On Mon, May 25, 2020 at 01:54:29PM -0500, David Gilman wrote: > And I misunderstood how bad it was. I thought it was reading little > header structs

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-25 Thread David Gilman
Updated patches are attached, I ditched the gmail web interface so hopefully this works. Not mentioned in Justin's feedback: I dropped the extra sort in the test as it's no longer necessary. I also added a parallel dump -> parallel restore -> dump test run for the directory format to get some free

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-23 Thread Justin Pryzby
On Sat, May 23, 2020 at 03:54:30PM -0400, David Gilman wrote: > I've rounded this patch out with a test and I've set up the commitfest > website for this thread. The latest patches are attached and I think > they are ready for review. Thanks. https://commitfest.postgresql.org/28/2568/ I'm not sur

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-23 Thread David Gilman
I've rounded this patch out with a test and I've set up the commitfest website for this thread. The latest patches are attached and I think they are ready for review. On Wed, May 20, 2020 at 11:05 PM David Gilman wrote: > > I did some more digging. To keep everyone on the same page there are > fo

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-20 Thread David Gilman
I did some more digging. To keep everyone on the same page there are four different ways to order TOCs: 1. topological order, 2. dataLength order, size of the table, is always zero when pg_dump can't seek, 3. dumpId order, which should be thought as random but roughly correlates to topological ord

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-20 Thread Tom Lane
David Gilman writes: >> I think the PG11 >> commit you mentioned (548e5097) happens to make some databases fail in >> parallel restore that previously worked (I didn't check). > Correct, if you do the bisect around that yourself you'll see > pg_restore start failing with the expected "possibly du

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-20 Thread David Gilman
Your understanding of the issue is mostly correct: > I think the PG11 > commit you mentioned (548e5097) happens to make some databases fail in > parallel restore that previously worked (I didn't check). Correct, if you do the bisect around that yourself you'll see pg_restore start failing with th

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-19 Thread Justin Pryzby
I started fooling with this at home while our ISP is broke (pardon my brevity). Maybe you also saw commit b779ea8a9a2dc3a089b3ac152b1ec4568bfeb26f "Fix pg_restore so parallel restore doesn't fail when the input file doesn't contain data offsets (which it won't, if pg_dump thought its output wasn't

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-19 Thread Justin Pryzby
On Sat, May 16, 2020 at 04:57:46PM -0400, David Gilman wrote: > If pg_dump can't seek on its output stream when writing a dump in the > custom archive format (possibly because you piped its stdout to a file) > it can't update that file with data offsets. These files will often > break parallel rest

Warn when parallel restoring a custom dump without data offsets

2020-05-16 Thread David Gilman
If pg_dump can't seek on its output stream when writing a dump in the custom archive format (possibly because you piped its stdout to a file) it can't update that file with data offsets. These files will often break parallel restoration. Warn when the user is doing pg_restore on such a file to give