Re: pg_dump test instability

2018-09-14 Thread Tom Lane
Peter Eisentraut writes: > On 13/09/2018 23:03, Tom Lane wrote: >> Attached updated patch does it like that and makes the cosmetic >> adjustments you suggested. I also went ahead and did the renaming >> of par_prev/par_next/par_list_xxx that I'd suggested upthread. >> I think this is committable

Re: pg_dump test instability

2018-09-14 Thread Peter Eisentraut
On 13/09/2018 23:03, Tom Lane wrote: > Attached updated patch does it like that and makes the cosmetic > adjustments you suggested. I also went ahead and did the renaming > of par_prev/par_next/par_list_xxx that I'd suggested upthread. > I think this is committable ... Yes, this looks good to me

Re: pg_dump test instability

2018-09-13 Thread Tom Lane
I wrote: > Peter Eisentraut writes: >> I see. Why not shift all items up to the i'th up by one place, instead >> of moving only the first one? That way the sortedness would be >> preserved. Otherwise we'd move the first one into the middle, then >> sorting would move it to the front again, etc.

Re: pg_dump test instability

2018-09-13 Thread Tom Lane
Peter Eisentraut writes: > On 12/09/2018 18:06, Tom Lane wrote: >> No. In both code paths, the array slot at index first_te is being >> physically dropped from the set of valid entries (by incrementing >> first_te). In the first path, that slot holds the item we want to >> remove logically from

Re: pg_dump test instability

2018-09-13 Thread Peter Eisentraut
On 12/09/2018 18:06, Tom Lane wrote: > No. In both code paths, the array slot at index first_te is being > physically dropped from the set of valid entries (by incrementing > first_te). In the first path, that slot holds the item we want to > remove logically from the set, so that incrementing fi

Re: pg_dump test instability

2018-09-12 Thread Tom Lane
Peter Eisentraut writes: > Some small comments on the code: > Maybe add a ready_list_free() to go with ready_list_init(), instead of > calling pg_free(ready_list.tes) directly. > get_next_work_item() has been changed to remove the work item from the > ready_list. Maybe rename to something like p

Re: pg_dump test instability

2018-09-12 Thread Peter Eisentraut
On 28/08/2018 20:47, Tom Lane wrote: > Here's a proposed patch for this. It removes the hacking of the TOC list > order, solving Peter's original problem, and instead sorts-by-size > in the actual parallel dump or restore control code. I have reviewed this patch. I haven't done any major perform

Re: pg_dump test instability

2018-08-28 Thread Tom Lane
Stephen Frost writes: > Parallel *restore* can be done from either a custom-format dump or from > a directory-format dump. I agree that we should seperate the concerns > and perform independent sorting on the restore side of things based on > the relative sizes of tables in the dump (be it custom

Re: pg_dump test instability

2018-08-27 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> However, at least for the directory-format case (which I think is the > >> only one supported for parallel restore), we could make it compare the > >> file sizes of the TABLE

Re: pg_dump test instability

2018-08-27 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> However, at least for the directory-format case (which I think is the >> only one supported for parallel restore), we could make it compare the >> file sizes of the TABLE DATA items. That'd work pretty well as a proxy >> for both t

Re: pg_dump test instability

2018-08-27 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Peter Eisentraut writes: > > In a non-data_only dump, the order of the tables doesn't matter, because > > the foreign keys are added at the very end. In parallel dumps, the > > tables are in addition sorted by size, so the resultant order is >

Re: pg_dump test instability

2018-08-27 Thread Tom Lane
Peter Eisentraut writes: > In a non-data_only dump, the order of the tables doesn't matter, because > the foreign keys are added at the very end. In parallel dumps, the > tables are in addition sorted by size, so the resultant order is > different from a single-threaded dump. This can be seen by

Re: pg_dump test instability

2018-08-27 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > During the development of an unrelated feature, I have experienced > failures in a pg_dump test case, specifically > > t/002_pg_dump.pl ... 1825/6052 > # Failed test 'defaults_parallel: should not dump COPY > fk_refer

pg_dump test instability

2018-08-26 Thread Peter Eisentraut
During the development of an unrelated feature, I have experienced failures in a pg_dump test case, specifically t/002_pg_dump.pl ... 1825/6052 # Failed test 'defaults_parallel: should not dump COPY fk_reference_test_table second' # at t/002_pg_dump.pl line 3454. This test sets up two tab