Re: Parallel Append can break run-time partition pruning

2021-11-12 Thread Justin Pryzby
On Mon, Nov 02, 2020 at 01:50:57PM +1300, David Rowley wrote: > On Tue, 27 Oct 2020 at 19:40, Amit Langote wrote: > > Some comments: > > Thanks for having a look at this. > > I've made some adjustments to those comments and pushed. commit a929e17e5 doesn't appear in the v14 release notes, but I

Re: Parallel Append can break run-time partition pruning

2020-11-01 Thread Amit Langote
On Mon, Nov 2, 2020 at 9:51 AM David Rowley wrote: > > On Tue, 27 Oct 2020 at 19:40, Amit Langote wrote: > > Some comments: > > Thanks for having a look at this. > > I've made some adjustments to those comments and pushed. Thank you. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Parallel Append can break run-time partition pruning

2020-11-01 Thread David Rowley
On Tue, 27 Oct 2020 at 19:40, Amit Langote wrote: > Some comments: Thanks for having a look at this. I've made some adjustments to those comments and pushed. David

Re: Parallel Append can break run-time partition pruning

2020-10-26 Thread Amit Langote
On Sun, Oct 25, 2020 at 10:06 AM David Rowley wrote: > On Fri, 16 Oct 2020 at 03:01, Amit Langote wrote: > > > > Going over the last few emails, it seems that David held off from > > committing the patch, because of the lack of confidence in its > > robustness. With the patch, a sub-partitioned

Re: Parallel Append can break run-time partition pruning

2020-10-24 Thread David Rowley
On Fri, 16 Oct 2020 at 03:01, Amit Langote wrote: > > Going over the last few emails, it seems that David held off from > committing the patch, because of the lack of confidence in its > robustness. With the patch, a sub-partitioned child's partial > Append's child paths will *always* be pulled

Re: Parallel Append can break run-time partition pruning

2020-10-15 Thread Amit Langote
Robert forwarded me a link to an email sent to -general list, where the reported problem seems to be the same problem that was being discussed here. https://www.postgresql.org/message-id/flat/d0f6d811-8946-eb9f-68e2-1a8a7f80ff21%40a-kretschmer.de Going over the last few emails, it seems that Davi

Re: Parallel Append can break run-time partition pruning

2020-04-28 Thread Robert Haas
On Sat, Apr 25, 2020 at 7:25 AM David Rowley wrote: > I looked at this again and I don't think what I've got is right. Apart from the considerations which you raised, I think there's also a costing issue here. For instance, suppose we have an Append with three children. Two of them have partial p

Re: Parallel Append can break run-time partition pruning

2020-04-25 Thread David Rowley
On Thu, 23 Apr 2020 at 22:37, David Rowley wrote: > For now. I'd still like to get a bit more confidence that the only > noticeable change in the outcome here is that we're now pulling up > sub-Appends in all cases. I've read the code a number of times and > just can't quite see any room for anyth

Re: Parallel Append can break run-time partition pruning

2020-04-24 Thread Amit Langote
On Thu, Apr 23, 2020 at 7:37 PM David Rowley wrote: > On Thu, 23 Apr 2020 at 02:37, Amit Langote wrote: > > Regarding the patch, I had been assuming that the "pa" in > > pa_subpaths_valid stands for "parallel append", so it using the > > variable as is in the new code structure would be misleadin

Re: Parallel Append can break run-time partition pruning

2020-04-23 Thread David Rowley
On Thu, 23 Apr 2020 at 02:37, Amit Langote wrote: > Regarding the patch, I had been assuming that the "pa" in > pa_subpaths_valid stands for "parallel append", so it using the > variable as is in the new code structure would be misleading. Maybe, > parallel_subpaths_valid? I started making anoth

Re: Parallel Append can break run-time partition pruning

2020-04-22 Thread David Rowley
On Thu, 23 Apr 2020 at 14:37, Robert Haas wrote: > > On Wed, Apr 22, 2020 at 7:36 PM David Rowley wrote: > > If there was some reason that a Parallel Append could come out more > > expensive, then maybe we could just create a non-parallel Append using > > the same subpath list and add_partial_pat

Re: Parallel Append can break run-time partition pruning

2020-04-22 Thread Robert Haas
On Wed, Apr 22, 2020 at 7:36 PM David Rowley wrote: > If there was some reason that a Parallel Append could come out more > expensive, then maybe we could just create a non-parallel Append using > the same subpath list and add_partial_path() it. I just don't quite > see how that would ever win tho

Re: Parallel Append can break run-time partition pruning

2020-04-22 Thread David Rowley
On Thu, 23 Apr 2020 at 11:39, Tom Lane wrote: > > David Rowley writes: > > On Thu, 23 Apr 2020 at 11:11, Tom Lane wrote: > >> Well, anytime the parallel startup cost is significant, for starters. > >> But maybe we account for that at some other point, like when building > >> the Gather? > > > Ye

Re: Parallel Append can break run-time partition pruning

2020-04-22 Thread Tom Lane
David Rowley writes: > On Thu, 23 Apr 2020 at 11:11, Tom Lane wrote: >> Well, anytime the parallel startup cost is significant, for starters. >> But maybe we account for that at some other point, like when building >> the Gather? > Yeah. There's no mention of parallel_setup_cost or parallel_tupl

Re: Parallel Append can break run-time partition pruning

2020-04-22 Thread David Rowley
On Thu, 23 Apr 2020 at 11:11, Tom Lane wrote: > > David Rowley writes: > > Given the same set of paths, when would a non-parallel append be > > cheaper than a parallel one? > > Well, anytime the parallel startup cost is significant, for starters. > But maybe we account for that at some other poin

Re: Parallel Append can break run-time partition pruning

2020-04-22 Thread Tom Lane
David Rowley writes: > Given the same set of paths, when would a non-parallel append be > cheaper than a parallel one? Well, anytime the parallel startup cost is significant, for starters. But maybe we account for that at some other point, like when building the Gather? r

Re: Parallel Append can break run-time partition pruning

2020-04-22 Thread David Rowley
On Thu, 23 Apr 2020 at 02:37, Amit Langote wrote: > One thing I remain concerned about is that it appears like we are no > longer leaving the choice between parallel and non-parallel Append to > the cost machinery which is currently the case. AFAICS with patched, > as long as parallel Append is e

Re: Parallel Append can break run-time partition pruning

2020-04-22 Thread Amit Langote
On Wed, Apr 22, 2020 at 12:22 PM David Rowley wrote: > On Tue, 21 Apr 2020 at 15:03, David Rowley wrote: > > I wonder if the fix should be more something along the lines of trying > > to merge things do we only generate a single partial path. That way > > we wouldn't be at the mercy of the logic

Re: Parallel Append can break run-time partition pruning

2020-04-21 Thread David Rowley
On Tue, 21 Apr 2020 at 15:03, David Rowley wrote: > I wonder if the fix should be more something along the lines of trying > to merge things do we only generate a single partial path. That way > we wouldn't be at the mercy of the logic in add_partial_path() to > accept or reject the path based on

Re: Parallel Append can break run-time partition pruning

2020-04-20 Thread Amit Langote
Hi David, On Tue, Apr 21, 2020 at 12:03 PM David Rowley wrote: > On Fri, 17 Apr 2020 at 19:08, Amit Langote wrote: > > On Wed, Apr 15, 2020 at 4:18 PM David Rowley wrote: > > > I'm a bit divided on what the correct fix is. If I blame Parallel > > > Append for not trying hard enough to pull up

Re: Parallel Append can break run-time partition pruning

2020-04-20 Thread David Rowley
On Fri, 17 Apr 2020 at 19:08, Amit Langote wrote: > > On Wed, Apr 15, 2020 at 4:18 PM David Rowley wrote: > > I'm a bit divided on what the correct fix is. If I blame Parallel > > Append for not trying hard enough to pull up the lower Append in > > accumulate_append_subpath(), then clearly the p

Re: Parallel Append can break run-time partition pruning

2020-04-19 Thread David Rowley
On Fri, 17 Apr 2020 at 19:08, Amit Langote wrote: > While looking at this, I observed that the PartitionPruneInfo of the > top-level Append (the one that later gets thrown out) contains bogus > information: > >{PARTITIONPRUNEINFO >:prune_infos (( > {PARTITIONEDRELPRUNEINFO > :r

Re: Parallel Append can break run-time partition pruning

2020-04-17 Thread Amit Langote
On Wed, Apr 15, 2020 at 4:18 PM David Rowley wrote: > I'm a bit divided on what the correct fix is. If I blame Parallel > Append for not trying hard enough to pull up the lower Append in > accumulate_append_subpath(), then clearly the parallel append code is > to blame. I spent some time trying

Parallel Append can break run-time partition pruning

2020-04-15 Thread David Rowley
I had a report from the wilds that run-time partition pruning was not working in certain cases. After some investigation and obtaining the mockup of the actual case, I discovered that the problem was down to accumulate_append_subpath() hitting the case where it does not pullup a Parallel Append wh