Re: Ordered Partitioned Table Scans

2019-04-07 Thread Amit Langote
On 2019/04/06 18:06, Julien Rouhaud wrote: > On Sat, Apr 6, 2019 at 2:45 AM David Rowley > wrote: >> >> On Sat, 6 Apr 2019 at 12:26, Tom Lane wrote: >>> Pushed with some hacking, mostly trying to improve the comments. >> >> Great! Many thanks for improving those and pushing it. >> >> Many thanks

Re: Ordered Partitioned Table Scans

2019-04-06 Thread Julien Rouhaud
On Sat, Apr 6, 2019 at 2:45 AM David Rowley wrote: > > On Sat, 6 Apr 2019 at 12:26, Tom Lane wrote: > > Pushed with some hacking, mostly trying to improve the comments. > > Great! Many thanks for improving those and pushing it. > > Many thanks to Julien, Antonin for their detailed reviews on this

Re: Ordered Partitioned Table Scans

2019-04-05 Thread David Rowley
On Sat, 6 Apr 2019 at 12:26, Tom Lane wrote: > Pushed with some hacking, mostly trying to improve the comments. Great! Many thanks for improving those and pushing it. Many thanks to Julien, Antonin for their detailed reviews on this. Thanks Amit for your input on this as well. Much appreciated.

Re: Ordered Partitioned Table Scans

2019-04-05 Thread Tom Lane
David Rowley writes: > On Wed, 3 Apr 2019 at 14:01, Amit Langote > wrote: >> I don't have any more comments. > Great. Many thanks for having a look at this. Going by [1], Julien > also seems pretty happy, so I'm going to change this over to ready for > committer. Pushed with some hacking, mos

Re: Ordered Partitioned Table Scans

2019-04-03 Thread Julien Rouhaud
Hi, On Wed, Apr 3, 2019 at 7:47 AM David Rowley wrote: > > On Wed, 3 Apr 2019 at 14:01, Amit Langote > wrote: > > I don't have any more comments. > > Great. Many thanks for having a look at this. Going by [1], Julien > also seems pretty happy, so I'm going to change this over to ready for > co

Re: Ordered Partitioned Table Scans

2019-04-02 Thread David Rowley
On Wed, 3 Apr 2019 at 14:01, Amit Langote wrote: > I don't have any more comments. Great. Many thanks for having a look at this. Going by [1], Julien also seems pretty happy, so I'm going to change this over to ready for committer. [1] https://www.postgresql.org/message-id/caobau_yptqbfqcp5jyj

Re: Ordered Partitioned Table Scans

2019-04-02 Thread Amit Langote
On 2019/04/03 3:10, David Rowley wrote: > On Wed, 3 Apr 2019 at 01:26, Amit Langote wrote: >> +#include "nodes/pathnodes.h" >> ... >> +partitions_are_ordered(struct RelOptInfo *partrel) >> >> Maybe, "struct" is unnecessary? > > I just left it there so that the signature matched the header file. >

Re: Ordered Partitioned Table Scans

2019-04-02 Thread David Rowley
On Wed, 3 Apr 2019 at 01:26, Amit Langote wrote: > +#include "nodes/pathnodes.h" > ... > +partitions_are_ordered(struct RelOptInfo *partrel) > > Maybe, "struct" is unnecessary? I just left it there so that the signature matched the header file. Looking around for examples I see make_partition_pru

Re: Ordered Partitioned Table Scans

2019-04-02 Thread Amit Langote
Hi David, On Tue, Apr 2, 2019 at 8:49 PM David Rowley wrote: > I ended up rewording the entire thing and working on the header > comment for the function too. I think previously it wasn't that well > defined what "ordered" meant. I added a mention that we expect that > NULLs, if possible must com

Re: Ordered Partitioned Table Scans

2019-04-02 Thread David Rowley
On Tue, 2 Apr 2019 at 14:26, Amit Langote wrote: > How about extending the sentence about when the optimization is disabled > as follows: > > "If we find a Datum in a partition that's greater than one previously > already seen, then values could become out of order and we'd have to > disable the o

Re: Ordered Partitioned Table Scans

2019-04-01 Thread Amit Langote
Hi, On 2019/03/29 7:59, David Rowley wrote: > On Fri, 29 Mar 2019 at 00:00, Amit Langote > wrote: >> >> On 2019/03/28 8:04, David Rowley wrote: >>> If it's *always* scanned last then it's fine for ORDER BY partkey >>> NULLS LAST. If they have ORDER BY partkey NULLS FIRST then we won't >>> match

Re: Ordered Partitioned Table Scans

2019-03-28 Thread David Rowley
On Fri, 29 Mar 2019 at 00:00, Amit Langote wrote: > > On 2019/03/28 8:04, David Rowley wrote: > > If it's *always* scanned last then it's fine for ORDER BY partkey > > NULLS LAST. If they have ORDER BY partkey NULLS FIRST then we won't > > match on the pathkeys. > > Sorry, I had meant to say that

Re: Ordered Partitioned Table Scans

2019-03-28 Thread Amit Langote
Hi David, On 2019/03/28 8:04, David Rowley wrote: > On Wed, 27 Mar 2019 at 21:24, Amit Langote > wrote: >> Noticed a typo. >> >> + * multiple subpaths then we can't make guarantees about the >> + * order tuples in those subpaths, so we must leave the >> >> order of

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Thu, 28 Mar 2019 at 15:40, David Rowley wrote: > Thanks for the review. I've attached a patch that mostly just moved > the code around. Here's one that fixes up the compiler warning from the last one. Thanks CF bot... -- David Rowley http://www.2ndQuadrant.com/ PostgreSQ

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Thu, 28 Mar 2019 at 14:34, Amit Langote wrote: > > On 2019/03/28 7:29, David Rowley wrote: > > On Wed, 27 Mar 2019 at 19:48, Amit Langote > > It does need to know how many partitions the partitioned table has, > > which it gets from partrel->nparts, so yeah, RelOptInfo is probably > > needed. I

Re: Ordered Partitioned Table Scans

2019-03-27 Thread Amit Langote
Hi, On 2019/03/28 7:29, David Rowley wrote: > On Wed, 27 Mar 2019 at 19:48, Amit Langote > wrote: >> Sorry if this was discussed before, but why does this patch add any new >> code to partprune.c? AFAICT, there's no functionality changes to the >> pruning code. > > You're right. It probably sho

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Wed, 27 Mar 2019 at 21:24, Amit Langote wrote: > Noticed a typo. > > + * multiple subpaths then we can't make guarantees about the > + * order tuples in those subpaths, so we must leave the > > order of tuples? I'll fix that. Thanks. > Again, sorry if this was

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Wed, 27 Mar 2019 at 19:48, Amit Langote wrote: > Sorry if this was discussed before, but why does this patch add any new > code to partprune.c? AFAICT, there's no functionality changes to the > pruning code. You're right. It probably shouldn't be there. There's a bit of a lack of a good home

Re: Ordered Partitioned Table Scans

2019-03-27 Thread Amit Langote
On 2019/03/27 15:48, Amit Langote wrote: > Hi David, > > Sorry if this was discussed before, but why does this patch add any new > code to partprune.c? AFAICT, there's no functionality changes to the > pruning code. > > Both > > +bool > +partkey_is_bool_constant_for_query(RelOptInfo *partrel, i

Re: Ordered Partitioned Table Scans

2019-03-26 Thread Amit Langote
Hi David, Sorry if this was discussed before, but why does this patch add any new code to partprune.c? AFAICT, there's no functionality changes to the pruning code. Both +bool +partkey_is_bool_constant_for_query(RelOptInfo *partrel, int partkeycol) and +static bool +matches_boolean_partition_

Re: Ordered Partitioned Table Scans

2019-03-26 Thread David Rowley
Thanks for having another look. On Wed, 27 Mar 2019 at 00:22, Julien Rouhaud wrote: > A few, mostly nitpicking, comments: > > + if (rel->part_scheme != NULL && IS_SIMPLE_REL(rel) && > + partitions_are_ordered(root, rel)) > > shouldn't the test be IS_PARTITIONED_REL(rel) instead of testing

Re: Ordered Partitioned Table Scans

2019-03-26 Thread Julien Rouhaud
On Tue, Mar 26, 2019 at 3:13 AM David Rowley wrote: > > On Tue, 26 Mar 2019 at 09:02, Julien Rouhaud wrote: > > FTR this patch doesn't apply since single child [Merge]Append > > suppression (8edd0e7946) has been pushed. > > Thanks for letting me know. I've attached v14 based on current master.

Re: Ordered Partitioned Table Scans

2019-03-25 Thread David Rowley
On Tue, 26 Mar 2019 at 09:02, Julien Rouhaud wrote: > FTR this patch doesn't apply since single child [Merge]Append > suppression (8edd0e7946) has been pushed. Thanks for letting me know. I've attached v14 based on current master. -- David Rowley http://www.2ndQuadrant.com/

Re: Ordered Partitioned Table Scans

2019-03-25 Thread Julien Rouhaud
On Sun, Mar 24, 2019 at 11:06 AM David Rowley wrote: > > On Sat, 23 Mar 2019 at 19:42, Tom Lane wrote: > > > > David Rowley writes: > > > On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: > > >> BTW, another thing we could possibly do to answer this objection is to > > >> give the ordered-Append no

Re: Ordered Partitioned Table Scans

2019-03-24 Thread David Rowley
On Sat, 23 Mar 2019 at 19:42, Tom Lane wrote: > > David Rowley writes: > > On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: > >> BTW, another thing we could possibly do to answer this objection is to > >> give the ordered-Append node an artificially pessimistic startup cost, > >> such as the sum or

Re: Ordered Partitioned Table Scans

2019-03-24 Thread David Rowley
On Sun, 24 Mar 2019 at 05:16, Julien Rouhaud wrote: > ISTM that a query like > SELECT * FROM nested ORDER BY 1, 2; > could simply append all the partitions in the right order (or generate > a tree of ordered appends), but: > > QUERY PLAN > --

Re: Ordered Partitioned Table Scans

2019-03-23 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 3:01 PM David Rowley wrote: > > On Thu, 20 Dec 2018 at 01:58, Julien Rouhaud wrote: > > > > The multi-level partitioning case is another > > thing that would need to be handled for instance (and that's the main > > reason I couldn't submit a new patch when I was working on

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
David Rowley writes: > On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: >> BTW, another thing we could possibly do to answer this objection is to >> give the ordered-Append node an artificially pessimistic startup cost, >> such as the sum or the max of its children's startup costs. That's >> pretty

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: > BTW, another thing we could possibly do to answer this objection is to > give the ordered-Append node an artificially pessimistic startup cost, > such as the sum or the max of its children's startup costs. That's > pretty ugly and unprincipled, but

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 23 Mar 2019 at 04:56, Tom Lane wrote: > > David Rowley writes: > > Append has the additional > > saving of not having to determine to perform a sort on the top row > > from each subpath. > > Uh, what? sorted-Append and MergeAppend would need pre-sorts on > exactly the same set of childre

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Julien Rouhaud
On Fri, Mar 22, 2019 at 7:19 PM Robert Haas wrote: > > On Fri, Mar 22, 2019 at 12:40 PM Tom Lane wrote: > > Robert Haas writes: > > > On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: > > >> In cases where, say, the first child requires no sort but also doesn't > > >> emit very many rows, while

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Robert Haas
On Fri, Mar 22, 2019 at 12:40 PM Tom Lane wrote: > Robert Haas writes: > > On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: > >> In cases where, say, the first child requires no sort but also doesn't > >> emit very many rows, while the second child requires an expensive sort, > >> the planner wi

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
Robert Haas writes: > On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: >> In cases where, say, the first child requires no sort but also doesn't >> emit very many rows, while the second child requires an expensive sort, >> the planner will have a ridiculously optimistic opinion of the cost of >>

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Robert Haas
On Fri, Mar 22, 2019 at 12:21 PM Tom Lane wrote: > Once again: this objection is not a "death sentence for this patch". > I simply wish to suppress the option to generate an ordered Append > when some of the inputs would require an added sort step. As long > as we have pre-ordered paths for all c

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
Robert Haas writes: > On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: >> In cases where, say, the first child requires no sort but also doesn't >> emit very many rows, while the second child requires an expensive sort, >> the planner will have a ridiculously optimistic opinion of the cost of >>

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Robert Haas
On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: > In cases where, say, the first child requires no sort but also doesn't > emit very many rows, while the second child requires an expensive sort, > the planner will have a ridiculously optimistic opinion of the cost of > fetching slightly more rows

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
David Rowley writes: > Thanks for explaining. I see where you're coming from now. I think > this point would carry more weight if using the Append instead of the > MergeAppend were worse in some cases as we could end up using an > inferior plan accidentally. However, that's not the case. The Ap

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Simon Riggs
On Fri, 22 Mar 2019 at 11:39, Tom Lane wrote: > Simon Riggs writes: > > I agree that the issue of mixing sorts at various points will make > nonsense > > of the startup cost/total cost results. > > Right. > > > I don't see LIMIT costing being broken as a reason to restrict this > > optimization.

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
Simon Riggs writes: > I agree that the issue of mixing sorts at various points will make nonsense > of the startup cost/total cost results. Right. > I don't see LIMIT costing being broken as a reason to restrict this > optimization. I would ask that we allow improvements to the important use > c

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 23 Mar 2019 at 04:12, Tom Lane wrote: > The reason why I'm skeptical about LIMIT with a plan of the form > append-some-sorts-together is that there are going to be large > discontinuities in the cost-vs-number-of-rows-returned graph, > ie, every time you finish one child plan and start the

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Simon Riggs
On Fri, 22 Mar 2019 at 11:12, Tom Lane wrote: > David Rowley writes: > > On Sat, 9 Mar 2019 at 10:52, Tom Lane wrote: > >>> This can be a huge win for queries of the form "ORDER BY partkey LIMIT > >>> x". Even if the first subpath(s) aren't natively ordered, not all of > >>> the sorts should a

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
David Rowley writes: > On Sat, 9 Mar 2019 at 10:52, Tom Lane wrote: >>> This can be a huge win for queries of the form "ORDER BY partkey LIMIT >>> x". Even if the first subpath(s) aren't natively ordered, not all of >>> the sorts should actually be performed. >> [ shrug... ] We've got no realis

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 9 Mar 2019 at 10:52, Tom Lane wrote: > > Julien Rouhaud writes: > > On Fri, Mar 8, 2019 at 9:15 PM Tom Lane wrote: > >> I think you should remove all that > >> and restrict this optimization to the case where all the subpaths are > >> natively ordered --- if we have to insert Sorts, it's

Re: Ordered Partitioned Table Scans

2019-03-15 Thread David Rowley
On Sat, 16 Mar 2019 at 04:22, David Rowley wrote: > I've attached an updated patch which fixes the conflict with 0a9d7e1f6d8 ... and here's the one that I should have sent. (renamed to v12 to prevent confusion) -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Developme

Re: Ordered Partitioned Table Scans

2019-03-15 Thread Robert Haas
On Fri, Mar 8, 2019 at 3:15 PM Tom Lane wrote: > I took a quick look through this and I'm not very happy with it. > It seems to me that the premise ought to be "just use an Append > if we can prove the output would be ordered anyway", but that's not > what we actually have here: instead you're add

Re: Ordered Partitioned Table Scans

2019-03-15 Thread David Rowley
I've attached an updated patch which fixes the conflict with 0a9d7e1f6d8 -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services v11-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-.patch Description: Binary data

Re: Ordered Partitioned Table Scans

2019-03-08 Thread David Rowley
On Sat, 9 Mar 2019 at 09:14, Tom Lane wrote: > I took a quick look through this and I'm not very happy with it. > It seems to me that the premise ought to be "just use an Append > if we can prove the output would be ordered anyway", but that's not > what we actually have here: instead you're addin

Re: Ordered Partitioned Table Scans

2019-03-08 Thread David Rowley
On Sat, 9 Mar 2019 at 10:52, Tom Lane wrote: > > Julien Rouhaud writes: > > On Fri, Mar 8, 2019 at 9:15 PM Tom Lane wrote: > >> I think you should remove all that > >> and restrict this optimization to the case where all the subpaths are > >> natively ordered --- if we have to insert Sorts, it's

Re: Ordered Partitioned Table Scans

2019-03-08 Thread Tom Lane
Julien Rouhaud writes: > On Fri, Mar 8, 2019 at 9:15 PM Tom Lane wrote: >> I think you should remove all that >> and restrict this optimization to the case where all the subpaths are >> natively ordered --- if we have to insert Sorts, it's hardly going to move >> the needle to worry about simplif

Re: Ordered Partitioned Table Scans

2019-03-08 Thread Julien Rouhaud
On Fri, Mar 8, 2019 at 9:15 PM Tom Lane wrote: > > David Rowley writes: > > [ v9-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-f.patch ] > > I think you should remove all that > and restrict this optimization to the case where all the subpaths are > natively ordered --- if we have to inser

Re: Ordered Partitioned Table Scans

2019-03-08 Thread Tom Lane
David Rowley writes: > [ v9-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-f.patch ] I took a quick look through this and I'm not very happy with it. It seems to me that the premise ought to be "just use an Append if we can prove the output would be ordered anyway", but that's not what we a

Re: Ordered Partitioned Table Scans

2019-03-05 Thread David Rowley
On Wed, 6 Mar 2019 at 07:17, Robert Haas wrote: > > On Wed, Dec 19, 2018 at 5:08 PM David Rowley > wrote: > > With my idea for using live_parts, we'll process the partitions > > looking for interleaved values on each query, after pruning takes > > place. In this case, we'll see the partitions are

Re: Ordered Partitioned Table Scans

2019-03-05 Thread Robert Haas
On Wed, Dec 19, 2018 at 5:08 PM David Rowley wrote: > With my idea for using live_parts, we'll process the partitions > looking for interleaved values on each query, after pruning takes > place. In this case, we'll see the partitions are naturally ordered. I > don't really foresee any issues with

Re: Ordered Partitioned Table Scans

2019-03-04 Thread David Rowley
Thanks a lot for taking the time to look at this. On Tue, 5 Mar 2019 at 03:03, Antonin Houska wrote: > As for the latest version (v8-0001-...) I've only caught a small typo: "When > the first subpath needs sorted ...". It was probably meant "... needs sort > ...". That was a sort of short way of

Re: Ordered Partitioned Table Scans

2019-03-04 Thread Antonin Houska
David Rowley wrote: > On Mon, 5 Nov 2018 at 10:46, David Rowley > wrote: > > On 1 November 2018 at 22:05, Antonin Houska wrote: > > > I think these conditions are too restrictive: > > > > > > /* > > > * Determine if these pathkeys match the partition order, or > > > reverse >

Re: Ordered Partitioned Table Scans

2019-02-23 Thread David Rowley
On Thu, 31 Jan 2019 at 16:29, David Rowley wrote: > I've attached a rebased patch which fixes up the recent conflicts. No > other changes were made. Rebased version due to a new call to create_append_path() added in ab5fcf2b0. No other changes made. -- David Rowley http://www

Re: Ordered Partitioned Table Scans

2019-02-03 Thread Michael Paquier
On Thu, Jan 31, 2019 at 04:29:56PM +1300, David Rowley wrote: > I've attached a rebased patch which fixes up the recent conflicts. No > other changes were made. Moved to next CF, waiting for review. -- Michael signature.asc Description: PGP signature

Re: Ordered Partitioned Table Scans

2019-01-30 Thread David Rowley
I've attached a rebased patch which fixes up the recent conflicts. No other changes were made. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services v7-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-f.patch Description: B

Re: Ordered Partitioned Table Scans

2019-01-06 Thread Julien Rouhaud
On Sun, Jan 6, 2019 at 4:24 AM David Rowley wrote: > > On Thu, 20 Dec 2018 at 18:20, Julien Rouhaud wrote: > > > > > > No, what I'm proposing is to store if the partitions are naturally > > ordered or not, *and* recheck after pruning if that property could > > have changed (so if some partitions

Re: Ordered Partitioned Table Scans

2019-01-05 Thread David Rowley
On Thu, 20 Dec 2018 at 18:20, Julien Rouhaud wrote: > > On Wed, Dec 19, 2018 at 11:08 PM David Rowley > wrote: > > create table listp (a int) partition by list (a); > > create table listp12 partition of listp for values in(1,2); > > create table listp03 partition of listp for vlaues in(0,3); > >

Re: Ordered Partitioned Table Scans

2018-12-19 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 11:08 PM David Rowley wrote: > > On Thu, 20 Dec 2018 at 09:48, Julien Rouhaud wrote: > > On Wed, Dec 19, 2018 at 3:01 PM David Rowley > > wrote: > > > If so, I'd question why the default partition > > > is so special? Pruning of any of the other partitions could turn a >

Re: Ordered Partitioned Table Scans

2018-12-19 Thread David Rowley
On Thu, 20 Dec 2018 at 09:48, Julien Rouhaud wrote: > On Wed, Dec 19, 2018 at 3:01 PM David Rowley > wrote: > > If so, I'd question why the default partition > > is so special? Pruning of any of the other partitions could turn a > > naturally unordered LIST partitioned table into a naturally ord

Re: Ordered Partitioned Table Scans

2018-12-19 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 3:01 PM David Rowley wrote: > > On Thu, 20 Dec 2018 at 01:58, Julien Rouhaud wrote: > > I'm objecting to processing for all partitions, but processing for > just non-pruned partitions seems fine to me. If there are 10k > partitions and we pruned none of them, then planning

Re: Ordered Partitioned Table Scans

2018-12-19 Thread David Rowley
On Thu, 20 Dec 2018 at 01:58, Julien Rouhaud wrote: > I don't see why we should drop this flag. If we know that the > partitions are naturally ordered, they'll still be ordered after some > partitions have been prune, so we can skip later checks if we already > have the information. The only rem

Re: Ordered Partitioned Table Scans

2018-12-19 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 1:18 PM David Rowley wrote: > > On Wed, 19 Dec 2018 at 23:25, Julien Rouhaud wrote: > > > > I see. But since for now the optimisation will only be done > > considering all partitions, I still think that it's better to store a > > bool flag in the PartitionDesc to describe

Re: Ordered Partitioned Table Scans

2018-12-19 Thread David Rowley
On Wed, 19 Dec 2018 at 23:25, Julien Rouhaud wrote: > On Wed, Dec 19, 2018 at 10:51 AM David Rowley > wrote: > > The reason I'm keen to leave this alone today is that determining > > which partitions are pruned requires looking at each partition's > > RelOptInfo and checking if it's marked as a d

Re: Ordered Partitioned Table Scans

2018-12-19 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 10:51 AM David Rowley wrote: > > On Wed, 19 Dec 2018 at 20:40, Julien Rouhaud wrote: > > > If I understand correctly, the new behavior is controlled by > > partitions_are_ordered(), but it only checks for declared partitions, > > not partitions that survived pruning. Did

Re: Ordered Partitioned Table Scans

2018-12-19 Thread David Rowley
On Wed, 19 Dec 2018 at 20:40, Julien Rouhaud wrote: > I started to look at v5. Thanks for giving this a look over. > If I understand correctly, the new behavior is controlled by > partitions_are_ordered(), but it only checks for declared partitions, > not partitions that survived pruning. Did I

Re: Ordered Partitioned Table Scans

2018-12-18 Thread Julien Rouhaud
Hi, On Thu, Nov 22, 2018 at 11:27 AM David Rowley wrote: > > On Mon, 5 Nov 2018 at 10:46, David Rowley > wrote: > > On 1 November 2018 at 22:05, Antonin Houska wrote: > > > I think these conditions are too restrictive: > > > > > > /* > > > * Determine if these pathkeys match t

Re: Ordered Partitioned Table Scans

2018-11-22 Thread David Rowley
On Mon, 5 Nov 2018 at 10:46, David Rowley wrote: > On 1 November 2018 at 22:05, Antonin Houska wrote: > > I think these conditions are too restrictive: > > > > /* > > * Determine if these pathkeys match the partition order, or reverse > > * partition order. It can't mat

Re: Ordered Partitioned Table Scans

2018-11-04 Thread David Rowley
On 1 November 2018 at 22:05, Antonin Houska wrote: > I think these conditions are too restrictive: > > /* > * Determine if these pathkeys match the partition order, or reverse > * partition order. It can't match both, so only go to the trouble of > * checking th

Re: Ordered Partitioned Table Scans

2018-11-01 Thread Antonin Houska
David Rowley wrote: > On 1 November 2018 at 04:01, Antonin Houska wrote: > > * As for the logic, I found generate_mergeappend_paths() to be the most > > interesting part: > > > > Imagine table partitioned by "i", so "partition_pathkeys" is {i}. > > > > partition 1: > > > > i | j > > --+-- > >

Re: Ordered Partitioned Table Scans

2018-10-31 Thread David Rowley
On 1 November 2018 at 04:01, Antonin Houska wrote: > * As for the logic, I found generate_mergeappend_paths() to be the most > interesting part: > > Imagine table partitioned by "i", so "partition_pathkeys" is {i}. > > partition 1: > > i | j > --+-- > 0 | 0 > 1 | 1 > 0 | 1 > 1 | 0 > > partition

Re: Ordered Partitioned Table Scans

2018-10-31 Thread Antonin Houska
David Rowley wrote: > On 31 October 2018 at 13:05, David Rowley > wrote: > >>> On 28 October 2018 at 03:49, Julien Rouhaud wrote: > >> I've registered as a reviewer. I still didn't have a deep look at > >> the patch yet, but thanks a lot for working on it! > > > > Thanks for signing up to re

Re: Ordered Partitioned Table Scans

2018-10-30 Thread David Rowley
On 31 October 2018 at 13:05, David Rowley wrote: >>> On 28 October 2018 at 03:49, Julien Rouhaud wrote: >> I've registered as a reviewer. I still didn't have a deep look at >> the patch yet, but thanks a lot for working on it! > > Thanks for signing up to review. I need to send another revisio

Re: Ordered Partitioned Table Scans

2018-10-30 Thread David Rowley
On 31 October 2018 at 12:24, Julien Rouhaud wrote: > On Mon, Oct 29, 2018 at 1:44 AM David Rowley > wrote: >> >> On 28 October 2018 at 03:49, Julien Rouhaud wrote: >> > I just had a look at your patch. I see that you implemented only a >> > subset of the possible optimizations (only the case fo

Re: Ordered Partitioned Table Scans

2018-10-30 Thread Julien Rouhaud
On Mon, Oct 29, 2018 at 1:44 AM David Rowley wrote: > > On 28 October 2018 at 03:49, Julien Rouhaud wrote: > > I just had a look at your patch. I see that you implemented only a > > subset of the possible optimizations (only the case for range > > partitionoing without subpartitions). This has

Re: Ordered Partitioned Table Scans

2018-10-29 Thread David Rowley
On 29 October 2018 at 13:44, David Rowley wrote: > v2 of the patch is attached. I've not had time yet to give it a > throughout post write review, but on first look it seems okay. Added to the November 'fest. https://commitfest.postgresql.org/20/1850/ -- David Rowley http://

Re: Ordered Partitioned Table Scans

2018-10-28 Thread David Rowley
Thanks for looking at this. On 28 October 2018 at 03:49, Julien Rouhaud wrote: > I just had a look at your patch. I see that you implemented only a > subset of the possible optimizations (only the case for range > partitionoing without subpartitions). This has been previously > discussed, but w

Re: Ordered Partitioned Table Scans

2018-10-27 Thread Julien Rouhaud
Hi, On Fri, Oct 26, 2018 at 1:01 PM Julien Rouhaud wrote: > On Fri, Oct 26, 2018 at 6:40 AM David Rowley > wrote: > > > > On 26 October 2018 at 16:52, Amit Langote > > wrote: > > > I recall Ronan Dunklau and Julien Rouhaud had proposed a patch for this > > > last year, but the partitioning-rel

Re: Ordered Partitioned Table Scans

2018-10-26 Thread Julien Rouhaud
Hi, On Fri, Oct 26, 2018 at 6:40 AM David Rowley wrote: > > On 26 October 2018 at 16:52, Amit Langote > wrote: > > I recall Ronan Dunklau and Julien Rouhaud had proposed a patch for this > > last year, but the partitioning-related planning code hadn't advanced then > > as much as it has today,

Re: Ordered Partitioned Table Scans

2018-10-25 Thread David Rowley
On 26 October 2018 at 16:52, Amit Langote wrote: > I recall Ronan Dunklau and Julien Rouhaud had proposed a patch for this > last year, but the partitioning-related planning code hadn't advanced then > as much as it has today, so they sort of postponed working on it. > Eventually their patch was r

Re: Ordered Partitioned Table Scans

2018-10-25 Thread Amit Langote
On 2018/10/26 11:50, David Rowley wrote: > RANGE partitioning of time-series data is quite a common range to use > partitioning, and such tables tend to grow fairly large. I thought > since we always store RANGE partitioned tables in the PartitionDesc in > ascending range order that it might be us

Ordered Partitioned Table Scans

2018-10-25 Thread David Rowley
RANGE partitioning of time-series data is quite a common range to use partitioning, and such tables tend to grow fairly large. I thought since we always store RANGE partitioned tables in the PartitionDesc in ascending range order that it might be useful to make use of this and when the required pa