Re: [HACKERS] Allowing join removals for more join types

2014-07-16 Thread Tom Lane
David Rowley writes: > On Wed, Jul 16, 2014 at 1:17 PM, Tom Lane wrote: >> Notably, I felt that pathnode.c was a pretty questionable place to be >> exporting distinctness-proof logic from, and after some reflection decided >> to move those functions to analyzejoins.c; that's certainly a better pl

Re: [HACKERS] Allowing join removals for more join types

2014-07-16 Thread David Rowley
On Wed, Jul 16, 2014 at 1:17 PM, Tom Lane wrote: > David Rowley writes: > > I've attached an updated patch which puts in some fast path code for > > subquery type joins. I'm really not too sure on a good name for this > > function. I've ended up with query_supports_distinctness() which I'm not >

Re: [HACKERS] Allowing join removals for more join types

2014-07-15 Thread Tom Lane
David Rowley writes: > I've attached an updated patch which puts in some fast path code for > subquery type joins. I'm really not too sure on a good name for this > function. I've ended up with query_supports_distinctness() which I'm not > that keen on, but I didn't manage to come up with anything

Re: [HACKERS] Allowing join removals for more join types

2014-07-11 Thread David Rowley
On Wed, Jul 9, 2014 at 12:59 PM, Tom Lane wrote: > David Rowley writes: > > On 9 July 2014 09:27, Tom Lane wrote: > >> On review it looks like analyzejoins.c would possibly benefit from an > >> earlier fast-path check as well. > > > Do you mean for non-subqueries? There already is a check to se

Re: [HACKERS] Allowing join removals for more join types

2014-07-08 Thread Tom Lane
David Rowley writes: > On 9 July 2014 09:27, Tom Lane wrote: >> On review it looks like analyzejoins.c would possibly benefit from an >> earlier fast-path check as well. > Do you mean for non-subqueries? There already is a check to see if the > relation has no indexes. Oh, sorry, that was a typ

Re: [HACKERS] Allowing join removals for more join types

2014-07-08 Thread David Rowley
On 9 July 2014 09:27, Tom Lane wrote: > David Rowley writes: > > On Tue, Jul 8, 2014 at 4:28 AM, Tom Lane wrote: > >> I'm a bit skeptical as to whether testing for that case is actually > worth > >> any extra complexity. Do you have a compelling use-case? But anyway, > >> if we do want to all

Re: [HACKERS] Allowing join removals for more join types

2014-07-08 Thread Tom Lane
David Rowley writes: > On Tue, Jul 8, 2014 at 4:28 AM, Tom Lane wrote: >> I'm a bit skeptical as to whether testing for that case is actually worth >> any extra complexity. Do you have a compelling use-case? But anyway, >> if we do want to allow it, why does it take any more than adding a check

Re: [HACKERS] Allowing join removals for more join types

2014-07-08 Thread David Rowley
On Tue, Jul 8, 2014 at 4:28 AM, Tom Lane wrote: > David Rowley writes: > > On Mon, Jul 7, 2014 at 4:15 AM, Tom Lane wrote: > >> I poked around to see if we didn't have some code already for that, and > >> soon found that not only do we have such code > (equality_ops_are_compatible) > >> but act

Re: [HACKERS] Allowing join removals for more join types

2014-07-07 Thread Tom Lane
David Rowley writes: > On Mon, Jul 7, 2014 at 4:15 AM, Tom Lane wrote: >> I poked around to see if we didn't have some code already for that, and >> soon found that not only do we have such code (equality_ops_are_compatible) >> but actually almost this entire patch duplicates logic that already e

Re: [HACKERS] Allowing join removals for more join types

2014-07-07 Thread David Rowley
On Mon, Jul 7, 2014 at 4:15 AM, Tom Lane wrote: > David Rowley writes: > > On 6 July 2014 03:20, Tom Lane wrote: > >> Just to note that I've started looking at this, and I've detected a > rather > >> significant omission: there's no check that the join operator has > anything > >> to do with th

Re: [HACKERS] Allowing join removals for more join types

2014-07-06 Thread Tom Lane
David Rowley writes: > On 6 July 2014 03:20, Tom Lane wrote: >> Just to note that I've started looking at this, and I've detected a rather >> significant omission: there's no check that the join operator has anything >> to do with the subquery's grouping operator. > hmm, good point. If I underst

Re: [HACKERS] Allowing join removals for more join types

2014-07-05 Thread David Rowley
On 6 July 2014 03:20, Tom Lane wrote: > David Rowley writes: > > Attached is a delta patch between version 1.2 and 1.3, and also a > > completely updated patch. > > Just to note that I've started looking at this, and I've detected a rather > significant omission: there's no check that the join o

Re: [HACKERS] Allowing join removals for more join types

2014-07-05 Thread Tom Lane
David Rowley writes: > Attached is a delta patch between version 1.2 and 1.3, and also a > completely updated patch. Just to note that I've started looking at this, and I've detected a rather significant omission: there's no check that the join operator has anything to do with the subquery's grou

Re: [HACKERS] Allowing join removals for more join types

2014-06-26 Thread Simon Riggs
On 26 June 2014 10:01, David Rowley wrote: >> Did you comment on the transitive closure question? Should we add a >> test for that, whether or not it works yet? >> > > In my previous email. > > I could change the the following to use c.id in the targetlist and group by > clause, but I'm not reall

Re: [HACKERS] Allowing join removals for more join types

2014-06-26 Thread David Rowley
On Wed, Jun 25, 2014 at 10:03 AM, Simon Riggs wrote: > On 23 June 2014 12:06, David Rowley wrote: > > >> It's not clear to me where you get the term "sortclause" from. This is > >> either the groupclause or distinctclause, but in the test cases you > >> provide this shows this has nothing at all

Re: [HACKERS] Allowing join removals for more join types

2014-06-26 Thread David Rowley
On Sun, Jun 22, 2014 at 11:51 PM, Simon Riggs wrote: > On 17 June 2014 11:04, David Rowley wrote: > > On Wed, Jun 4, 2014 at 12:50 AM, Noah Misch wrote: > >> > >> As a point of procedure, I recommend separating the semijoin support > into > >> its > >> own patch. Your patch is already not smal

Re: [HACKERS] Allowing join removals for more join types

2014-06-24 Thread Simon Riggs
On 24 June 2014 23:48, Tom Lane wrote: > Simon Riggs writes: >> Other than that it looks pretty good to commit, so I'll wait a week >> for other objections then commit. > > I'd like to review this before it goes in. I've been waiting for it to > get marked "ready for committer" though. I'll lea

Re: [HACKERS] Allowing join removals for more join types

2014-06-24 Thread Tom Lane
Simon Riggs writes: > Other than that it looks pretty good to commit, so I'll wait a week > for other objections then commit. I'd like to review this before it goes in. I've been waiting for it to get marked "ready for committer" though. regards, tom lane -- Sent via

Re: [HACKERS] Allowing join removals for more join types

2014-06-24 Thread Simon Riggs
On 23 June 2014 12:06, David Rowley wrote: >> It's not clear to me where you get the term "sortclause" from. This is >> either the groupclause or distinctclause, but in the test cases you >> provide this shows this has nothing at all to do with sorting since >> there is neither an order by or a s

Re: [HACKERS] Allowing join removals for more join types

2014-06-22 Thread Simon Riggs
On 22 June 2014 12:51, Simon Riggs wrote: > Looks good on initial look. Tests 2 and 3 seem to test the same thing. There are no tests which have multiple column clauselist/sortlists, nor tests for cases where the clauselist is a superset of the sortlist. Test comments should refer to "join rem

Re: [HACKERS] Allowing join removals for more join types

2014-06-22 Thread Simon Riggs
On 17 June 2014 11:04, David Rowley wrote: > On Wed, Jun 4, 2014 at 12:50 AM, Noah Misch wrote: >> >> As a point of procedure, I recommend separating the semijoin support into >> its >> own patch. Your patch is already not small; delaying non-essential parts >> will >> make the essential parts m

Re: [HACKERS] Allowing join removals for more join types

2014-06-17 Thread David Rowley
On Wed, Jun 4, 2014 at 12:50 AM, Noah Misch wrote: > As a point of procedure, I recommend separating the semijoin support into > its > own patch. Your patch is already not small; delaying non-essential parts > will > make the essential parts more accessible to reviewers. > > In the attached patc

Re: [HACKERS] Allowing join removals for more join types

2014-06-09 Thread Robert Haas
On Mon, Jun 2, 2014 at 11:42 AM, Tom Lane wrote: > Stephen Frost writes: >> * Tom Lane (t...@sss.pgh.pa.us) wrote: >>> TBH I think that trying to do anything at all for inner joins is probably >>> a bad idea. The cases where the optimization could succeed are so narrow >>> that it's unlikely to

Re: [HACKERS] Allowing join removals for more join types

2014-06-06 Thread David Rowley
On Fri, Jun 6, 2014 at 11:44 AM, Tom Lane wrote: > Noah Misch writes: > > On Thu, Jun 05, 2014 at 02:12:33AM +0200, Andres Freund wrote: > >> A bit more crazy, but how about trying trying to plan joins with a added > >> one-time qual that checks the size of the deferred trigger queue? Then > >>

Re: [HACKERS] Allowing join removals for more join types

2014-06-05 Thread Noah Misch
On Thu, Jun 05, 2014 at 07:44:31PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Thu, Jun 05, 2014 at 02:12:33AM +0200, Andres Freund wrote: > >> A bit more crazy, but how about trying trying to plan joins with a added > >> one-time qual that checks the size of the deferred trigger queue? The

Re: [HACKERS] Allowing join removals for more join types

2014-06-05 Thread Tom Lane
Noah Misch writes: > On Thu, Jun 05, 2014 at 02:12:33AM +0200, Andres Freund wrote: >> A bit more crazy, but how about trying trying to plan joins with a added >> one-time qual that checks the size of the deferred trigger queue? Then >> we wouldn't even need special case plans. > That, too, sound

Re: [HACKERS] Allowing join removals for more join types

2014-06-05 Thread Noah Misch
On Thu, Jun 05, 2014 at 02:12:33AM +0200, Andres Freund wrote: > On 2014-06-04 20:04:07 -0400, Noah Misch wrote: > > On Wed, Jun 04, 2014 at 10:14:42AM -0400, Tom Lane wrote: > > > It's possible that we could apply the optimization only to queries that > > > have been issued directly by a client, b

Re: [HACKERS] Allowing join removals for more join types

2014-06-04 Thread Andres Freund
On 2014-06-04 20:04:07 -0400, Noah Misch wrote: > On Wed, Jun 04, 2014 at 10:14:42AM -0400, Tom Lane wrote: > > It's possible that we could apply the optimization only to queries that > > have been issued directly by a client, but that seems rather ugly and > > surprise-filled. > > ... such as thi

Re: [HACKERS] Allowing join removals for more join types

2014-06-04 Thread Noah Misch
On Wed, Jun 04, 2014 at 10:14:42AM -0400, Tom Lane wrote: > David Rowley writes: > > On Wed, Jun 4, 2014 at 11:50 AM, Noah Misch wrote: > >> When a snapshot can see modifications that queued referential integrity > >> triggers for some FK constraint, that constraint is not guaranteed to hold > >>

Re: [HACKERS] Allowing join removals for more join types

2014-06-04 Thread Tom Lane
David Rowley writes: > On Wed, Jun 4, 2014 at 11:50 AM, Noah Misch wrote: >> When a snapshot can see modifications that queued referential integrity >> triggers for some FK constraint, that constraint is not guaranteed to hold >> within the snapshot until those triggers have fired. > I remember

Re: [HACKERS] Allowing join removals for more join types

2014-06-04 Thread David Rowley
On Wed, Jun 4, 2014 at 11:50 AM, Noah Misch wrote: > On Wed, May 28, 2014 at 08:39:32PM +1200, David Rowley wrote: > > The attached patch allows join removals for both sub queries with left > > joins and also semi joins where a foreign key can prove the existence of > > the record. > > When a sna

Re: [HACKERS] Allowing join removals for more join types

2014-06-03 Thread Noah Misch
On Wed, May 28, 2014 at 08:39:32PM +1200, David Rowley wrote: > The attached patch allows join removals for both sub queries with left > joins and also semi joins where a foreign key can prove the existence of > the record. When a snapshot can see modifications that queued referential integrity tr

Re: [HACKERS] Allowing join removals for more join types

2014-06-02 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> TBH I think that trying to do anything at all for inner joins is probably >> a bad idea. The cases where the optimization could succeed are so narrow >> that it's unlikely to be worth adding cycles to every query to check. > I agr

Re: [HACKERS] Allowing join removals for more join types

2014-06-02 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > David Rowley writes: > > I'm not quite there with inner joins yet. I'm still getting my head around > > just where the join quals are actually stored. > > TBH I think that trying to do anything at all for inner joins is probably > a bad idea. The cases wh

Re: [HACKERS] Allowing join removals for more join types

2014-06-02 Thread Tom Lane
David Rowley writes: > I'm not quite there with inner joins yet. I'm still getting my head around > just where the join quals are actually stored. TBH I think that trying to do anything at all for inner joins is probably a bad idea. The cases where the optimization could succeed are so narrow th

Re: [HACKERS] Allowing join removals for more join types

2014-05-28 Thread David Rowley
On Sun, May 25, 2014 at 5:42 AM, Tom Lane wrote: > David Rowley writes: > > I agree that there are not many cases left to remove the join that remain > > after is_simple_subquery() has decided not to pullup the subquery. Some > of > > the perhaps more common cases would be having windowing funct

Re: [HACKERS] Allowing join removals for more join types

2014-05-24 Thread Tom Lane
David Rowley writes: > I agree that there are not many cases left to remove the join that remain > after is_simple_subquery() has decided not to pullup the subquery. Some of > the perhaps more common cases would be having windowing functions in the > subquery as this is what you need to do if you

Re: [HACKERS] Allowing join removals for more join types

2014-05-23 Thread David Rowley
On Sat, May 24, 2014 at 3:13 AM, Tom Lane wrote: > David Rowley writes: > > I've just had a bit of a look at implementing checks allowing subqueries > > with unique indexes on the join cols being removed, > > I'm a bit confused by this statement of the problem. I thought the idea > was to recog

Re: [HACKERS] Allowing join removals for more join types

2014-05-23 Thread Tom Lane
David Rowley writes: > I've just had a bit of a look at implementing checks allowing subqueries > with unique indexes on the join cols being removed, I'm a bit confused by this statement of the problem. I thought the idea was to recognize that subqueries with DISTINCT or GROUP BY clauses produce

Re: [HACKERS] Allowing join removals for more join types

2014-05-23 Thread David Rowley
On Fri, May 23, 2014 at 8:28 PM, Dilip kumar wrote: > On 23 May 2014 12:43 David Rowley Wrote, > > > > >I'm hitting a bit of a roadblock on point 1. Here's a snipped from my > latest attempt: > > > > > if (bms_membership(innerrel->relids) == > BMS_SINGLETON) > > >

Re: [HACKERS] Allowing join removals for more join types

2014-05-23 Thread Dilip kumar
On 23 May 2014 12:43 David Rowley Wrote, >I'm hitting a bit of a roadblock on point 1. Here's a snipped from my latest >attempt: > if (bms_membership(innerrel->relids) == BMS_SINGLETON) > { > int subqueryrelid = > bms_si

Re: [HACKERS] Allowing join removals for more join types

2014-05-23 Thread David Rowley
On Mon, May 19, 2014 at 5:47 PM, Dilip kumar wrote: > > > So I think now when you are considering this join removal for subqueries > then this can consider other case also like unique index inside subquery, > > because in attached patch unique index is considered only if its > RTE_RELATION > >

Re: [HACKERS] Allowing join removals for more join types

2014-05-21 Thread David Rowley
On Tue, May 20, 2014 at 11:22 PM, Tom Lane wrote: > David Rowley writes: > > I'm also now wondering if I need to do some extra tests in the existing > > code to ensure that the subquery would have had no side affects. > > You should probably at least refuse the optimization if the subquery's > t

Re: [HACKERS] Allowing join removals for more join types

2014-05-20 Thread Tom Lane
David Rowley writes: > I'm also now wondering if I need to do some extra tests in the existing > code to ensure that the subquery would have had no side affects. You should probably at least refuse the optimization if the subquery's tlist contains volatile functions. Functions that return sets m

Re: [HACKERS] Allowing join removals for more join types

2014-05-20 Thread David Rowley
On Mon, May 19, 2014 at 9:22 PM, Dilip kumar wrote: > On 19 May 2014 12:15 David Rowley Wrote, > > > > > > May be we can convert my above example like below à in this case we > have unique index on field a and we are limiting it by first 100 tuple > (record are already order because of index)

Re: [HACKERS] Allowing join removals for more join types

2014-05-19 Thread Dilip kumar
On 19 May 2014 12:15 David Rowley Wrote, >I think you are right here, it would be correct to remove that join, but I >also think that the query in question could be quite easily be written as: >select t1.a from t1 left join t2 on t1.a=t2.b; >Where the join WILL be removed. The distinct clause h

Re: [HACKERS] Allowing join removals for more join types

2014-05-18 Thread David Rowley
On Mon, May 19, 2014 at 5:47 PM, Dilip kumar wrote: > On 18 May 2014 16:38 David Rowley Wrote > > > > Sound like a good idea to me.. > > > > I have one doubt regarding the implementation, consider the below query > > > > Create table t1 (a int, b int); > > Create table t2 (a int, b int); > > > >

Re: [HACKERS] Allowing join removals for more join types

2014-05-18 Thread Dilip kumar
On 18 May 2014 16:38 David Rowley Wrote Sound like a good idea to me.. I have one doubt regarding the implementation, consider the below query Create table t1 (a int, b int); Create table t2 (a int, b int); Create unique index on t2(b); select x.a from t1 x left join (select distinct t2.a a1,

Re: [HACKERS] Allowing join removals for more join types

2014-05-18 Thread David Rowley
On Sat, May 17, 2014 at 8:57 PM, David Rowley wrote: > I'm currently in the early stages of looking into expanding join removals. > > Currently left outer joins can be removed if none of the columns of the > table are required for anything and the table being joined is a base table > that contain

Re: [HACKERS] Allowing join removals for more join types

2014-05-17 Thread David Rowley
On Sun, May 18, 2014 at 2:55 AM, Tom Lane wrote: > David Rowley writes: > > It looks like the existing join removals are done quite early in the > > planning and redundant joins are removed before any subqueries from that > > query are planned. So this innerrel->subroot->parse has not been done

Re: [HACKERS] Allowing join removals for more join types

2014-05-17 Thread Tom Lane
David Rowley writes: > It looks like the existing join removals are done quite early in the > planning and redundant joins are removed before any subqueries from that > query are planned. So this innerrel->subroot->parse has not been done yet. > It seems to be done later in query_planner() when ma

Re: [HACKERS] Allowing join removals for more join types

2014-05-17 Thread David Rowley
On Sat, May 17, 2014 at 8:57 PM, David Rowley wrote: > I'm currently in the early stages of looking into expanding join removals. > > As I said above, I'm in the early stages of looking at this and I'm > currently a bit confused. Basically I've put a breakpoint at the top of > the join_is_removab

[HACKERS] Allowing join removals for more join types

2014-05-17 Thread David Rowley
I'm currently in the early stages of looking into expanding join removals. Currently left outer joins can be removed if none of the columns of the table are required for anything and the table being joined is a base table that contains a unique index on all columns in the join clause. The case I