Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-14 Thread David Rowley
On 13 February 2015 at 20:52, Michael Paquier wrote: > > > On Sun, Nov 23, 2014 at 8:23 PM, David Rowley > wrote: > >> >> As the patch stands there's still a couple of FIXMEs in there, so there's >> still a bit of work to do yet. >> Comments are welcome >> > > Hm, if there is still work to do, w

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-14 Thread David Rowley
There does not seem to be a delete button, so marking as "rejected" due to this now being a duplicate entry for this patch. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-13 Thread Michael Paquier
On Fri, Feb 13, 2015 at 5:12 PM, Andres Freund wrote: > On 2015-02-13 17:06:14 +0900, Michael Paquier wrote: > > On Fri, Feb 13, 2015 at 4:57 PM, Marko Tiikkaja wrote: > > > > > On 2/13/15 8:52 AM, Michael Paquier wrote: > > > > > >> On Sun, Nov 23, 2014 at 8:23 PM, David Rowley > > >> wrote: >

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-13 Thread Andres Freund
On 2015-02-13 17:06:14 +0900, Michael Paquier wrote: > On Fri, Feb 13, 2015 at 4:57 PM, Marko Tiikkaja wrote: > > > On 2/13/15 8:52 AM, Michael Paquier wrote: > > > >> On Sun, Nov 23, 2014 at 8:23 PM, David Rowley > >> wrote: > >> > >>> As the patch stands there's still a couple of FIXMEs in the

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-13 Thread Michael Paquier
On Fri, Feb 13, 2015 at 4:57 PM, Marko Tiikkaja wrote: > On 2/13/15 8:52 AM, Michael Paquier wrote: > >> On Sun, Nov 23, 2014 at 8:23 PM, David Rowley >> wrote: >> >>> As the patch stands there's still a couple of FIXMEs in there, so there's >>> still a bit of work to do yet. >>> Comments are we

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-12 Thread Marko Tiikkaja
On 2/13/15 8:52 AM, Michael Paquier wrote: On Sun, Nov 23, 2014 at 8:23 PM, David Rowley wrote: As the patch stands there's still a couple of FIXMEs in there, so there's still a bit of work to do yet. Comments are welcome Hm, if there is still work to do, we may as well mark this patch as re

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-12 Thread Michael Paquier
On Sun, Nov 23, 2014 at 8:23 PM, David Rowley wrote: > > As the patch stands there's still a couple of FIXMEs in there, so there's > still a bit of work to do yet. > Comments are welcome > Hm, if there is still work to do, we may as well mark this patch as rejected as-is, also because it stands

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-11-23 Thread David Rowley
On Wed, Nov 19, 2014 at 11:49 PM, David Rowley wrote: > On Sun, Nov 16, 2014 at 12:19 PM, David Rowley > wrote: > >> On Sun, Nov 16, 2014 at 10:09 AM, Simon Riggs >> wrote: >> >>> >>> I propose that we keep track of whether there are any potentially >>> skippable joins at the top of the plan. W

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-11-19 Thread David Rowley
On Sun, Nov 16, 2014 at 12:19 PM, David Rowley wrote: > On Sun, Nov 16, 2014 at 10:09 AM, Simon Riggs > wrote: > >> >> I propose that we keep track of whether there are any potentially >> skippable joins at the top of the plan. When we begin execution we do >> a single if test to see if there is

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-11-15 Thread David Rowley
On Sun, Nov 16, 2014 at 10:09 AM, Simon Riggs wrote: > On 15 October 2014 11:03, David Rowley wrote: > > > The explain analyze from the above query looks like: > > test=# explain (analyze, costs off, timing off) select count(*) from t1 > > inner join t2 on t1.t2_id=t2.id; > >

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-11-15 Thread Simon Riggs
On 15 October 2014 11:03, David Rowley wrote: > The explain analyze from the above query looks like: > test=# explain (analyze, costs off, timing off) select count(*) from t1 > inner join t2 on t1.t2_id=t2.id; > QUERY PLAN >

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-15 Thread David Rowley
On Thu, Oct 9, 2014 at 12:40 AM, Andres Freund wrote: > On 2014-10-09 00:21:44 +1300, David Rowley wrote: > > Ok, so I've been hacking away at this for a couple of evenings and I > think > > I have a working prototype finally! > > Cool! > > Patch attached. > > So it seems it's not quite as effi

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-08 Thread Andres Freund
On 2014-10-09 00:21:44 +1300, David Rowley wrote: > Ok, so I've been hacking away at this for a couple of evenings and I think > I have a working prototype finally! Cool! > So it seems it's not quite as efficient as join removal at planning time, > but still a big win when it's possible to perfor

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-08 Thread David Rowley
On Tue, Oct 7, 2014 at 3:46 AM, Robert Haas wrote: > On Mon, Oct 6, 2014 at 5:57 AM, David Rowley wrote: > > Can anyone shed any light on how I might determine where the scan rel is > in > > the tree? I need to find it so I can check if the RangeTblEntry is > marked as > > skip-able. > > I think

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 10:59 AM, Andres Freund wrote: > On 2014-10-06 10:46:09 -0400, Robert Haas wrote: >> This seems messy, though. Can't the deferred trigger queue become >> non-empty at pretty much any point in time? At exactly what point are >> we making this decision, and how do we know th

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-06 Thread Andres Freund
On 2014-10-06 10:46:09 -0400, Robert Haas wrote: > This seems messy, though. Can't the deferred trigger queue become > non-empty at pretty much any point in time? At exactly what point are > we making this decision, and how do we know the correct answer can't > change after that point? What we'v

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 5:57 AM, David Rowley wrote: >> Hm, right. But that doesn't seem like a fatal problem to me. The planner >> knows about t1/t2 and Seq(t1), Seq(t2), not just Hash(Seq(t2)). So it >> can tell the HashJoin node that when the 'shortcut' qualifier is true, >> it should source eve

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-06 Thread David Rowley
On Wed, Oct 1, 2014 at 1:34 AM, Andres Freund wrote: > On 2014-10-01 01:03:35 +1300, David Rowley wrote: > > On Wed, Oct 1, 2014 at 12:01 AM, Andres Freund > > wrote: > > > > > On 2014-09-30 23:25:45 +1300, David Rowley wrote: > > > > > > > > I've not quite gotten my head around how we might sto

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-30 Thread Andres Freund
On 2014-10-01 01:03:35 +1300, David Rowley wrote: > On Wed, Oct 1, 2014 at 12:01 AM, Andres Freund > wrote: > > > On 2014-09-30 23:25:45 +1300, David Rowley wrote: > > > > > > I've not quite gotten my head around how we might stop the unneeded > > > relation from being the primary path to join th

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-30 Thread David Rowley
On Wed, Oct 1, 2014 at 12:01 AM, Andres Freund wrote: > On 2014-09-30 23:25:45 +1300, David Rowley wrote: > > > > I've not quite gotten my head around how we might stop the unneeded > > relation from being the primary path to join the other inner relations, > > i.e. what would stop the planner ma

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-30 Thread Andres Freund
On 2014-09-30 23:25:45 +1300, David Rowley wrote: > On Tue, Sep 30, 2014 at 12:42 AM, Andres Freund > wrote: > > > On 2014-09-29 22:42:57 +1300, David Rowley wrote: > > > > > I've made a change to the patch locally to ignore foreign > > > keys that are marked as deferrable. > > > > I have serious

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-30 Thread David Rowley
On Tue, Sep 30, 2014 at 12:42 AM, Andres Freund wrote: > On 2014-09-29 22:42:57 +1300, David Rowley wrote: > > > I've made a change to the patch locally to ignore foreign > > keys that are marked as deferrable. > > I have serious doubts about the general usefulness if this is onlyu > going to be

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Tom Lane
Andres Freund writes: > On 2014-09-29 10:12:25 -0400, Tom Lane wrote: >> I see. So why aren't we simply ignoring deferrable FKs when making the >> optimization? That pushes it back from depending on execution-time state >> (unsafe) to depending on table DDL (safe). > IIRC there's some scenarios

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Andres Freund
On 2014-09-29 10:12:25 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2014-09-28 10:41:56 -0400, Tom Lane wrote: > >> If this optimization only works in that scenario, it's dead in the water, > >> because that assumption is unsupportable. The planner does not in general > >> use the same q

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Tom Lane
Andres Freund writes: > On 2014-09-28 10:41:56 -0400, Tom Lane wrote: >> If this optimization only works in that scenario, it's dead in the water, >> because that assumption is unsupportable. The planner does not in general >> use the same query snapshot as the executor, so even in an immediate-

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Andres Freund
On 2014-09-29 22:42:57 +1300, David Rowley wrote: > On Mon, Sep 29, 2014 at 2:41 AM, Andres Freund > wrote: > > > On 2014-09-28 17:32:21 +1300, David Rowley wrote: > > > My understanding of foreign keys is that any pending foreign key triggers > > > will be executed just before the query complete

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Andres Freund
On 2014-09-28 10:41:56 -0400, Tom Lane wrote: > David Rowley writes: > > Please correct anything that sounds wrong here, but my understanding is > > that we'll always plan a query right before we execute it, with the > > exception of PREPARE statements where PostgreSQL will cache the query plan >

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread David Rowley
On Mon, Sep 29, 2014 at 2:41 AM, Andres Freund wrote: > On 2014-09-28 17:32:21 +1300, David Rowley wrote: > > My understanding of foreign keys is that any pending foreign key triggers > > will be executed just before the query completes, so we should only ever > > encounter pending foreign key tr

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-28 Thread Tom Lane
David Rowley writes: > Please correct anything that sounds wrong here, but my understanding is > that we'll always plan a query right before we execute it, with the > exception of PREPARE statements where PostgreSQL will cache the query plan > when the prepare statement is first executed. If this

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-28 Thread Andres Freund
On 2014-09-28 17:32:21 +1300, David Rowley wrote: > My understanding of foreign keys is that any pending foreign key triggers > will be executed just before the query completes, so we should only ever > encounter pending foreign key triggers during planning when we're planning > a query that's bein

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-27 Thread David Rowley
On Fri, Sep 26, 2014 at 12:36 AM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 09/16/2014 01:20 PM, David Rowley wrote: > >> + /* >> +* We mustn't allow any joins to be removed if there are any >> pending >> +* foreign key triggers in the queue. This could happen

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-25 Thread Heikki Linnakangas
On 09/16/2014 01:20 PM, David Rowley wrote: + /* +* We mustn't allow any joins to be removed if there are any pending +* foreign key triggers in the queue. This could happen if we are planning +* a query that has been executed from within a volatile function and th

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-16 Thread David Rowley
On Sat, Sep 13, 2014 at 1:38 AM, Tom Lane wrote: > David Rowley writes: > > On Fri, Sep 12, 2014 at 3:35 AM, Robert Haas > wrote: > >> I haven't read the patch, but I think the question is why this needs > >> to be different than what we do for left join removal. > > > I discovered over here ->

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread Tom Lane
David Rowley writes: > On Fri, Sep 12, 2014 at 3:35 AM, Robert Haas wrote: >> I haven't read the patch, but I think the question is why this needs >> to be different than what we do for left join removal. > I discovered over here -> > http://www.postgresql.org/message-id/CAApHDvo5wCRk7uHBuMHJaDp

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread David Rowley
On Fri, Sep 12, 2014 at 3:47 AM, Tom Lane wrote: > Robert Haas writes: > > On Thu, Sep 11, 2014 at 7:14 AM, David Rowley > wrote: > >> 5. I've added a flag to pg_class called relhasfkey. Currently this gets > set > >> to true when a foreign key is added, though I've added nothing to set it > >>

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread David Rowley
On Fri, Sep 12, 2014 at 3:35 AM, Robert Haas wrote: > On Thu, Sep 11, 2014 at 7:14 AM, David Rowley > wrote: > > > > 1. I don't think that I'm currently handling removing eclass members > > properly. So far the code just removes the Vars that belong to the > relation > > being removed. I likely

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-11 Thread Tom Lane
Robert Haas writes: > On Thu, Sep 11, 2014 at 7:14 AM, David Rowley wrote: >> 5. I've added a flag to pg_class called relhasfkey. Currently this gets set >> to true when a foreign key is added, though I've added nothing to set it >> back to false again. I notice that relhasindex gets set back to

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-11 Thread Robert Haas
On Thu, Sep 11, 2014 at 7:14 AM, David Rowley wrote: > Here's a quick demo, of the patch at work: > > test=# create table c (id int primary key); > CREATE TABLE > test=# create table b (id int primary key, c_id int not null references > c(id)); > CREATE TABLE > test=# create table a (id int primar

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-11 Thread David Rowley
On Thu, Aug 28, 2014 at 6:23 AM, Tom Lane wrote: > > If the majority of the added code is code that will be needed for > less-bogus optimizations, it might be all right; but I'd kind of want to > see the less-bogus optimizations working first. > > That seems fair. Likely there'd be not a great dea

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread Tom Lane
Jim Nasby writes: > On 8/26/14, 8:40 AM, Heikki Linnakangas wrote: >> Just so everyone is on the same page on what kind of queries this helps >> with, here are some examples from the added regression tests: >> > -- Test join removals for semi and anti joins > CREATE TEMP TABLE b (id INT NOT NULL

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread Jim Nasby
On 8/26/14, 8:40 AM, Heikki Linnakangas wrote: Just so everyone is on the same page on what kind of queries this helps with, here are some examples from the added regression tests: -- Test join removals for semi and anti joins CREATE TEMP TABLE b (id INT NOT NULL PRIMARY KEY, val INT); CREATE

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread David Rowley
On Wed, Aug 27, 2014 at 1:40 AM, Heikki Linnakangas wrote: > On 08/26/2014 03:28 PM, David Rowley wrote: > >> Any ideas or feedback on this would be welcome >> > > Before someone spends time reviewing this patch, are you sure this is > worth the effort? It seems like very narrow use case to me. I

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 03:28 PM, David Rowley wrote: Any ideas or feedback on this would be welcome Before someone spends time reviewing this patch, are you sure this is worth the effort? It seems like very narrow use case to me. I understand removing LEFT and INNER joins, but the case for SEMI and AN

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-26 Thread David Rowley
On Tue, Aug 5, 2014 at 10:35 PM, David Rowley wrote: > > Currently most of my changes are in analyzejoin.c, but I did also have to > make changes to load the foreign key constraints so that they were > available to the planner. One thing that is currently lacking, which would > likely be needed,

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-16 Thread David Rowley
On Sun, Aug 10, 2014 at 11:48 PM, David Rowley wrote: > > I've attached an updated version of the patch which fixes up some > incorrect logic in the foreign key matching code, plus various comment > improvements. > I've made a few updated to the patch to simplify some logic in the code which a

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-10 Thread David Rowley
On Tue, Aug 5, 2014 at 10:35 PM, David Rowley wrote: > > The patch (attached) is also now able to detect when a NOT EXISTS clause > cannot produce any records at all. > > I've attached an updated version of the patch which fixes up some incorrect logic in the foreign key matching code, plus vari

[HACKERS] Patch to support SEMI and ANTI join removal

2014-08-05 Thread David Rowley
I've been working away at allowing semi and anti joins to be added to the list of join types that our join removal code supports. The basic idea is that we can removal a semi or anti join if the left hand relation references the relation that's being semi/anti joined if the join condition matches