Re: [HACKERS] Asynchronous execution on FDW

2015-08-10 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Aug 10, 2015 at 3:23 AM, Heikki Linnakangas wrote: > > I've marked this as rejected in the commitfest, because others are > > working on a more general solution with parallel workers. That's still > > work-in-progress, and it's not certain if

Re: [HACKERS] Asynchronous execution on FDW

2015-08-10 Thread Robert Haas
On Mon, Aug 10, 2015 at 3:23 AM, Heikki Linnakangas wrote: > I've marked this as rejected in the commitfest, because others are > working on a more general solution with parallel workers. That's still > work-in-progress, and it's not certain if it's going to make it into > 9.6, but if it does it w

Re: [HACKERS] Asynchronous execution on FDW

2015-08-10 Thread Heikki Linnakangas
I've marked this as rejected in the commitfest, because others are working on a more general solution with parallel workers. That's still work-in-progress, and it's not certain if it's going to make it into 9.6, but if it does it will largely render this obsolete. We can revisit this patch later in

Re: [HACKERS] Asynchronous execution on FDW

2015-07-24 Thread Kouhei Kaigai
Hello Horiguchi-san, > > > As for ForeignScan, it is merely an API for FDW and does nothing > > > substantial so it would have nothing special to do. As for > > > postgres_fdw, current patch restricts one execution per one > > > foreign server at once by itself. We would have to provide > > > anot

Re: [HACKERS] Asynchronous execution on FDW

2015-07-23 Thread Kyotaro HORIGUCHI
Hello, At Thu, 23 Jul 2015 09:38:39 +, Kouhei Kaigai wrote in <9a28c8860f777e439aa12e8aea7694f80111b...@bpxm15gp.gisp.nec.co.jp> > I expected workloads like single shot scan on a partitioned large > fact table on DWH system. Yep, if workload is expected to rescan > so frequently, its expecte

Re: [HACKERS] Asynchronous execution on FDW

2015-07-23 Thread Kouhei Kaigai
> > If we have ParallelAppend node that kicks a background worker process for > > each underlying child node in parallel, does ForeignScan need to do > > something > > special? > > Although I don't see the point of the background worker in your > story but at least for ParalleMergeAppend, it woul

Re: [HACKERS] Asynchronous execution on FDW

2015-07-22 Thread Kyotaro HORIGUCHI
Hello, > Let me ask an elemental question. > > If we have ParallelAppend node that kicks a background worker process for > each underlying child node in parallel, does ForeignScan need to do something > special? Although I don't see the point of the background worker in your story but at least f

Re: [HACKERS] Asynchronous execution on FDW

2015-07-22 Thread Kouhei Kaigai
Subject: Re: [HACKERS] Asynchronous execution on FDW > > Hello, thank you for the comment. > > At Fri, 17 Jul 2015 14:34:53 -0400, Robert Haas wrote > in > > On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas wrote: > > > At a quick glance, I think this has all t

Re: [HACKERS] Asynchronous execution on FDW

2015-07-22 Thread Kyotaro HORIGUCHI
Hello, thank you for the comment. At Fri, 17 Jul 2015 14:34:53 -0400, Robert Haas wrote in > On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas wrote: > > At a quick glance, I think this has all the same problems as starting the > > execution at ExecInit phase. The correct way to do this is to

Re: [HACKERS] Asynchronous execution on FDW

2015-07-17 Thread Robert Haas
On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas wrote: > At a quick glance, I think this has all the same problems as starting the > execution at ExecInit phase. The correct way to do this is to kick off the > queries in the first IterateForeignScan() call. You said that "ExecProc > phase does

Re: [HACKERS] Asynchronous execution on FDW

2015-07-10 Thread Kyotaro HORIGUCHI
Hi, > Currently there's no means to observe what it is doing from > outside, so the additional sixth patch is to output debug > messages about asynchronous execution. The sixth patch did not contain one message shown in the example. Attached is the revised version. Other patches are not changed.

Re: [HACKERS] Asynchronous execution on FDW

2015-07-06 Thread Kyotaro HORIGUCHI
Hello, thank you for looking this. If it is acceptable to reconstruct the executor nodes to have additional return state PREP_RUN or such (which means it needs one more call for the first tuple) , I'll modify the whole executor to handle the state in the next patch to do so. I haven't take the ad

Re: [HACKERS] Asynchronous execution on FDW

2015-07-03 Thread Heikki Linnakangas
On 07/02/2015 08:48 AM, Kyotaro HORIGUCHI wrote: - It was a problem when to give the first kick for async exec. It is not in ExecInit phase, and ExecProc phase does not fit, too. An extra phase ExecPreProc or something is too invasive. So I tried "pre-exec callback". Any init-node ca

Re: [HACKERS] Asynchronous execution on FDW

2015-07-02 Thread Kyotaro HORIGUCHI
Thank you. At Thu, 2 Jul 2015 16:02:27 +0900, Michael Paquier wrote in > On Thu, Jul 2, 2015 at 3:07 PM, Kyotaro HORIGUCHI > wrote: > > Ouch! I mistakenly made two CF entries for this patch. Could > > someone remove this entry for me? > > > > https://commitfest.postgresql.org/5/290/ > > > > Th

Re: [HACKERS] Asynchronous execution on FDW

2015-07-02 Thread Michael Paquier
On Thu, Jul 2, 2015 at 3:07 PM, Kyotaro HORIGUCHI wrote: > Ouch! I mistakenly made two CF entries for this patch. Could > someone remove this entry for me? > > https://commitfest.postgresql.org/5/290/ > > The correct entry is "/5/291/" Done. -- Michael -- Sent via pgsql-hackers mailing list (

Re: [HACKERS] Asynchronous execution on FDW

2015-07-01 Thread Kyotaro HORIGUCHI
Ouch! I mistakenly made two CF entries for this patch. Could someone remove this entry for me? https://commitfest.postgresql.org/5/290/ The correct entry is "/5/291/" == Hello. This is the new version of FDW async exection feature. The status of this feature is as follows, as of the last co

[HACKERS] Asynchronous execution on FDW

2015-07-01 Thread Kyotaro HORIGUCHI
Hello. This is the new version of FDW async exection feature. The status of this feature is as follows, as of the last commitfest. - Async execution is valuable to have. - But do the first kick in ExecInit phase is wrong. So the design outline of this version is as following, - The patch set co