Re: Parallel Foreign Scans - need advice

2019-05-16 Thread Thomas Munro
On Fri, May 17, 2019 at 12:45 AM Korry Douglas wrote: > It sounds like there is no reliable way to get the information that I’m > looking for, is that right? Correct. And if there were, it could only be used to write bugs. Let me see if I can demonstrate... I'll use the file_fdw patch from th

Re: Parallel Foreign Scans - need advice

2019-05-16 Thread Korry Douglas
> That's only a superficial problem. You don't even know if or when the > workers that are launched will all finish up running your particular > node, because (for example) they might be sent to different children > of a Parallel Append node above you (AFAICS there is no way for a > participant

Re: Parallel Foreign Scans - need advice

2019-05-15 Thread Thomas Munro
On Thu, May 16, 2019 at 5:46 AM Korry Douglas wrote: > >> But, nworkers_launched is always set to 0 in > >> InitializeDSMForeignScan(), so that won’t work. Any other ideas? > > > > At that state it's simply not yet known how many workers will be > > actually launched (they might not start success

Re: Parallel Foreign Scans - need advice

2019-05-15 Thread Korry Douglas
>> But, nworkers_launched is always set to 0 in >> InitializeDSMForeignScan(), so that won’t work. Any other ideas? > > At that state it's simply not yet known how many workers will be > actually launched (they might not start successfully or such). Why do > you need to know it there and not la

Re: Parallel Foreign Scans - need advice

2019-05-15 Thread Andres Freund
Hi, Don't top quote on these list... On 2019-05-15 13:31:59 -0400, Korry Douglas wrote: > Thanks for the quick answer Andres. You’re right - it was > parallel_tuple_cost that was getting in my way; my query returns about 6 > million rows so I guess that can add up. > > If I change parallel_t

Re: Parallel Foreign Scans - need advice

2019-05-15 Thread Korry Douglas
Thanks for the quick answer Andres. You’re right - it was parallel_tuple_cost that was getting in my way; my query returns about 6 million rows so I guess that can add up. If I change parallel_tuple_scan from 0.1 to 0.0001, I get a parallel foreign scan. With 4 workers, that reduces my execu

Re: Parallel Foreign Scans - need advice

2019-05-15 Thread Andres Freund
Hi, On 2019-05-15 12:55:33 -0400, Korry Douglas wrote: > Hi all, I’m working on an FDW that would benefit greatly from parallel > foreign scan. I have implemented the callbacks described > here:https://www.postgresql.org/docs/devel/fdw-callbacks.html#FDW-CALLBACKS-PARALLEL. > and I see a big i