On Thu, Sep 14, 2017 at 9:41 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Sep 11, 2017 at 9:25 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I think the patch stores only non-partial paths in decreasing order, >> what if partial paths having more costs follows those paths? > > The general picture here is that we don't want the leader to get stuck > inside some long-running operation because then it won't be available > to read tuples from the workers. On the other hand, we don't want to > just have the leader do no work because that might be slow. And in > most cast cases, the leader will be the first participant to arrive at > the Append node, because of the worker startup time. So the idea is > that the workers should pick expensive things first, and the leader > should pick cheap things first. >
At a broader level, the idea is good, but I think it won't turn out exactly like that considering your below paragraph which indicates that it is okay if leader picks a partial path that is costly among other partial paths as a leader won't be locked with that. Considering this is a good design for parallel append, the question is do we really need worker and leader to follow separate strategy for choosing next path. I think the patch will be simpler if we can come up with a way for the worker and leader to use the same strategy to pick next path to process. How about we arrange the list of paths such that first, all partial paths will be there and then non-partial paths and probably both in decreasing order of cost. Now, both leader and worker can start from the beginning of the list. In most cases, the leader will start at the first partial path and will only ever need to scan non-partial path if there is no other partial path left. This is not bulletproof as it is possible that some worker starts before leader in which case leader might scan non-partial path before all partial paths are finished, but I think we can avoid that as well if we are too worried about such cases. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
