On Mon, Aug 20, 2018 at 11:09 AM Thomas Munro <thomas.mu...@enterprisedb.com> wrote: > 2. Teach your GetForeignPath function to do something like this: > [blah blah blah]
I was pinged off-list by someone who is working on a parallel-aware FDW, who asked if I still had the test code I mentioned up-thread. While digging that out, I couldn't resist hacking it a bit more until it gave the right answers, only sooner: $ seq -f '%20.0f' 1 10000000 > numbers.csv create extension file_fdw; create server files foreign data wrapper file_fdw; create foreign table numbers (n int) server files options (filename '/path/to/numbers.csv', format 'csv'); explain select count(*) from numbers; select count(*) from numbers; Non-parallel: 2.6s 1 worker: 1.4s 2 workers: 0.9s 3 workers: 0.7s Finally, I can do parallel hash joins between CSV files! select count(*) from numbers n1 join numbers n2 using (n); Non-parallel: 11.4s 1 worker: 6.6s 2 workers: 4.8s 3 workers: 4.1s There are probably some weird fence-post or accounting bugs hiding in this patch -- it has to count bytes carefully, and deal with some edge cases around lines that span chunks. It's only a rough draft, but might eventually serve as a useful example of a parallel-aware FDW. -- Thomas Munro https://enterprisedb.com
0001-Make-file_fdw-parallel-aware.patch
Description: Binary data
