On Fri, Apr 17, 2020 at 7:44 PM Andres Freund <and...@anarazel.de> wrote: > This suggest that pipes do have a considerably higher overhead on > windows, but that it's not all that terrible if one takes care to use > large buffers in each pipe element. > > It's notable though that even the simplest use of a pipe does add a > considerable overhead compared to using the files directly.
Thanks for these results. I think that this shows that it's probably not a great idea to force everything to go through pipes in every case, but on the other hand, there's no reason to be a particularly scared of the performance implications of letting some things go through pipes. For instance, if we decide that LZ4 compression is going to be a good choice for most users, we might want to do that in-process rather than via pipes. However, if somebody wants to pipe through an external compressor that they prefer, that's going to be a little slower, but not necessarily to a degree that creates big problems. People with bigger databases will need to be more careful about which options they choose, but that's kind of inevitable. Do you agree? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
