I’m a latecomer to the discussion, but as a word of introduction, I’m working with Stephen, and I have started looking over the temp file proposal with the idea of helping it move along.
I’ll start by summarizing the temp file proposal and its goals. >From a high level, the proposed code: * Creates an fread/fwrite replacement (BufFileStream) for buffering data to a single file. * Updates BufFile, which reads/writes a set of files, to use BufFileStream internally. * Does not impact the normal (page cached) database I/O. * Updates all the other places where fread/fwrite and read/write are used. * Creates and removes transient files. I see the following goals: * Unify all the “other” file accesses into a single, consistent API. * Integrate with VFDs. * Integrate transient files with transactions and tablespaces. * Create a consolidated framework where features like encryption and compression can be more easily added. * Maintain good streaming performance. Is this a fair description of the proposal? For myself, I’d like to map out how features like compression and encryption would fit into the framework, more as a sanity check than anything else, and I’d like to look closer at some of the implementation details. But at the moment, I want to make sure I have the proper high-level view of the temp file proposal. From: Robert Haas <robertmh...@gmail.com> Date: Wednesday, September 21, 2022 at 11:54 AM To: Antonin Houska <a...@cybertec.at> Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Peter Eisentraut <peter.eisentr...@enterprisedb.com>, Stephen Frost <sfr...@snowman.net> Subject: Re: Temporary file access API On Mon, Aug 8, 2022 at 2:26 PM Antonin Houska <a...@cybertec.at> wrote: > > I don't think that (3) is a separate advantage from (1) and (2), so I > > don't have anything separate to say about it. > > I thought that the uncontrollable buffer size is one of the things you > complaint about in > > https://www.postgresql.org/message-id/CA+TgmoYGjN_f=fcerx49bzjhng+gocty+a+xhnrwcvvdy8u...@mail.gmail.com Well, I think that email is mostly complaining about there being no buffering at all in a situation where it would be advantageous to do some buffering. But that particular code I believe is gone now because of the shared-memory stats collector, and when looking through the patch set, I didn't see that any of the cases that it touched were similar to that one. -- Robert Haas EDB: http://www.enterprisedb.com
