Hi hackers, At Neon, we've been working on removing the file system dependency from PostgreSQL and replacing it with a distributed storage layer. For now, we've seen most success in this by replacing the implementation of the smgr API, but it did require some core modifications like those proposed early last year by Anastasia [0].
As mentioned in the previous thread, there are several reasons why you would want to use a non-default storage manager: storage-level compression, encryption, and disk limit quotas [0]; offloading of cold relation data was also mentioned [1]. In the thread on Anastasia's patch, Yura Sokolov mentioned that instead of a hook-based smgr extension, a registration-based smgr would be preferred, with integration into namespaces. Please find attached an as of yet incomplete patch that starts to do that. The patch is yet incomplete (as it isn't derived from Anastasia's patch), but I would like comments on this regardless, as this is a fairly fundamental component of PostgreSQL that is being modified, and it is often better to get comments early in the development cycle. One significant issue that I've seen so far are that catcache is not guaranteed to be available in all backends that need to do smgr operations, and I've not yet found a good solution. Changes compared to HEAD: - smgrsw is now dynamically allocated and grows as new storage managers are loaded (during shared_preload_libraries) - CREATE TABLESPACE has new optional syntax USING smgrname (option [, ...]) - tablespace storage is (planned) fully managed by smgr through some new smgr apis Changes compared to Anastasia's patch: - extensions do not get to hook and replace the api of the smgr code directly - they are hidden behind the smgr registry. Successes: - 0001 passes tests (make check-world) - 0002 builds without warnings (make) TODO: - fix dependency failures when catcache is unavailable - tablespace redo is currently broken with 0002 - fix tests for 0002 - ensure that pg_dump etc. works with the new tablespace storage manager options Looking forward to any comments, suggestions and reviews. Kind regards, Matthias van de Meent Neon (https://neon.tech/) [0] https://www.postgresql.org/message-id/CAP4vRV6JKXyFfEOf%3Dn%2Bv5RGsZywAQ3CTM8ESWvgq%2BS87Tmgx_g%40mail.gmail.com [1] https://www.postgresql.org/message-id/d365f19f-bc3e-4f96-a91e-8db130497...@yandex-team.ru
v1-0001-Expose-f_smgr-to-extensions-for-manual-implementa.patch
Description: Binary data
v1-0002-Prototype-Allow-tablespaces-to-specify-which-SMGR.patch
Description: Binary data
