On 28/12/2025 4:53 PM, Henson Choi wrote:
Subject: Re: RFC: PostgreSQL Storage I/O Transformation Hooks
Hi Konstantin,
I have great respect for the work being done on the extensible SMGR API.
It is a powerful tool for use cases that require replacing the entire
storage layer (like Neon's architecture).
However, I believe we should distinguish between Storage Management
(where/how data is stored) and Data Transformation (what the data looks
like). I see a strong case for both approaches to coexist for the
following practical reasons:
1. Separation of Concerns and Safety
Is it reasonable to ask cryptography experts to clone the entire SMGR
implementation and maintain code they don't fully understand just to
insert encryption logic? If an extension developer clones md.c to add
encryption, they become responsible for the fundamental integrity of
PostgreSQL's file I/O. Any bug in their cloned storage logic could lead
to data loss unrelated to encryption itself.
2. The Maintenance Debt of "Cloning"
When md.c receives critical security patches or bug fixes in the core,
every TDE extension maintainer would need to manually backport those
changes to their specific SMGR implementation. This creates a fragmented
ecosystem where security extensions might actually introduce storage
vulnerabilities by running outdated cloned logic.
3. Minimalist Integration
The hook approach allows crypto experts to focus strictly on transform()
and reverse_transform(). The complex storage orchestration remains with
the PostgreSQL core where it is most rigorously tested. This is a cleaner
separation of responsibilities: the core provides the trusted pipeline,
and the extension provides the specialized transformation.
Conclusion:
I believe these hooks provide a "low-barrier, high-safety" path for data
transformation that the SMGR API—by its very nature of being a full
replacement—cannot easily provide. Let's provide the SMGR for those who
want to reinvent the storage, and hooks for those who simply want to
secure the data.
Best regards,
Henson Choi
I do not think that custom SMGR API contradicts to the idea of Data
Transformation.
Do you know about decorator pattern?
If you want to implement i.e. data encryption, you definitely do not
need to write your storage manager from the scratch.
Obviously you can (and should) use standard storage manager (md.c) for
actually performing IO.
But your storage manager can perform some extra action prior of after
IO, for example encrypt data before write and decrypt it after read.
So any pre/post/instead hooks can be easily implemented using custom SMGR.
Opposite unfortunately is not possible. You can not for example
implement encryption+compression using hooks.
But you can easily do it using custom SMGR: this is how compressed file
system (CFS) was implemented in PgPro.
