On 28/12/2025 9:49 AM, Henson Choi wrote:
RFC: PostgreSQL Storage I/O Transformation Hooks
Infrastructure for a Technical Protocol Between RDBMS Core and
Data Security Experts
*Author:* Henson Choi [email protected]
*Date:* 2025-12-28
*PostgreSQL Version:* master (Development)
------------------------------------------------------------------------
1. Summary & Motivation
This RFC proposes the introduction of minimal hooks into the
PostgreSQL storage layer and the addition of a *Transformation ID*
field to the |PageHeader|.
A Diplomatic Protocol Between Expert Groups
The core motivation of this proposal is *“Separation of Concerns and
Mutual Respect.”*
Historically, discussions around Transparent Data Encryption (TDE)
have often felt like putting security experts on trial in a foreign
court—specifically, the “Court of RDBMS.” It is time to treat them not
as defendants to be judged by database-specific rules, but as an
*equal neighboring community* with their own specialized sovereignty.
*The issue has never been a failure of technology, but rather a
misplacement of the focal point.* While previous discussions were
mired in the technicalities of “how to hardcode encryption into the
core,” this proposal shifts the debate toward an architectural
solution: “what interface the core should provide to external experts.”
* *RDBMS Experts* provide a trusted pipeline responsible for data
I/O paths and consistency.
* *Security Experts* take responsibility for the specialized domain
of encryption algorithms and key management.
This hook system functions as a *Technical Protocol*—a high-level
agreement that allows these two expert groups to exchange data
securely without encroaching on each other’s territory.
------------------------------------------------------------------------
2. Design Principles
1. *Delegation of Authority:* The core remains independent of
specific encryption standards, providing a “free territory” where
security experts can respond to an ever-changing security landscape.
2. *Diplomatic Convention:* The Transformation ID acts as a
communication protocol between the engine and the extension. The
engine uses this ID to identify the state of the data and hands
over control to the appropriate expert (the extension).
3. *Minimal Interference:* Overhead is kept near zero when hooks are
not in use, ensuring the native performance of the PostgreSQL engine.
------------------------------------------------------------------------
3. Proposal Specifications
3.1 The Interface (Hook Points)
We allow intervention by security experts through five contact points
along the I/O path:
* *Read/Write Hooks:* |mdread_post|, |mdwrite_pre|, |mdextend_pre|
(Transformation of the data area)
* *WAL Hooks:* |xlog_insert_pre|, |xlog_decode_pre| (Transformation
of transaction logs)
3.2 The Protocol Identifier (PageHeader Transformation ID)
We allocate 5 bits of |pd_flags| to define the “Security State” of a
page. This serves as a *Status Message* sent by the security expert to
the engine, utilized for key versioning and as a migration marker.
------------------------------------------------------------------------
4. Reference Implementation: |contrib/test_tde|
A Standard Code of Conduct for Security Experts
This reference implementation exists not as a commercial product, but
to define the *Standards of the Diplomatic Protocol* that
encryption/decryption experts must follow when entering the PostgreSQL
domain.
1. *Deterministic IV Derivation:* Demonstrates how to achieve
cryptographic safety by trusting unique values provided by the
engine (e.g., LSN).
2. *Critical Section Safety:* Defines memory management regulations
that security logic must follow within “Critical Sections” to
maintain system stability.
3. *Hook Chaining:* Demonstrates a cooperative structure that allows
peaceful coexistence with other expert tools (e.g., compression,
auditing).
------------------------------------------------------------------------
5. Scope
* *In-Scope:* Backend hook infrastructure, Transformation ID field,
and reference code demonstrating diplomatic protocol compliance.
* *Out-of-Scope:* Specific Key Management Systems (KMS), selection
of specific cryptographic algorithms, and integration with
external tools.
This proposal represents a strategic diplomatic choice: rather than
the PostgreSQL core assuming all security responsibilities, it grants
security experts a *sovereign territory through extensions* where they
can perform at their best.
I wonder if instead of support a lot of extra hooks it will be better to
provide extensible SMGR API:
https://www.postgresql.org/message-id/flat/CAPP%3DHha_wV1MV9yR70QZ5pk5dtNP%2BbOyBiFxPmrMKqnQeKMAwQ%40mail.gmail.com#ab0da3412525c7501ea17f3d4c602bbf
It seems to be much more straightforward, convenient and flexible
mechanism than adding hooks, which can be used for many other purposes
except transparent encryption.
