Thanks for the proposal Xinli! It sounds very useful and I also just left some comments.
On Mon, Sep 29, 2025 at 8:42 PM Gang Wu <[email protected]> wrote: > This thread was accidentally in my spam folder. > > I have left some comments with regard to the implication on the Iceberg > rest catalog side. > > Best, > Gang > > On Tue, Sep 30, 2025 at 5:44 AM Huaxin Gao <[email protected]> wrote: > >> Thanks for the proposal. I think it's in the right direction. I left some >> comments and will take another look when time allows. >> >> Huaxin >> >> On 2025/09/27 17:27:29 Xinli shang wrote: >> > Hi all, >> > >> > I’d like to propose adding *native incremental replication* to Iceberg >> > tables. >> > >> > *Motivation:* Many production deployments require cross–data center >> backup >> > and data locality. Today this is usually handled by external services, >> > which adds operational overhead and introduces failure modes outside >> > Iceberg’s transactional boundary. Integrating replication into the >> commit >> > workflow would simplify operations and improve consistency. >> > >> > *Proposal:* An optional replication phase in the commit process would >> > automatically copy data files and metadata to one or more targets (e.g., >> > S3, HDFS, GCS, Azure). Replication is configured via table properties >> and >> > supports both synchronous (immediate consistency, higher latency) and >> > asynchronous (background retries, eventual consistency) modes. This >> > provides built-in disaster recovery, data locality optimization, and >> > cross-region analytics without external tool >> > >> > Full draft proposal with design details is here: >> > 👉 Incremental Iceberg Replication Proposal >> > < >> https://docs.google.com/document/d/1yrVLs0CQyIHs9WbBVx_EK6ad419Adsl9xHozpmQEMrs/edit?tab=t.0#heading=h.aa5ph23raz9l >> > >> > >> > Thanks, >> > Xinli >> > >> >
