Hi all, I'd like to give a short update on the Iceberg design discussed earlier and ask for another round of feedback.
Since the original proposal, the design has changed significantly based on the review and discussion in PR #1842. The biggest change is that we no longer plan to introduce Iceberg-specific SQL syntax or new system catalogs into the Cloudberry kernel. The earlier approach used syntax such as: CREATE FOREIGN CATALOG ... CREATE FOREIGN VOLUME ... CREATE ICEBERG TABLE ... During the review, we discussed whether Iceberg-specific syntax should live in the core when the actual Iceberg implementation is provided by an extension. We have now moved away from that approach. The current design is implemented entirely as a datalake_fdw extension and uses existing PostgreSQL interfaces and syntax: CREATE SERVER cat FOREIGN DATA WRAPPER iceberg_catalog_fdw OPTIONS (...); CREATE SERVER vol FOREIGN DATA WRAPPER iceberg_volume_fdw OPTIONS (...); CREATE TABLE t (a int, b text) USING iceberg WITH (catalog = 'cat', volume = 'vol'); In other words: - CREATE TABLE ... USING iceberg uses the existing Table AM syntax. - Catalog and storage configuration use existing Foreign Server / User Mapping infrastructure. - No new parser syntax is required. - No new Iceberg-specific system catalogs are added to the kernel. - The Iceberg implementation, including the Table AM and the catalog/volume wrappers, lives in the extension. PR #1842 has been repurposed to reflect this direction: https://github.com/apache/cloudberry/pull/1842 The current PR is intentionally still a skeleton. CREATE TABLE and DROP TABLE work end-to-end against a stub metadata engine, while scan/write/catalog/object-storage implementations will be added incrementally. At this stage, the main thing I'd like to get feedback on is the overall extension boundary before we continue adding the full implementation. In particular: 1. Does using CREATE TABLE ... USING iceberg as the SQL entry point look like the right direction? 2. Does representing Catalog and Volume using PostgreSQL foreign servers provide a reasonable abstraction? 3. Does keeping the whole Iceberg subsystem in an extension, without introducing Iceberg-specific kernel objects, look like a better long-term direction? The original architecture discussion is still here: https://github.com/apache/cloudberry/discussions/1683 Feedback on the updated direction would be very helpful before we move further with the implementation. Thanks, Xiaoyu Liu On Mon, 20 Apr 2026 15:59:34 +0800, xiaoyu liu <[email protected]> wrote: > Hi all, > > I'd like to propose adding native Apache Iceberg support to Cloudberry > as part of the datalake_fdw extension, and to collect community feedback > before we open-source the implementation. > > == Motivation == > > Cloudberry today lacks a transactional read/write entry point for lake > formats. The existing PXF-based FDW is read-mostly, has no Catalog > concept, and cannot share snapshots with the wider Iceberg ecosystem > (Spark / Trino / Flink). This proposal introduces Iceberg tables as > first-class "lake tables" in CB with full SELECT / INSERT / UPDATE / > DELETE / VACUUM, Schema Evolution, and Read Committed isolation, while > keeping metadata format fully compatible with the community. > > == Key design points == > > * Iceberg Table AM — Iceberg tables register as a dedicated Table AM > (not a pure FDW), so UPDATE/DELETE/ctid and transactional semantics > come for free; data I/O is delegated to a Volume FDW. > > * Catalog × Volume × Table abstraction — Catalog (Polaris / Hive / > Builtin) and Volume (S3 / HDFS) are two separate FDWs, freely > composable. A Builtin Catalog is provided for zero-dependency > deployments. > > * Metadata delegated to an out-of-process Java agent — datalake_agent > wraps iceberg-java and runs as a jar launched and supervised by a PG > bgworker (datalake_proxy). The PG↔agent RPC channel is gRPC. This > keeps wire-format compatibility with the community and makes Iceberg > version upgrades a jar swap. > > * Transactional semantics — a per-transaction Metadata Tracker maps > Iceberg's optimistic CAS onto PG's MVCC, supporting > Read-Your-Own-Writes, Read Committed across concurrent writers, > SAVEPOINT, and asynchronous cleanup of orphaned files. > > * MPP execution — only the QD talks to the agent; data read/write is > parallel on QEs directly against object storage. > > == Read more / discuss == > > The full design document (architecture, end-to-end flows, MPP model, > limitations and open questions) and the discussion thread are here: > > https://github.com/apache/cloudberry/discussions/1683 > > The design doc ends with a list of review topics we're > most interested in hearing opinions on. Looking forward to your > feedback on the thread. > > Thanks, > Xiaoyu Liu --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
