GitHub user yjhjstz edited a discussion: [Proposal] Cloudberry Multi-Catalog
System Design Proposal
### Proposers
@yjhjstz @my-ship-it
### Proposal Status
Under Discussion
### Abstract
## Cloudberry Multi-Catalog System Design Proposal
Based on the comprehensive analysis provided and research into Apache Doris's
Iceberg Catalog implementation, I'll create a detailed proposal for
Cloudberry's multi-catalog system design. The key idea is that no `CREATE
FOREIGN TABLE` is required. [USAGE](#SQL-DDL-USAGE)
## Executive Summary
Apache Cloudberry, as an advanced open-source MPP database derived from
PostgreSQL[1][2], has a unique opportunity to implement a cutting-edge
multi-catalog system that extends its current two-tier naming (database.table)
to a three-tier hierarchy (catalog.database.table)[3]. This proposal outlines a
comprehensive design for a plugin-based catalog framework that enables
federated queries across multiple data sources while maintaining PostgreSQL
compatibility and leveraging Cloudberry's MPP architecture.
### Motivation
## Background and Motivation
Apache Cloudberry currently supports external data access through Foreign Data
Wrappers (FDW)[4][5] and Platform Extension Framework (PXF)[6], but lacks the
unified multi-catalog approach that modern data lakehouse architectures demand.
Apache Doris has successfully implemented multi-catalog functionality[3][7]
that enables seamless integration with external data sources like Iceberg,
Hive, and JDBC databases. This capability is crucial for organizations building
modern data platforms that need to query across heterogeneous data sources.
The proposed multi-catalog system addresses several key challenges:
- **Data Silos**: Enable unified access to data across multiple systems without
data movement
- **Vendor Lock-in**: Provide open-source alternative to proprietary data
federation solutions
- **Performance**: Leverage Cloudberry's MPP architecture for distributed query
execution
- **Standards Compliance**: Maintain PostgreSQL/SQL standards while extending
functionality
### Implementation
## Core Architecture Design
### 1. Three-Tier Naming Hierarchy
The system extends Cloudberry's current naming convention from `database.table`
to `catalog.database.table`, enabling:
- **Backward Compatibility**: Default internal catalog maintains existing
behavior
- **Federated Queries**: Cross-catalog JOINs and analytics
- **Namespace Isolation**: Clear separation between internal and external data
sources
### 2. Plugin-Based Catalog Framework
The architecture follows a layered approach similar to Apache Doris's
multi-catalog implementation[3]:
```
┌─────────────────────────────────────────┐
│ SQL Parser (3-part names) │
│ catalog.database.table │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Metadata Virtualization │
│ Subsystem (MVS) │
├─────────────────────────────────────────┤
│ • Catalog Registry │
│ • Name Resolution Hooks │
│ • Dynamic Object Factory │
│ • Schema/Type Converters │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Plugin Interface Layer │
├─────────────────────────────────────────┤
│ Rust+pgrx │ C++ Native │ Python/Java │
│ Plugins │ Plugins │ Services │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ External Data Sources │
│ Iceberg │ Hive │ JDBC │ S3 │ Delta... │
└─────────────────────────────────────────┘
```
### 3. Metadata Virtualization Subsystem (MVS)
The MVS serves as the core component that manages catalog registration,
metadata caching, and dynamic schema resolution[8]. Key components include:
#### Catalog Registry
- **Global Registry**: Hash table storing registered catalog implementations
- **Thread Safety**: LWLock protection for concurrent access
- **Health Monitoring**: Background process for catalog availability checks
#### Name Resolution Engine
- **Hook Chain**: Pluggable resolution mechanisms for different catalog types
- **Cache Management**: Multi-level caching with TTL-based expiration
- **Dynamic Discovery**: On-demand metadata fetching from external sources
#### Virtual Object Factory
- **Dynamic Relations**: Generate PostgreSQL Relation objects from external
schemas
- **Type Mapping**: Convert external data types to PostgreSQL types
- **Memory Management**: Proper cleanup and resource management
## Rust+pgrx Iceberg Plugin Implementation
### Technical Architecture
The Iceberg plugin leverages modern Rust capabilities through the pgrx
framework[9][10], which provides safe and efficient PostgreSQL extension
development:
```rust
// Core FFI interface for metadata retrieval
#[no_mangle]
pub extern "C" fn get_table_desc_dynamic(
catalog_type: *const c_char,
catalog_config: *const c_char,
database_name: *const c_char,
table_name: *const c_char,
error_code: *mut c_int,
) -> *mut TableDesc
```
### Key Advantages
1. **Performance**: Direct C FFI with zero JVM overhead
2. **Safety**: Rust's memory safety prevents common extension crashes
3. **Ecosystem**: Leverages official iceberg-rust library[11][12]
4. **Maintainability**: Modern language with excellent tooling
### Iceberg Integration Features
Based on Apache Doris's comprehensive Iceberg support[13][14], the plugin will
provide:
- **Catalog Types**: REST, Hive Metastore, AWS Glue, Hadoop
- **Table Formats**: V1/V2 with Position Delete and Equality Delete support
- **Time Travel**: Query historical snapshots
- **Schema Evolution**: Dynamic schema updates without downtime
- **Partition Transforms**: Hidden partitioning and layout evolution
## FDW Integration Strategy
The multi-catalog system integrates seamlessly with Cloudberry's existing
Foreign Data Wrapper infrastructure[4][5]:
### Query Execution Flow
1. **SQL Parsing**: Extended parser recognizes three-part names
2. **Analyze Phase**: MVS hooks resolve external table metadata
3. **Planning**: Standard PostgreSQL planner with foreign table support
4. **Execution**: FDW-based data scanning with MPP distribution
### MPP-Aware Design
- **Segment Distribution**: Metadata pushed to all query executors
- **Parallel Scanning**: Distributed reading of external data sources
- **Plan Serialization**: Schema information embedded in execution plans
### SQL-DDL-USAGE
```sql
-- Create external catalog
CREATE CATALOG iceberg_catalog WITH (
TYPE = 'iceberg',
CATALOG_TYPE = 'rest',
CATALOG_URI = 'http://localhost:8181',
WAREHOUSE = 's3://my-bucket/warehouse',
CACHE_TTL = 300
);
-- Query across catalogs
SELECT i.product_id, c.customer_name, i.sales_amount
FROM iceberg_catalog.sales.transactions i
JOIN internal.crm.customers c ON i.customer_id = c.id;
```
## Performance Optimization
### Multi-Level Caching Strategy
1. **Process-Level Cache**: Fast local metadata cache with TTL
2. **Shared Memory Cache**: Cross-process metadata sharing
3. **Coordinator Distribution**: QD pushes metadata to segments
### Query Optimization
- **Predicate Pushdown**: Push filters to external data sources
- **Partition Pruning**: Leverage external partitioning information
- **Statistics Integration**: Use external table statistics for cost-based
optimization
## Implementation Roadmap
### Phase 1: Core Infrastructure
- Metadata Virtualization Subsystem implementation
- Catalog registry and name resolution engine
- Basic three-part name SQL parsing support
### Phase 2: Rust Iceberg Plugin
- pgrx-based Iceberg plugin development
- FFI interface and async runtime setup
- Basic REST catalog support
### Phase 3: FDW Integration
- FDW handler implementation
- Query planning and execution integration
- MPP-aware metadata distribution
### Phase 4: Advanced Features
- Additional catalog types (Hive, AWS Glue)
- Performance optimizations
- Comprehensive testing and documentation
### Phase 5: Production Hardening
- Error handling and recovery
- Monitoring and observability
- Security and access control
## Benefits and Impact
### For Users
- **Unified Data Access**: Query across multiple data sources with SQL
- **No Vendor Lock-in**: Open-source alternative to proprietary solutions
- **Performance**: Leverage Cloudberry's MPP architecture
- **PostgreSQL Compatibility**: Familiar SQL interface and tooling
### For Ecosystem
- **Modern Data Architecture**: Enable lakehouse patterns with open formats
- **Community Growth**: Attract users from data lake and analytics space
- **Innovation Platform**: Foundation for future data federation features
### Technical Advantages
- **Rust Safety**: Memory-safe extensions prevent database crashes
- **Async Performance**: Non-blocking metadata operations
- **Cloud Native**: Designed for object storage and containerized deployments
## Conclusion
This multi-catalog system design positions Apache Cloudberry as a leading
open-source data analytics platform capable of competing with modern data
lakehouse solutions. By leveraging proven architectural patterns from Apache
Doris[13][3] and implementing them with modern technologies like Rust and
pgrx[9][10], Cloudberry can provide a robust, performant, and maintainable
solution for federated data analytics.
The proposed system maintains full backward compatibility while enabling
powerful new capabilities for modern data architectures. The plugin-based
approach ensures extensibility, while the Rust implementation provides safety
and performance that traditional C extensions cannot match.
This proposal represents a significant step forward for Apache Cloudberry in
the competitive data analytics landscape, providing users with the tools they
need to build modern, scalable data platforms.
## References
[1] Apache Cloudberry (Incubating) - The Apache Software Foundation
[13] When Doris Meets Iceberg: A Data Engineer's Redemption
[14] Using Doris and Iceberg - Apache Doris
[2] Apache Cloudberry Feature Overview
[6] apache/cloudberry-pxf: Platform Extension Framework
[3] Build a federated query solution with Apache Doris
[4] ALTER FOREIGN DATA WRAPPER - Apache Cloudberry
[5] CREATE FOREIGN DATA WRAPPER - Apache Cloudberry
[7] Apache Doris speeds up data reporting
[9] Develop Database Extensions Using PGRX - Apache Cloudberry
[11] iceberg - Rust - Official Apache Iceberg implementation
[10] pgcentralfoundation/pgrx: Build Postgres Extensions with Rust
[12] apache/iceberg-rust - Official Apache Iceberg Rust implementation
[8] Metadata and Data Virtualization Explained - Altoros
### Rollout/Adoption Plan
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/1297
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@cloudberry.apache.org
For additional commands, e-mail: dev-h...@cloudberry.apache.org
