GitHub user MisterRaindrop added a comment to the discussion: [DISCUSSION]
Making Apache Cloudberry an Agent-Native Analytical Database
Thanks for the suggestion. I ran a local correctness spike with `pg_duckdb` and
DuckDB’s official `lance` extension.
The test environment was Cloudberry 3.0.0-devel / PostgreSQL 16.9 with three
primary segments, using a temporary compatibility port of `pg_duckdb` main and
DuckDB/Lance 1.5.4. This is not yet a performance comparison with `lance_fdw`;
the goal was first to verify the execution and resource-management boundaries.
### 1. Prefiltered vector Top-K works, but runs on the QD
A filtered query using:
```sql
lance_vector_search(..., k = 2, prefilter = true)
```
returned the expected two rows from a five-row Lance dataset.
I added probes to the `CustomScan` begin and execute callbacks. Every
invocation reported the dispatcher role with `segindex = -1`; there were no
corresponding executions on the QEs.
The full Lance scan also returned exactly five rows, so there was no duplicate
scan in this path. However, this is because the scan ran only once on the
QD—not because fragments were assigned across segments.
The Top-K is therefore global only in the single-process DuckDB sense:
```text
QD backend
→ DuckDB
→ Lance vector search over the whole dataset
→ final Top-K
```
There are no per-QE candidate sets and no Cloudberry `Gather Motion ... Merge
Key` stage. This can be reasonable for small retrieval results, but it is not
distributed Top-K.
### 2. Joining with a distributed Cloudberry table does not currently work
I also tested a Lance scan joined with a native Cloudberry table distributed
across all three segments.
DuckDB’s PostgreSQL table reader invoked the Cloudberry planner and produced:
```text
Gather Motion 3:1
-> Seq Scan on native_dim
Filter: id >= 1 AND id <= 5
```
However, execution then failed with:
```text
Unexpected Motion Node Id: 1
```
The outer `DuckDBScan` plan does not contain the Cloudberry slice/interconnect
metadata required to execute that nested Motion.
So the current limitation is stronger than merely losing parallelism: a
distributed native-table join needs explicit Cloudberry planner/executor
integration before it can run correctly.
### 3. DuckDB memory is outside Cloudberry’s current accounting
For a DuckDB query with a 512 MB DuckDB limit, the QD backend reached
approximately:
```text
process RSS: 605 MB
Cloudberry tracked vmem: 15 MB
active QEs: 0
```
This confirms that DuckDB’s allocator is not accounted through Cloudberry’s
`VmemTracker`.
DuckDB’s own limit still works independently. With `duckdb.max_memory = 64 MB`,
a larger allocation failed at about 50.2 MB used with a DuckDB out-of-memory
error.
A production integration would therefore need an explicit memory-grant bridge:
- obtain a query/session grant from Cloudberry before starting DuckDB;
- derive `duckdb.max_memory` from that grant;
- account and release allocations through Cloudberry on success, cancellation
and error;
- also bound DuckDB threads and temporary spill space.
Simply replacing DuckDB allocations with `gp_malloc` is probably insufficient
because DuckDB uses worker threads, while Cloudberry’s current
memory-protection path expects allocations from the backend owner thread.
### Compatibility note
`pg_duckdb` did not compile or install on this Cloudberry build without
PostgreSQL/Cloudberry API compatibility changes. The extension installation SQL
also required a trigger compatibility adjustment. Some dynamic `duckdb.query`
result shapes triggered a QD crash in `CdbDispatchPlan` in this temporary port,
so this is not yet a supported integration.
My current conclusion is:
- `pg_duckdb + lance` is technically interesting as an optional QD-only
retrieval path;
- it could reduce maintenance for Lance format, S3, FTS and hybrid-search
functionality;
- it does not currently replace `lance_fdw` for fragment ownership, MPP scans,
joins with distributed tables, distributed Top-K or Cloudberry memory
accounting.
So the two approaches could eventually be complementary, but `lance_fdw` still
covers the Cloudberry-specific execution work that DuckDB/pg_duckdb does not
provide today.
GitHub link:
https://github.com/apache/cloudberry/discussions/1967#discussioncomment-18879715
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
