GitHub user MisterRaindrop edited a comment on the discussion: [DISCUSSION] Making Apache Cloudberry an Agent-Native Analytical Database
## Progress update, and a question about where `lance_fdw` should live A short update on the roadmap agreed above, and one decision I'd like the community's view on before going further. ### Where things stand **Phase 0 – MCP retrieval tools.** Merged in #1991: `list_searchable_columns`, `vector_search`, `fulltext_search` and `hybrid_search` over pgvector and full-text columns. **Phases 1–3 – `lance_fdw`.** Implemented as a separate extension, as agreed here, at [MisterRaindrop/cloudberry-lance](https://github.com/MisterRaindrop/cloudberry-lance). It links a pinned `lance-c` and runs its regression and stability suites in CI against a three-segment cluster. - **Phase 1, read-only scan.** The QD pins a dataset version and assigns every fragment to exactly one segment through `fdw_private`, so `mpp_execute 'all segments'` returns no duplicate and no missing rows. Filters and projections are pushed down when Cloudberry and Lance agree exactly on their meaning. Structs are read as composites and maps as `jsonb`, and Arrow batches are charged to the memory accounting. Local paths and S3 both work. - **Phase 2, vector Top-K pushdown.** `lance_fdw` defines `<->`, `<=>` and `<#>` on `real[]`, with the same names as pgvector, so no pgvector dependency is needed. `WHERE ... ORDER BY emb <-> $1 LIMIT k` becomes one Lance nearest-neighbour search with the filter applied as a prefilter, which was the acceptance test asked for above. - `lance_fdw.nprobes` and `lance_fdw.refine_factor` are exposed. With IVF_PQ, `refine_factor` is what restores recall. - A cancel or `statement_timeout` now reaches a running search: about 20 ms median, where it used to wait for the search to finish (250–400 ms on 4M rows). - **Phase 3, distributed Top-K.** When the index has several segments, the QD sends the search to every segment. Each segment searches a contiguous range of fragments, its covering index segments, and brute-forces unindexed fragments, with no duplicates and no losses; the QD merges the per-segment Top-K. This needed no new `lance-c` API: restricting the search to a fragment set was enough. - With a single-segment index it stays on one segment, and `lance_fdw.vector_search_mode` overrides the choice. - Without an index, on the same host: 1M rows went from 93 to 74 ms, and 4M rows from 329 to 224 ms. **MCP over Lance.** A follow-up PR to `mcp-server` (coming shortly) makes `lance_fdw` foreign tables discoverable, readable and searchable with `vector_search`. It also bounds what one tool call returns, by rows, value length and bytes, and keeps credentials out of `get_table_ddl`. ### The question: separate repository, or `contrib/`? @my-ship-it's point above was to keep Lance out of the core build so that Cloudberry would not need a Rust toolchain, and I agreed. I'd like to check whether that still holds, because two things are now clearer than they were then: 1. **`lance_fdw` is plain C.** All the Rust lives in `lance-c`, the official C library from the Lance project, which ships as an ordinary shared library. The FDW never calls into Rust directly. 2. **The out-of-tree cost is real.** The FDW uses MPP-specific hooks: fragment assignment through `fdw_private`, dispatch to QEs, and the planner path for Top-K. A change in Cloudberry can break it without anyone in this repository noticing, and users have to find and build a second project. That leaves three options I can see: | Option | Cloudberry build | Source release | Cost | |---|---|---|---| | **A. Keep it separate** (current) | unchanged | unchanged | Breakage is found late, and users install a second project | | **B. `contrib/lance_fdw`, built only with `--with-lance`**, linking an installed `liblance_c` the way `--with-libxml` links libxml2 | no Rust or cargo; off by default | C sources only, no Rust crates | CI needs a `liblance_c` binary for x86_64 and aarch64; `configure` checks the pinned `lance-c` version | | **C. `contrib/` with `lance-c` vendored as a submodule** and built by cargo | needs Rust when enabled | must account for several hundred crates in LICENSE/NOTICE | the toolchain discussion we wanted to avoid | My preference is **B**, if the community is open to it. It keeps the core build exactly as it is, and puts the FDW where Cloudberry's own CI exercises it. If the answer is A, that is fine too, and I'll keep developing and releasing it from the separate repository. If B is acceptable, I'd propose landing it as a sequence of reviewable PRs rather than one large one: 1. Build integration (`--with-lance`, off by default), options, `IMPORT FOREIGN SCHEMA`, and the parallel read-only scan with type mapping. 2. Filter and projection pushdown. 3. Vector operators and Top-K pushdown, including cancel during a search. 4. Distributed Top-K. Each PR would come with its regression tests. The extension would start at 1.0 in-tree. Questions for the community: 1. Is **B** acceptable, or should `lance_fdw` stay a separate project (**A**)? 2. Is there a precedent or preference for providing a prebuilt third-party library to Cloudberry's CI? 3. Any objection to that PR split? ### Next: Phase 4 Phase 4 (writing AI-derived results back) is being designed along the lines discussed above: - The extraction runs outside the database, driven by the agent through MCP. - Results go into ordinary Cloudberry tables: one typed column per extracted field, upserted by a source key, with `model` and `extracted_at` columns recording where each row came from. I'll post the design separately. see https://github.com/MisterRaindrop/cloudberry-lance GitHub link: https://github.com/apache/cloudberry/discussions/1967#discussioncomment-18847704 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
