Hi Wangshu, Thanks for writing this. I am afraid that gmail flagged this thread as spam, so other people may not have seen it. I'm replying to boost it to the list if I can. https://lists.apache.org/thread.html/7jr1l0ojljzdt3p13396nzp2vlbzzlkx
Kenn On Tue, Oct 27, 2026 at 1:33 AM Pang Wangshu <[email protected]> wrote: > Hi all, > > Following up on this proposal: I've opened a tracking issue so the > discussion is also visible on GitHub: > https://github.com/apache/beam/issues/40837 > > It has the same summary, the four open questions, and a link to the design > doc. Feedback is welcome either here or on the issue. The question I'd most > like input on is #2 (pyvespa as an optional extra, which pulls in docker, > lxml, fastcore and httpx, vs. a thin client over Vespa's /document/v1 HTTP > API), since it decides the dependency footprint of the first PR. > > Thanks, > Wangshu > > On Sat, Oct 24, 2026 09:49 AM, Pang Wangshu <[email protected]> wrote: > >> Hi Beam devs, >> >> I'd like to propose adding Vespa as a write target for the Python RAG >> ingestion module (apache_beam.ml.rag.ingestion), alongside the existing >> BigQuery, Postgres/AlloyDB/Cloud SQL, MySQL, Spanner, Milvus and Qdrant >> configs. I'm currently working on the Qdrant enrichment handler (#40629) >> and would like to follow it with Vespa. >> >> The short version: >> - A VespaWriteConfig that plugs into VectorDatabaseWriteTransform and >> maps EmbeddableItem to Vespa document fields: id, content, metadata, dense >> embedding (indexed tensor) and sparse embedding (mapped tensor). >> - Writes go through pyvespa's feed_iterable from a batching DoFn, with >> per-document error collection; puts are idempotent by document id, so >> bundle retries are safe. >> - The module only feeds documents. Users deploy their own Vespa >> application package and schema. >> - pyvespa (Apache-2.0) as a new optional extra, apache-beam[vespa]. >> - A retrieval (enrichment) handler would come as a separate follow-up PR. >> >> Design doc (open for comments): >> https://docs.google.com/document/d/1cC5wezcXugmyHDbt-xuP6zrlKia337dJObWNKnZZJvM >> >> >> A few questions where I'd value maintainers' input: >> 1. Is Vespa a welcome addition to ml.rag ingestion? >> 2. pyvespa as an optional extra, or a thin client over Vespa's >> /document/v1 HTTP API to keep dependencies small? >> 3. testcontainers-python has no Vespa module, so the IT would start >> vespaengine/vespa with a generic container and deploy a minimal app package >> (tens of seconds). Is it fine to run it with the Milvus/Qdrant ITs, or >> should it be postcommit only? >> 4. Default operation: full put, or partial update with create=true? >> >> If there are no objections, I plan to open a GitHub issue and a first PR >> in early November. >> >> Thanks, >> Wangshu >> >
