Hi all, Following up on this proposal: I've opened a tracking issue so the discussion is also visible on GitHub: https://github.com/apache/beam/issues/40837
It has the same summary, the four open questions, and a link to the design doc. Feedback is welcome either here or on the issue. The question I'd most like input on is #2 (pyvespa as an optional extra, which pulls in docker, lxml, fastcore and httpx, vs. a thin client over Vespa's /document/v1 HTTP API), since it decides the dependency footprint of the first PR. Thanks, Wangshu On Sat, Oct 24, 2026 09:49 AM, Pang Wangshu <[email protected]> wrote: > Hi Beam devs, > > I'd like to propose adding Vespa as a write target for the Python RAG > ingestion module (apache_beam.ml.rag.ingestion), alongside the existing > BigQuery, Postgres/AlloyDB/Cloud SQL, MySQL, Spanner, Milvus and Qdrant > configs. I'm currently working on the Qdrant enrichment handler (#40629) > and would like to follow it with Vespa. > > The short version: > - A VespaWriteConfig that plugs into VectorDatabaseWriteTransform and maps > EmbeddableItem to Vespa document fields: id, content, metadata, dense > embedding (indexed tensor) and sparse embedding (mapped tensor). > - Writes go through pyvespa's feed_iterable from a batching DoFn, with > per-document error collection; puts are idempotent by document id, so > bundle retries are safe. > - The module only feeds documents. Users deploy their own Vespa > application package and schema. > - pyvespa (Apache-2.0) as a new optional extra, apache-beam[vespa]. > - A retrieval (enrichment) handler would come as a separate follow-up PR. > > Design doc (open for comments): > https://docs.google.com/document/d/1cC5wezcXugmyHDbt-xuP6zrlKia337dJObWNKnZZJvM > > > A few questions where I'd value maintainers' input: > 1. Is Vespa a welcome addition to ml.rag ingestion? > 2. pyvespa as an optional extra, or a thin client over Vespa's > /document/v1 HTTP API to keep dependencies small? > 3. testcontainers-python has no Vespa module, so the IT would start > vespaengine/vespa with a generic container and deploy a minimal app package > (tens of seconds). Is it fine to run it with the Milvus/Qdrant ITs, or > should it be postcommit only? > 4. Default operation: full put, or partial update with create=true? > > If there are no objections, I plan to open a GitHub issue and a first PR > in early November. > > Thanks, > Wangshu >
