Hi Kenn, all, Thanks Kenn for boosting the thread. Here is a short status update.
Design changes since the first email (the doc is updated): - The proposal now writes through a thin client over Vespa's /document/v1 HTTP API using requests, which is already a core Beam dependency. pyvespa would pull in docker, lxml, fastcore and httpx, so there is no new dependency and no apache-beam[vespa] extra. pyvespa stays listed as an alternative. - The default operation is a full put (replace by document id), the same as the Qdrant writer's upsert. "update" does a partial update with create=true, using only assign operations, so bundle retries stay idempotent. - Retries: HTTP 429 (Vespa's backpressure signal) is retried with backoff. 5xx responses and connection errors are retried up to max_retries. Other errors, such as a 400 for a field missing from the schema, fail the bundle at once with the document id and Vespa's message. Progress: - A draft PR with a working prototype is up: https://github.com/apache/beam/pull/40881 It includes unit tests and an integration test that starts vespaengine/vespa in testcontainers, deploys a minimal application package and reads the documents back. Both pass locally. It is a draft for discussion, not for merge review yet. - LibenHailu has joined as a collaborator on the tracking issue ( https://github.com/apache/beam/issues/40837). We have split up the remaining implementation questions there, starting with a throughput comparison against pyvespa's feed client, which should help with question 2 below. Questions where maintainers' input would help most: 1. Is Vespa a welcome addition to ml.rag ingestion? 2. Thin /document/v1 client (proposed) or pyvespa as an optional extra? 3. Should the container-based IT run with the Milvus/Qdrant ITs (proposed), or postcommit only? 4. Default operation: full put (proposed), or partial update with create=true? 5. Should the first PR fail the bundle on documents that keep failing, as the Qdrant writer does (proposed)? Or should it emit them as a separate output for a dead-letter sink? Design doc: https://docs.google.com/document/d/1cC5wezcXugmyHDbt-xuP6zrlKia337dJObWNKnZZJvM Feedback is welcome here, on the issue or on the PR. Thanks, Wangshu On Wed, Oct 28, 2026 11:35 AM, Kenneth Knowles <[email protected]> wrote: > Hi Wangshu, > > Thanks for writing this. I am afraid that gmail flagged this thread as > spam, so other people may not have seen it. I'm replying to boost it to the > list if I can. > https://lists.apache.org/thread.html/7jr1l0ojljzdt3p13396nzp2vlbzzlkx > > Kenn > > On Tue, Oct 27, 2026 at 1:33 AM Pang Wangshu <[email protected]> > wrote: > >> Hi all, >> >> Following up on this proposal: I've opened a tracking issue so the >> discussion is also visible on GitHub: >> https://github.com/apache/beam/issues/40837 >> >> It has the same summary, the four open questions, and a link to the >> design doc. Feedback is welcome either here or on the issue. The question >> I'd most like input on is #2 (pyvespa as an optional extra, which pulls in >> docker, lxml, fastcore and httpx, vs. a thin client over Vespa's >> /document/v1 HTTP API), since it decides the dependency footprint of the >> first PR. >> >> Thanks, >> Wangshu >> >> On Sat, Oct 24, 2026 09:49 AM, Pang Wangshu <[email protected]> >> wrote: >> >>> Hi Beam devs, >>> >>> I'd like to propose adding Vespa as a write target for the Python RAG >>> ingestion module (apache_beam.ml.rag.ingestion), alongside the existing >>> BigQuery, Postgres/AlloyDB/Cloud SQL, MySQL, Spanner, Milvus and Qdrant >>> configs. I'm currently working on the Qdrant enrichment handler (#40629) >>> and would like to follow it with Vespa. >>> >>> The short version: >>> - A VespaWriteConfig that plugs into VectorDatabaseWriteTransform and >>> maps EmbeddableItem to Vespa document fields: id, content, metadata, dense >>> embedding (indexed tensor) and sparse embedding (mapped tensor). >>> - Writes go through pyvespa's feed_iterable from a batching DoFn, with >>> per-document error collection; puts are idempotent by document id, so >>> bundle retries are safe. >>> - The module only feeds documents. Users deploy their own Vespa >>> application package and schema. >>> - pyvespa (Apache-2.0) as a new optional extra, apache-beam[vespa]. >>> - A retrieval (enrichment) handler would come as a separate follow-up PR. >>> >>> Design doc (open for comments): >>> https://docs.google.com/document/d/1cC5wezcXugmyHDbt-xuP6zrlKia337dJObWNKnZZJvM >>> >>> >>> A few questions where I'd value maintainers' input: >>> 1. Is Vespa a welcome addition to ml.rag ingestion? >>> 2. pyvespa as an optional extra, or a thin client over Vespa's >>> /document/v1 HTTP API to keep dependencies small? >>> 3. testcontainers-python has no Vespa module, so the IT would start >>> vespaengine/vespa with a generic container and deploy a minimal app package >>> (tens of seconds). Is it fine to run it with the Milvus/Qdrant ITs, or >>> should it be postcommit only? >>> 4. Default operation: full put, or partial update with create=true? >>> >>> If there are no objections, I plan to open a GitHub issue and a first PR >>> in early November. >>> >>> Thanks, >>> Wangshu >>> >>
