Indeed, this went to my spam as well. Chimed in on the doc, but adding this sounds nice - thanks!
Thanks, Dann On Wed, Oct 28, 2026 at 1:34 PM Kenneth Knowles <[email protected]> wrote: > Hi Wangshu, > > Thanks for writing this. I am afraid that gmail flagged this thread as > spam, so other people may not have seen it. I'm replying to boost it to the > list if I can. > https://lists.apache.org/thread.html/7jr1l0ojljzdt3p13396nzp2vlbzzlkx > > Kenn > > On Tue, Oct 27, 2026 at 1:33 AM Pang Wangshu <[email protected]> > wrote: > >> Hi all, >> >> Following up on this proposal: I've opened a tracking issue so the >> discussion is also visible on GitHub: >> https://github.com/apache/beam/issues/40837 >> >> It has the same summary, the four open questions, and a link to the >> design doc. Feedback is welcome either here or on the issue. The question >> I'd most like input on is #2 (pyvespa as an optional extra, which pulls in >> docker, lxml, fastcore and httpx, vs. a thin client over Vespa's >> /document/v1 HTTP API), since it decides the dependency footprint of the >> first PR. >> >> Thanks, >> Wangshu >> >> On Sat, Oct 24, 2026 09:49 AM, Pang Wangshu <[email protected]> >> wrote: >> >>> Hi Beam devs, >>> >>> I'd like to propose adding Vespa as a write target for the Python RAG >>> ingestion module (apache_beam.ml.rag.ingestion), alongside the existing >>> BigQuery, Postgres/AlloyDB/Cloud SQL, MySQL, Spanner, Milvus and Qdrant >>> configs. I'm currently working on the Qdrant enrichment handler (#40629) >>> and would like to follow it with Vespa. >>> >>> The short version: >>> - A VespaWriteConfig that plugs into VectorDatabaseWriteTransform and >>> maps EmbeddableItem to Vespa document fields: id, content, metadata, dense >>> embedding (indexed tensor) and sparse embedding (mapped tensor). >>> - Writes go through pyvespa's feed_iterable from a batching DoFn, with >>> per-document error collection; puts are idempotent by document id, so >>> bundle retries are safe. >>> - The module only feeds documents. Users deploy their own Vespa >>> application package and schema. >>> - pyvespa (Apache-2.0) as a new optional extra, apache-beam[vespa]. >>> - A retrieval (enrichment) handler would come as a separate follow-up PR. >>> >>> Design doc (open for comments): >>> https://docs.google.com/document/d/1cC5wezcXugmyHDbt-xuP6zrlKia337dJObWNKnZZJvM >>> >>> >>> A few questions where I'd value maintainers' input: >>> 1. Is Vespa a welcome addition to ml.rag ingestion? >>> 2. pyvespa as an optional extra, or a thin client over Vespa's >>> /document/v1 HTTP API to keep dependencies small? >>> 3. testcontainers-python has no Vespa module, so the IT would start >>> vespaengine/vespa with a generic container and deploy a minimal app package >>> (tens of seconds). Is it fine to run it with the Milvus/Qdrant ITs, or >>> should it be postcommit only? >>> 4. Default operation: full put, or partial update with create=true? >>> >>> If there are no objections, I plan to open a GitHub issue and a first PR >>> in early November. >>> >>> Thanks, >>> Wangshu >>> >>
