Kevin Liang created SOLR-18424:
----------------------------------
Summary: Add native rerank for dense vectors
Key: SOLR-18424
URL: https://issues.apache.org/jira/browse/SOLR-18424
Project: Solr
Issue Type: Improvement
Reporter: Kevin Liang
Solr 10 has introduced support for binary quantized dense vectors (SOLR-17781),
however binary quantized vectors by themselves have a pretty noticeable recall
hit (only achieving ~60% recall compared to non-quantized KNN by some
benchmarks). The solution is to pair binary quantization with oversampling +
rerank phase. While it is possible to do this currently with Solr's generic
rerank function through something like:
{code:java}
q={!knn f=vector_quantized topK=100}[...]&rq={!rerank reRankDocs=100
reRankQuery=$rrq reRankWeight=1.0
reRankOperator=replace}&rrq={!func}vectorSimilarity(FLOAT32, DOT_PRODUCT,
vector_quantized, [...])&rows=10{code}
this is verbose, cumbersome, and easy to get subtley wrong behavior. Propose
that we support reranking directly in the knn query parser via an "oversample"
factor param such as
{code:java}
q={!knn f=vector_quantized topK=10 oversample=10}[0.12, 0.34, ...]{code}
"oversample" here meaning retrieve n * k documents, rerank with raw vectors,
and return top k.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
