alessandrobenedetti commented on code in PR #3468: URL: https://github.com/apache/solr/pull/3468#discussion_r2336867659
########## solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc: ########## @@ -316,6 +316,70 @@ preserved when `stored` is true. + Accepted values: `BOOLEAN` +=== BinaryBitQuantizedDenseVectorField + +Binary bit quantization is a different quantization technique that is even more aggressive in its compression and is able +to reduce each dimension from a 32 bit float down to a single bit. This is done by normalizing each dimension of a vector +relative to a centroid (mid-point pre-calculated against all vectors in the index) with the stored bit +representing whether the actual value is positive or negative of the centroid's value. A further "corrective factor" is computed +and stored to help compensate accuracy in the estimated distance. For more details, the original white paper can be viewed +https://arxiv.org/pdf/2405.12497[here]. Review Comment: So, in regards to citing the paper(s), I remember there has been a very long discussion at the time the contribution ended up in Lucene, between the authors of RaBitQ and Lucene contributors. What ended up being accepted was : > "The binary quantization format used here is a per-vector optimized scalar quantization. These > * ideas are evolutions of LVQ proposed in <a href="https://arxiv.org/abs/2304.04759";>Similarity > * search in the blink of an eye with compressed indices</a> by Cecilia Aguerrebere et al., the > * previous work on globally optimized scalar quantization in Apache Lucene, and <a > * href="https://arxiv.org/abs/1908.10396";>Accelerating Large-Scale Inference with Anisotropic > * Vector Quantization </a> by Ruiqi Guo et. al. Also see {@link > * org.apache.lucene.util.quantization.OptimizedScalarQuantizer}. Some of key features are: > * > * <ul> > * <li>Estimating the distance between two vectors using their centroid centered distance. This > * requires some additional corrective factors, but allows for centroid centering to occur. > * <li>Optimized scalar quantization to single bit level of centroid centered vectors. > * <li>Asymmetric quantization of vectors, where query vectors are quantized to half-byte (4 bits) > * precision (normalized to the centroid) and then compared directly against the single bit > * quantized vectors in the index. > * <li>Transforming the half-byte quantized query vectors in such a way that the comparison with > * single bit vectors can be done with bit arithmetic. > * </ul> > * > * A previous work related to improvements over regular LVQ is <a > * href="https://arxiv.org/abs/2409.09913";>Practical and Asymptotically Optimal Quantization of > * High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Search </a> by > * Jianyang Gao, et. al." (you can find it in the Javadocs of org.apache.lucene.codecs.lucene102.Lucene102BinaryQuantizedVectorsFormat). So given how sensible and long that discussion was, I would encourage replacing the short citation of RaBitQ with what was added in the official Lucene JavaDoc (as that should be a better attribution of merit of all the players involved). @benwtrent, am I correct? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
