kaivalnp commented on PR #14863: URL: https://github.com/apache/lucene/pull/14863#issuecomment-3276205689
Thanks @benwtrent ! I realized that this PR would give incorrect results if 8-bit quantization was used (added recently in #15148) -- because it used the `dotProduct` / `squareDistance` functions which assume input bytes to be signed I switched them over to `uint8DotProduct` and `uint8SquareDistance` for correctness with 8-bit quantization. I think the previous results with 7-bit quantization still hold, because the signed / unsigned functions produce the same output for 7-bit integers I'll try to run a test with 8-bit quantization too, I realized this PR will implicitly support it :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
