Hi lucene devs, With the advice of some lucene devs, I've been developing a shared-floor kNN collection <https://github.com/apache/lucene/pull/16357> for Lucene. Having a shared floor allows searches to have a collaborative results live over multiple shards. I've created a PoC search so a gRPC bidi coordinator stream can judge if it has collected the correct number of K values, rather than requiring all K values in an HNSW search as standard distributed search engines demand.
The surface area is small, and tests show no impact on baseline performance or recall. After developing and testing this for nearly a year, I have seen latency decrease by over 50% in high-latency environments for large values of K. For values of K < 100, the results vary but not any better or worse in most setups. In low latency environments (fast machines), the feature provides minimal impact for low values of K (under 100). More testing is needed to see if multiple machines could add value. Since I only have 2 fast machines so I can't test distributed shards with more. On a fleet of raspberry pis, it helps tremendously with multiple shards. I am still tweaking the implementation and I'm preparing for a large-scale test using over 500GB of text across 8 shards (I've tested 100GB with success). However, I would like to get a review at this stage to understand what else would be required to land this feature on the mainline. Or if someone has a fleet of fast machines, we can test a PoC search engine that collaborates lucene results. Also, I'd love some feedback and see if we can get next steps. Could someone please review the code and provide feedback? <https://github.com/apache/lucene/pull/16357> Best regards, Kristian Rickert
