I guess what I'm missing here is why cross-node coordination belongs in Lucene, which generally only concerns itself with operations on a single node. Maybe I haven't properly understood the proposal? My understanding is that the new classes would operate in a coordinator/collector/collator node that is responsible for calling services running on shards and then combining the results: is that right?
On Sat, Aug 15, 2026 at 8:02 PM Kristian Rickert <[email protected]> wrote: > > Hi lucene devs, > > With the advice of some lucene devs, I've been developing a shared-floor kNN > collection for Lucene. Having a shared floor allows searches to have a > collaborative results live over multiple shards. I've created a PoC search so > a gRPC bidi coordinator stream can judge if it has collected the correct > number of K values, rather than requiring all K values in an HNSW search as > standard distributed search engines demand. > > The surface area is small, and tests show no impact on baseline performance > or recall. After developing and testing this for nearly a year, I have seen > latency decrease by over 50% in high-latency environments for large values of > K. For values of K < 100, the results vary but not any better or worse in > most setups. In low latency environments (fast machines), the feature > provides minimal impact for low values of K (under 100). More testing is > needed to see if multiple machines could add value. Since I only have 2 fast > machines so I can't test distributed shards with more. > > On a fleet of raspberry pis, it helps tremendously with multiple shards. > > I am still tweaking the implementation and I'm preparing for a large-scale > test using over 500GB of text across 8 shards (I've tested 100GB with > success). However, I would like to get a review at this stage to understand > what else would be required to land this feature on the mainline. Or if > someone has a fleet of fast machines, we can test a PoC search engine that > collaborates lucene results. > > Also, I'd love some feedback and see if we can get next steps. > > Could someone please review the code and provide feedback? > > Best regards, > Kristian Rickert > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
