Just to follow up on that -- the general idea of using statistical estimates of the distribution of results among shards to reduce top-k sent to worker nodes make lots of sense (I guess followed up by a retry on select nodes if needed?) -- fully in support of this general idea. However, if that's what this is about, including classes to do that in Lucene cannot really present a complete solution since Lucene doesn't make requests in a distributed system; therefore (1) testing is best-effort simulation, (2) most distributed systems probably already have their own ways of doing this. I guess we can provide reference classes, but I think past attempts to do things like this (see replication module) haven't really evolved in a healthy way since they don't get used by the major distributed systems. Have you considered contributing the solution to Solr, for example? Do you know how OpenSearch and Elastic handle this? Would they be potential consumers?
On Sun, Aug 16, 2026 at 12:58 PM Michael Sokolov <[email protected]> wrote: > > I guess what I'm missing here is why cross-node coordination belongs > in Lucene, which generally only concerns itself with operations on a > single node. Maybe I haven't properly understood the proposal? My > understanding is that the new classes would operate in a > coordinator/collector/collator node that is responsible for calling > services running on shards and then combining the results: is that > right? > > On Sat, Aug 15, 2026 at 8:02 PM Kristian Rickert <[email protected]> wrote: > > > > Hi lucene devs, > > > > With the advice of some lucene devs, I've been developing a shared-floor > > kNN collection for Lucene. Having a shared floor allows searches to have a > > collaborative results live over multiple shards. I've created a PoC search > > so a gRPC bidi coordinator stream can judge if it has collected the correct > > number of K values, rather than requiring all K values in an HNSW search as > > standard distributed search engines demand. > > > > The surface area is small, and tests show no impact on baseline performance > > or recall. After developing and testing this for nearly a year, I have seen > > latency decrease by over 50% in high-latency environments for large values > > of K. For values of K < 100, the results vary but not any better or worse > > in most setups. In low latency environments (fast machines), the feature > > provides minimal impact for low values of K (under 100). More testing is > > needed to see if multiple machines could add value. Since I only have 2 > > fast machines so I can't test distributed shards with more. > > > > On a fleet of raspberry pis, it helps tremendously with multiple shards. > > > > I am still tweaking the implementation and I'm preparing for a large-scale > > test using over 500GB of text across 8 shards (I've tested 100GB with > > success). However, I would like to get a review at this stage to understand > > what else would be required to land this feature on the mainline. Or if > > someone has a fleet of fast machines, we can test a PoC search engine that > > collaborates lucene results. > > > > Also, I'd love some feedback and see if we can get next steps. > > > > Could someone please review the code and provide feedback? > > > > Best regards, > > Kristian Rickert > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
