RE: RE: Re: Re: Vector search problems SolR 9.6 (Lucene 9.10) vs. SolR 9.7 (Lucene 9.11)

2025-01-30 Thread Moll, Dr. Andreas
Hi, just a short follow up on this topic: The Lucene developers found the source of the problem and maybe will fix this issue: https://github.com/apache/lucene/issues/14180 Best regards Andreas

RE: Re: Re: Vector search problems SolR 9.6 (Lucene 9.10) vs. SolR 9.7 (Lucene 9.11)

2025-01-24 Thread Moll, Dr. Andreas
Hi David, Thank you for giving me to a good starting point for a test setup. I have written a Unit Test that should match our usecase but it cannot yet reproduce the error. My guess is that the error may also depend on the status of the segment files. I will give it another try in the coming

Re: Re: Vector search problems SolR 9.6 (Lucene 9.10) vs. SolR 9.7 (Lucene 9.11)

2025-01-23 Thread David Smiley
Hi Andreas, The test org.apache.solr.CursorPagingTest is a single-core test for cursorMark. Alternatively, perhaps you might want to modify a KNN query test like org.apache.solr.search.neural.KnnQParserTest. BTW, the correct capitalization of "Solr" is as I typed it, not as you are typing it. ~

RE: Re: Vector search problems SolR 9.6 (Lucene 9.10) vs. SolR 9.7 (Lucene 9.11)

2025-01-22 Thread Moll, Dr. Andreas
Hi, I was aware of the fact, that the tie breaker "sort by id" would only effect the search after the results of the KNN algorithm come in (which may be non-deterministic), but I guessed it to be a best practice to add it to all queries since e.g. searching with a cursor mark requires the tie

Re: Vector search problems SolR 9.6 (Lucene 9.10) vs. SolR 9.7 (Lucene 9.11)

2025-01-21 Thread Houston Putman
We have seen this, and when we tested against Solr 9.8 (currently being released), the error went away. It turned out to be a weird thing about the search executor, but we couldn't necessarily narrow down why it happened. Anyway, please test out with the new Solr 9.8.0 release when it is available

Re: Vector search problems SolR 9.6 (Lucene 9.10) vs. SolR 9.7 (Lucene 9.11)

2025-01-21 Thread Alessandro Benedetti
I'm not sure the KNN search supports multiple sort conditions. I should do some deep dive in the Lucene code but I don't have time in the foreseeable short future. I can imagine that anyway it would only support 're-ranking' the retrieved topK by the additional score condition, this can not affect

Vector search problems SolR 9.6 (Lucene 9.10) vs. SolR 9.7 (Lucene 9.11)

2025-01-21 Thread Moll, Dr. Andreas
Hi, I want to inform you about a behavior change in SolR 9.6 (Lucene 9.10) vs. SolR 9.7 (Lucene 9.11) for vector searches. We heavily rely on vector searches for embeddings in combination with filter queries on the parent documents. Our queries in general looked like this: select?q={ knn f=vector