I agree, you can definitely raise a bug for the debug, if you do me a favour and also test in no Cloud mode, it will help us to understand if it's a Solr bug or Lucene bug.
I also agree with your second point about the functional expectations, that is a very minor though, you can create the ticket and contribute a fix if you like, happy to review it! -------------------------- *Alessandro Benedetti* Director @ Sease Ltd. *Apache Lucene/Solr Committer* *Apache Solr PMC Member* e-mail: a.benede...@sease.io *Sease* - Information Retrieval Applied Consulting | Training | Open Source Website: Sease.io <http://sease.io/> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter <https://twitter.com/seaseltd> | Youtube <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github <https://github.com/seaseltd> On Thu, 19 Oct 2023 at 17:51, Mirko Sertic <mirko.ser...@web.de> wrote: > I've prepared a testcase. Given the following documents with > TESTEMBEDDING_EU_3 is a DenseVectorField with length 3 and euclidean > distance function. They are written to a collection made of two shards > with no further routing strategy, so they should be more or less evenly > distributed between the two shards: > > { > id: 'Position1', > TESTEMBEDDING_EU_3: [0, 0, 0] > } > { > id: 'Position2', > TESTEMBEDDING_EU_3: [0.1, 0.1, 0.1] > } > { > id: 'Position3', > TESTEMBEDDING_EU_3: [0.2, 0.2, 0.2] > } > { > id: 'Position4', > TESTEMBEDDING_EU_3: [0.3, 0.3, 0.3] > } > { > id: 'Position5', > TESTEMBEDDING_EU_3: [0.4, 0.4, 0.4] > } > { > id: 'Position6', > TESTEMBEDDING_EU_3: [0.5, 0.5, 0.5] > } > { > id: 'Position7', > TESTEMBEDDING_EU_3: [0.6, 0.6, 0.6] > } > { > id: 'Position8', > TESTEMBEDDING_EU_3: [0.7, 0.7, 0.7] > } > { > id: 'Position9', > TESTEMBEDDING_EU_3: [0.8, 0.8, 0.8] > } > { > id: 'Position10', > TESTEMBEDDING_EU_3: [0.9, 0.9, 0.9] > } > { > id: 'Position11', > TESTEMBEDDING_EU_3: [1.0, 1.0, 1.0] > } > > How I'll do a {!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0] query. > I'd expect a result with 3 documents, id:Position11 should be an exact > macht, and the nearst neighbors should be id:Position10 and > id:Position9. I'd also expect that the explain logging should mark these > tree as part of the topK=3. I get the following search result: > > { > "responseHeader": { > "zkConnected": true, > "status": 0, > "QTime": 35 > }, > "response": { > "numFound": 6, > "start": 0, > "maxScore": 1.0, > "numFoundExact": true, > "docs": [ > { > "id": "Position11", > "TESTEMBEDDING_3": [ > "1.0", > "1.0", > "1.0" > ], > "[shard]": > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/ > ", > "[explain]": "0.0 = not in top 3\n", > "score": 1.0 > }, > { > "id": "Position10", > "TESTEMBEDDING_3": [ > "0.9", > "0.9", > "0.9" > ], > "[shard]": > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/ > ", > "[explain]": "0.0 = not in top 3\n", > "score": 0.97087383 > }, > { > "id": "Position9", > "TESTEMBEDDING_3": [ > "0.8", > "0.8", > "0.8" > ], > "[shard]": > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/ > ", > "[explain]": "0.0 = not in top 3\n", > "score": 0.89285713 > }, > { > "id": "Position8", > "TESTEMBEDDING_3": [ > "0.7", > "0.7", > "0.7" > ], > "[shard]": > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/ > ", > "[explain]": "0.0 = not in top 3\n", > "score": 0.78740156 > }, > { > "id": "Position7", > "TESTEMBEDDING_3": [ > "0.6", > "0.6", > "0.6" > ], > "[shard]": > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/ > ", > "[explain]": "0.0 = not in top 3\n", > "score": 0.6756757 > }, > { > "id": "Position6", > "TESTEMBEDDING_3": [ > "0.5", > "0.5", > "0.5" > ], > "[shard]": > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/ > ", > "[explain]": "0.0 = not in top 3\n", > "score": 0.5714286 > } > ] > }, > "spellcheck": { > "suggestions": [], > "collations": [] > }, > "debug": { > "track": { > "rid": > > "fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless-465214", > "EXECUTE_QUERY": { > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/": > > { > "QTime": "23", > "ElapsedTime": "27", > "RequestPurpose": "GET_TOP_IDS,SET_TERM_STATS", > "NumFound": "3", > "Response": "{responseHeader={zkConnected=true, status=0, > QTime=23}, > response={numFound=3,numFoundExact=true,start=0,maxScore=1.0,docs=[SolrDocument{id=Position11, > > score=1.0}, SolrDocument{id=Position10, score=0.97087383}, > SolrDocument{id=Position7, score=0.6756757}]}, sort_values={}, > spellcheck={suggestions={!knn={numFound=0, startOffset=1, endOffset=5, > suggestion=[]}, =testembedding_eu_3={numFound=0, startOffset=7, > endOffset=26, suggestion=[]}, topk={numFound=16, startOffset=28, > endOffset=32, suggestion=[topp, topf, took, tonk, tops, topik, topo, > tock, tope, topa, topi, to k, tork, tolk, topb, topz]}, =3={numFound=0, > startOffset=32, endOffset=34, suggestion=[]}, [1={numFound=0, > startOffset=35, endOffset=37, suggestion=[]}, ,1={numFound=0, > startOffset=39, endOffset=41, suggestion=[]}, ,1={numFound=0, > startOffset=43, endOffset=45, suggestion=[]}}, > collations={collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topp=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topp}, > collationInternalRank=0}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 topf=3}[1.0,1.0,1.0], hits=6, > misspellingsAndCorrections={topk=topf}, collationInternalRank=1000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > took=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=took}, > collationInternalRank=2000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 tonk=3}[1.0,1.0,1.0], hits=6, > misspellingsAndCorrections={topk=tonk}, collationInternalRank=3000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > tops=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tops}, > collationInternalRank=4000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 topik=3}[1.0,1.0,1.0], hits=6, > misspellingsAndCorrections={topk=topik}, collationInternalRank=5000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topo=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topo}, > collationInternalRank=6000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 tock=3}[1.0,1.0,1.0], hits=6, > misspellingsAndCorrections={topk=tock}, collationInternalRank=7000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > tope=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tope}, > collationInternalRank=8000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 topa=3}[1.0,1.0,1.0], hits=6, > misspellingsAndCorrections={topk=topa}, collationInternalRank=9000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topi=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topi}, > collationInternalRank=10000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 to k=3}[1.0,1.0,1.0], hits=6, > misspellingsAndCorrections={topk=to k}, collationInternalRank=11000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > tork=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tork}, > collationInternalRank=12000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 tolk=3}[1.0,1.0,1.0], hits=6, > misspellingsAndCorrections={topk=tolk}, collationInternalRank=13000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topb=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topb}, > collationInternalRank=14000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 topz=3}[1.0,1.0,1.0], hits=6, > misspellingsAndCorrections={topk=topz}, collationInternalRank=15000}}, > originalTerms=[!knn, =testembedding_eu_3, topk, =3, [1, ,1, ,1]}, > debug={timing={time=23.0, prepare={time=0.0, query={time=0.0}, > facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0}, > highlight={time=0.0}, stats={time=0.0}, expand={time=0.0}, > terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}}, > process={time=23.0, query={time=0.0}, facet={time=0.0}, > facet_module={time=0.0}, mlt={time=0.0}, highlight={time=0.0}, > stats={time=0.0}, expand={time=0.0}, terms={time=0.0}, > spellcheck={time=23.0}, debug={time=0.0}}}}}" > }, > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/": > > { > "QTime": "24", > "ElapsedTime": "29", > "RequestPurpose": "GET_TOP_IDS,SET_TERM_STATS", > "NumFound": "3", > "Response": "{responseHeader={zkConnected=true, status=0, > QTime=24}, > response={numFound=3,numFoundExact=true,start=0,maxScore=0.89285713,docs=[SolrDocument{id=Position9, > > score=0.89285713}, SolrDocument{id=Position8, score=0.78740156}, > SolrDocument{id=Position6, score=0.5714286}]}, sort_values={}, > spellcheck={suggestions={!knn={numFound=0, startOffset=1, endOffset=5, > suggestion=[]}, =testembedding_eu_3={numFound=0, startOffset=7, > endOffset=26, suggestion=[]}, topk={numFound=16, startOffset=28, > endOffset=32, suggestion=[topp, took, topf, tops, topo, topik, tonk, > tope, tock, to k, topi, tork, tolk, top5, topc, topka]}, =3={numFound=0, > startOffset=32, endOffset=34, suggestion=[]}, [1={numFound=0, > startOffset=35, endOffset=37, suggestion=[]}, ,1={numFound=0, > startOffset=39, endOffset=41, suggestion=[]}, ,1={numFound=0, > startOffset=43, endOffset=45, suggestion=[]}}, > collations={collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topp=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topp}, > collationInternalRank=0}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 took=3}[1.0,1.0,1.0], hits=5, > misspellingsAndCorrections={topk=took}, collationInternalRank=1000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topf=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topf}, > collationInternalRank=2000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 tops=3}[1.0,1.0,1.0], hits=5, > misspellingsAndCorrections={topk=tops}, collationInternalRank=3000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topo=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topo}, > collationInternalRank=4000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 topik=3}[1.0,1.0,1.0], hits=5, > misspellingsAndCorrections={topk=topik}, collationInternalRank=5000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > tonk=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tonk}, > collationInternalRank=6000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 tope=3}[1.0,1.0,1.0], hits=5, > misspellingsAndCorrections={topk=tope}, collationInternalRank=7000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > tock=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tock}, > collationInternalRank=8000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 to k=3}[1.0,1.0,1.0], hits=5, > misspellingsAndCorrections={topk=to k}, collationInternalRank=9000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topi=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topi}, > collationInternalRank=10000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 tork=3}[1.0,1.0,1.0], hits=5, > misspellingsAndCorrections={topk=tork}, collationInternalRank=11000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > tolk=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tolk}, > collationInternalRank=12000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 top5=3}[1.0,1.0,1.0], hits=5, > misspellingsAndCorrections={topk=top5}, collationInternalRank=13000}, > collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 > topc=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topc}, > collationInternalRank=14000}, collation={collationQuery={!knn > f=TESTEMBEDDING_EU_3 topka=3}[1.0,1.0,1.0], hits=5, > misspellingsAndCorrections={topk=topka}, collationInternalRank=15000}}, > originalTerms=[!knn, =testembedding_eu_3, topk, =3, [1, ,1, ,1]}, > debug={timing={time=24.0, prepare={time=0.0, query={time=0.0}, > facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0}, > highlight={time=0.0}, stats={time=0.0}, expand={time=0.0}, > terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}}, > process={time=24.0, query={time=0.0}, facet={time=0.0}, > facet_module={time=0.0}, mlt={time=0.0}, highlight={time=0.0}, > stats={time=0.0}, expand={time=0.0}, terms={time=0.0}, > spellcheck={time=24.0}, debug={time=0.0}}}}}" > } > }, > "GET_FIELDS": { > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/": > > { > "QTime": "1", > "ElapsedTime": "3", > "RequestPurpose": "GET_FIELDS,GET_DEBUG,SET_TERM_STATS", > "NumFound": "3", > "Response": "{responseHeader={zkConnected=true, status=0, > QTime=1}, > response={numFound=3,numFoundExact=true,start=0,docs=[SolrDocument{id=Position9, > > TESTEMBEDDING_3=[0.8, 0.8, 0.8], > [shard]= > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/, > > [explain]=0.0 = not in top 3\n, score=0.89285713}, > SolrDocument{id=Position8, TESTEMBEDDING_3=[0.7, 0.7, 0.7], > [shard]= > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/, > > [explain]=0.0 = not in top 3\n, score=0.78740156}, > SolrDocument{id=Position6, TESTEMBEDDING_3=[0.5, 0.5, 0.5], > [shard]= > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/, > > [explain]=0.0 = not in top 3\n, score=0.5714286}]}, > debug={rawquerystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0], > querystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0], > parsedquery=KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]), > parsedquery_toString=KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3], > explain={Position9={match=false, value=0.0, description=not in top 3}, > Position8={match=false, value=0.0, description=not in top 3}, > Position6={match=false, value=0.0, description=not in top 3}}, > QParser=KnnQParser, timing={time=1.0, prepare={time=0.0, > query={time=0.0}, facet={time=0.0}, facet_module={time=0.0}, > mlt={time=0.0}, highlight={time=0.0}, stats={time=0.0}, > expand={time=0.0}, terms={time=0.0}, spellcheck={time=0.0}, > debug={time=0.0}}, process={time=0.0, query={time=0.0}, > facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0}, > highlight={time=0.0}, stats={time=0.0}, expand={time=0.0}, > terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}}}}}" > }, > " > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/": > > { > "QTime": "1", > "ElapsedTime": "3", > "RequestPurpose": "GET_FIELDS,GET_DEBUG,SET_TERM_STATS", > "NumFound": "3", > "Response": "{responseHeader={zkConnected=true, status=0, > QTime=1}, > response={numFound=3,numFoundExact=true,start=0,docs=[SolrDocument{id=Position10, > > TESTEMBEDDING_3=[0.9, 0.9, 0.9], > [shard]= > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/, > > [explain]=0.0 = not in top 3\n, score=0.97087383}, > SolrDocument{id=Position11, TESTEMBEDDING_3=[1.0, 1.0, 1.0], > [shard]= > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/, > > [explain]=0.0 = not in top 3\n, score=1.0}, SolrDocument{id=Position7, > TESTEMBEDDING_3=[0.6, 0.6, 0.6], > [shard]= > http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/, > > [explain]=0.0 = not in top 3\n, score=0.6756757}]}, > debug={rawquerystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0], > querystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0], > parsedquery=KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]), > parsedquery_toString=KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3], > explain={Position10={match=false, value=0.0, description=not in top 3}, > Position11={match=false, value=0.0, description=not in top 3}, > Position7={match=false, value=0.0, description=not in top 3}}, > QParser=KnnQParser, timing={time=1.0, prepare={time=0.0, > query={time=0.0}, facet={time=0.0}, facet_module={time=0.0}, > mlt={time=0.0}, highlight={time=0.0}, stats={time=0.0}, > expand={time=0.0}, terms={time=0.0}, spellcheck={time=0.0}, > debug={time=0.0}}, process={time=0.0, query={time=0.0}, > facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0}, > highlight={time=0.0}, stats={time=0.0}, expand={time=0.0}, > terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}}}}}" > } > } > }, > "timing": { > "time": 49.0, > "prepare": { > "time": 0.0, > "query": { > "time": 0.0 > }, > "facet": { > "time": 0.0 > }, > "facet_module": { > "time": 0.0 > }, > "mlt": { > "time": 0.0 > }, > "highlight": { > "time": 0.0 > }, > "stats": { > "time": 0.0 > }, > "expand": { > "time": 0.0 > }, > "terms": { > "time": 0.0 > }, > "spellcheck": { > "time": 0.0 > }, > "debug": { > "time": 0.0 > } > }, > "process": { > "time": 47.0, > "query": { > "time": 0.0 > }, > "facet": { > "time": 0.0 > }, > "facet_module": { > "time": 0.0 > }, > "mlt": { > "time": 0.0 > }, > "highlight": { > "time": 0.0 > }, > "stats": { > "time": 0.0 > }, > "expand": { > "time": 0.0 > }, > "terms": { > "time": 0.0 > }, > "spellcheck": { > "time": 47.0 > }, > "debug": { > "time": 0.0 > } > } > }, > "rawquerystring": "{!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0]", > "querystring": "{!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0]", > "parsedquery": > "KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3])", > "parsedquery_toString": > "KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]", > "QParser": "KnnQParser", > "explain": { > "Position11": { > "match": false, > "value": 0.0, > "description": "not in top 3" > }, > "Position10": { > "match": false, > "value": 0.0, > "description": "not in top 3" > }, > "Position9": { > "match": false, > "value": 0.0, > "description": "not in top 3" > }, > "Position8": { > "match": false, > "value": 0.0, > "description": "not in top 3" > }, > "Position7": { > "match": false, > "value": 0.0, > "description": "not in top 3" > }, > "Position6": { > "match": false, > "value": 0.0, > "description": "not in top 3" > } > } > } > } > > I got 6 search results. The top 3 of them are correct, and it seems that > two shards were queried, the results were combined and sorted correctly > by score. From my point of view is the result set size a functional bug, > as I requested the topK=3, but I got 6. > > The explain data is completely random. The description is wrong, the > match flag is wrong, and if value should be the score, it is also wrong. > This is a bug in the explain implementation, and regarding to this a > debug bug. > > Mirko > > Am 18.10.2023 um 16:56 schrieb Alessandro Benedetti: > > Let's distinguish if you are talking about a "debug" bug (the explain > > message) or a functional bug. > > > > -------------------------- > > *Alessandro Benedetti* > > Director @ Sease Ltd. > > *Apache Lucene/Solr Committer* > > *Apache Solr PMC Member* > > > > e-mail: a.benede...@sease.io > > > > > > *Sease* - Information Retrieval Applied > > Consulting | Training | Open Source > > > > Website: Sease.io <http://sease.io/> > > LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter > > <https://twitter.com/seaseltd> | Youtube > > <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github > > <https://github.com/seaseltd> > > > > > > On Wed, 18 Oct 2023 at 16:17, Mirko Sertic <mirko.ser...@web.de> wrote: > > > >> Am 18.10.2023 um 15:44 schrieb Alessandro Benedetti: > >>> -------------------------- > >>> *Alessandro Benedetti* > >>> Director @ Sease Ltd. > >>> *Apache Lucene/Solr Committer* > >>> *Apache Solr PMC Member* > >>> > >>> e-mail: a.benede...@sease.io > >>> > >>> > >>> *Sease* - Information Retrieval Applied > >>> Consulting | Training | Open Source > >>> > >>> Website: Sease.io <http://sease.io/> > >>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter > >>> <https://twitter.com/seaseltd> | Youtube > >>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github > >>> <https://github.com/seaseltd> > >>> > >>> > >>> On Wed, 18 Oct 2023 at 15:32, Mirko Sertic <mirko.ser...@web.de> > wrote: > >>> > >>>> Hi Alessandro, > >>>> thank you for clarification! I havent't read something about "topK is > >>>> per shard" in the documentation, but maybe I've missed that somewhere. > >>>> From a performance and implementation perspective, this might make > >>>> sense. From a users perspective, I'd expect that "topK" is per > >>>> collection, and not per shard, so the same behavior as "rows". I > >>>> should't have to think about the collection data layout while > >>>> formulating queries and interpreting the layouts. > >>>> Now, if "topK" is per shard, we assume that good matching results are > >>>> evenly distributed among the shards. This might not be always the > case. > >>>> If we use features like collapse & expand, block/join or explicit > >>>> document routing strategies, this might lead to strange results. In > this > >>>> case, it might be better to retrieve the topK per shard, join them all > >>>> together and sort them by score, and take topK from that sorted list > to > >>>> get the correct overall topK. What do you think about that? > >>>> > >>> That is what currently *should* happen. > >>> You can select the number of rows you like and get a subset of the > >>> NShards*K results. > >>> Are you observing something different? > >> Yes, if this does work correct, there should be no "not in top x" in > >> explain logs, but I see them, so I assume there is a bug somewhere. > >> > >>>> We've also found https://issues.apache.org/jira/browse/SOLR-16952, > >> which > >>>> might be correlated. > >>>> > >>> That seems to me like a bug for the stored fields functionality. > >>> > >>> > >>>> I also tried to understand > >>>> > >>>> > >> > https://github.com/apache/lucene/blob/723c05b047b4a66105636162bb3e0daaf2cc1de5/lucene/core/src/java/org/apache/lucene/search/AbstractKnnVectorQuery.java#L337 > >> , > >>>> but failed to get the details. > >>>> > >>>> I oberved the explain data further. Every document gets a score, which > >>>> is the result of the similarity function. However, some of them are > >>>> marked with "not in top 6 docs". So from my point of view those whould > >>>> not be part of the search result. > >>>> > >>> Can you elaborate on this? maybe through a full query and response with > >> the > >>> debug? > >>> Anyway, the debug score for vector-based search is currently almost > >>> useless, the score is not really explained. > >>> We know this and it's a Lucene level 'problem' (we haven't worked yet > on > >> a > >>> better way for doing it, is your company interested in sponsoring maybe > >> :) ) > >>> > >>>> Mirko > >>>> > >>>> Am 17.10.2023 um 18:16 schrieb Alessandro Benedetti: > >>>>> Hi Mirko, > >>>>> the topK is per shard. > >>>>> Then shards * k results are aggregated. > >>>>> Does it make sense? > >>>>> > >>>>> In regards to the debugging, it seems a bug, they all should be with > a > >>>>> score and within top-k > >>>>> -------------------------- > >>>>> *Alessandro Benedetti* > >>>>> Director @ Sease Ltd. > >>>>> *Apache Lucene/Solr Committer* > >>>>> *Apache Solr PMC Member* > >>>>> > >>>>> e-mail:a.benede...@sease.io > >>>>> > >>>>> > >>>>> *Sease* - Information Retrieval Applied > >>>>> Consulting | Training | Open Source > >>>>> > >>>>> Website: Sease.io<http://sease.io/> > >>>>> LinkedIn<https://linkedin.com/company/sease-ltd> | Twitter > >>>>> <https://twitter.com/seaseltd> | Youtube > >>>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github > >>>>> <https://github.com/seaseltd> > >>>>> > >>>>> > >>>>> On Tue, 17 Oct 2023 at 10:41, Mirko Sertic<mirko.ser...@web.de> > >> wrote: > >>>>>> Hey! > >>>>>> > >>>>>> Thank you for your help! > >>>>>> > >>>>>> We are running in cloud mode on GKE. Our index has 2 shards, and > every > >>>>>> shard has 2 replicas. The leader is a TLOG, the other replica is a > >> PULL. > >>>>>> Our main query is basically {!knn f=VECTOR_FIELD topK=10}[VECTOR > >> DATA]. > >>>>>> Thats it. > >>>>>> > >>>>>> I am really unsure how to debug this issue, Solr explain only gives > >> some > >>>>>> basic information. I really do not understand why I am getting more > >> than > >>>>>> topK=10 search results, and some of them do not match according to > the > >>>>>> explain data. > >>>>>> > >>>>>> Here is an example debug log of the parsed query: > >>>>>> > >>>>>> "rawquerystring" : "{!knn f=TEXTEMBEDDINGS_BESCHREIBUNG_768 > >>>>>> > >> > topK=6}[-0.72818565,-0.41219202,0.5225889,0.53035444,0.76394194,0.7767894,0.82073385,0.95900697,-0.36803666,-0.21959017,0.7140104,-0.13439204,0.06747756,1.3333143,0.2797215,1.3020324,0.13869889,-0.10991136,0.21142228,-0.36915645,-0.15180133,-0.62384933,-0.51568866,0.23904686,-0.61362225,0.026911864,0.36433217,-0.6443799,0.86989623,-0.0058394694,0.360967,-0.39565668,0.4483029,-0.26269752,0.24516392,0.44815013,0.43040967,0.3587644,0.873287,-0.10215642,-0.25633913,-0.21385787,-0.5121543,-0.103228725,0.3895751,-0.49162507,-0.9432531,-0.5980949,0.63356507,-0.40455475,0.9550473,0.7282631,-0.031743232,0.439844,0.21040638,-0.36207297,0.22014715,-0.5069794,0.4957589,-0.30554834,-0.8615413,-0.4926672,-0.013558676,-0.041132603,0.65787005,0.30416572,0.18375356,0.6744972,-0.6142544,-0.18229552,-0.11830064,1.0248266,0.2171961,1.0895737,-0.027262354,-0.17439449,0.74050045,-0.8744221,-0.16733684,0.93963474,-0.4188998,-1.1815201,-0.4237022,-1.4382845,0.9899152,-0.17618449,0.8811465,-0.1435171,0.1467631,0.20906292,-0.19986387,-0.07172442,-0.5412453,-0.38141707,-0.8519122,0.57226634,0.27108946,0.47601715,-0.50961906,0.7443681,-0.46236074,0.44012043,-0.64782083,0.12072155,0.17669038,0.8894314,-0.26589298,-1.0005188,-0.38552156,-0.17849512,-0.41564974,0.46785104,0.07669882,0.098340474,-0.40940204,-0.22534837,-0.46240783,-0.6026122,-0.150852,-0.056922164,-0.5217692,0.2522323,-0.05158736,0.4368061,0.07538188,-0.15538858,-0.27512375,-7.942749,-0.5104796,0.3684156,0.17436846,0.33592805,0.30256566,0.3045547,-0.8253086,0.39595905,-0.63918144,0.023157855,-0.19350462,-1.1046336,0.24591433,-0.35902688,-0.80178124,0.60646725,-0.30873522,0.3227124,0.14232452,-0.11611875,0.36796737,-0.53184104,0.30364224,0.28488177,-1.0550518,-0.42094457,-0.12320968,-0.9765126,-0.15129066,0.001331389,-0.27246574,0.4749163,0.4177817,0.2973763,-0.29119635,-0.3972071,0.59532064,0.10756115,0.23748569,-0.70517355,1.201706,0.502714,-0.26869884,-0.053134415,-0.66120774,-0.0039092028,-0.64522946,-0.041144915,-0.05124693,-0.9726303,-0.99400455,-0.07065071,-0.78489596,0.80071807,-0.13281979,-0.006903738,0.5823942,-0.2669629,1.6502038,0.7482211,-0.6007372,0.66805166,-0.3942953,-0.5034327,0.27077916,0.15055649,0.10765681,-0.12882604,0.24207312,0.18053184,-0.21453774,0.45982376,-0.5745059,0.2006089,0.50371104,0.8538735,0.97700334,0.05624181,-0.132805,0.033950403,-0.18651533,-0.06351512,-0.7937849,-0.6627913,-0.3048031,0.53663576,0.17241986,0.36505452,-0.09964923,0.13906284,0.5434706,-0.36837682,-0.35080168,0.46206424,-0.20238751,-0.27848598,0.2480285,0.5826801,-0.48960462,0.7367855,1.0053717,0.17819782,0.38643655,-0.16025475,-0.35159552,-0.0024995431,-0.022663444,0.05895573,-0.19539028,-0.36216947,-0.06652315,-0.13567717,-0.68101376,-0.3490555,0.55224353,0.045317963,-0.8309715,-0.24363773,0.4269611,0.24411823,-0.06477823,0.80422884,0.35045695,0.20747428,-0.3475987,0.1297931,0.09917005,0.007867369,0.38570535,0.13428706,1.3826305,-0.8623333,-1.0942854,0.3748387,-0.5336166,-0.35972998,-0.10899716,0.278379,0.94615966,-0.93360096,-0.9554229,-0.28387132,-0.02770327,-0.9298182,0.45993057,-0.65919405,0.1475371,-0.34568012,1.209977,0.2802038,0.45577803,0.12152732,0.87355286,1.5883585,0.6779906,0.17098032,1.0125086,-0.55774075,-0.38352862,0.16628654,0.15557681,0.1861148,0.04379474,-0.7305415,-0.030636007,0.024982145,0.67828315,0.42840484,-0.053252134,-0.4504099,-0.20452605,-0.1081517,-0.14191566,0.07114916,-0.078530826,0.17243765,-0.46307424,-0.008914786,0.26029265,0.52127206,-0.35917446,-0.015462582,-0.28368863,-0.7689151,-0.263665,0.49229744,0.7343957,0.24539919,-0.2738248,-0.12916541,1.5979353,0.2385032,-0.5589315,0.1753772,-0.57815063,0.13694653,-0.20679545,0.78396624,-0.26194802,0.16952795,0.009313789,0.053571712,-0.50364834,0.19213165,0.020542603,0.3242613,-0.21481745,0.46329287,0.1883949,-0.53527766,0.19803612,-0.10915735,-0.53217936,0.5851483,0.8633852,-0.22306518,0.19917397,-0.48938546,-0.3489624,0.26114023,0.5178694,-0.49423835,0.22051883,0.9698861,-0.023483349,0.5058909,-0.19136174,-0.06710764,-0.50730103,-0.5724341,-0.07458467,0.72695184,0.36962357,0.12050099,0.16827017,-0.36094713,-0.22090997,0.5247373,0.5981514,0.11581371,-0.14274465,0.110738344,0.5218388,-0.38720345,0.9173073,-0.8510778,-0.22613077,-0.2748853,0.2863147,-0.3778913,-0.18058139,0.37012923,-0.5144021,-0.5551922,0.16914673,0.78963786,1.1504575,-0.7211657,-0.63688916,-0.075630635,-0.89228654,0.8719732,-0.16148148,-0.093216084,-0.0649338,0.40419587,-0.5091749,0.3257159,-0.38114238,-0.7489807,-0.14795141,0.070623375,1.0296395,-0.18232572,0.045388192,-0.5071435,-0.23788519,-0.60791194,-0.25087252,0.009456612,0.8095627,-0.43995717,-0.21105917,-0.83442384,0.7455656,-0.9289396,-0.26139355,0.33529952,-0.7209875,0.44793245,-0.11075375,0.35380912,-0.49433497,1.061043,0.24260058,1.277706,-1.0634571,-0.028423825,-0.6881516,-0.2748189,0.32650438,0.22231913,-0.6346664,-0.32749617,0.6449113,0.4030831,0.086880766,-0.34941757,-0.32754892,0.44228384,-1.0537914,-0.4920558,0.11018584,-0.4000114,0.19572248,-0.006614799,-0.1560454,-0.6138168,-0.2874727,0.08457174,0.852747,0.020211289,-0.46017483,0.05079806,0.61822194,0.16241436,-0.418595,-0.44193658,-0.23446727,0.08835063,-0.86544657,0.039239086,0.47766128,-0.31095138,-0.545759,-0.09576557,-0.5694708,-0.10043368,-0.2653149,-0.1599367,0.13423382,-0.1830842,0.046266567,-0.08192452,-0.75070643,-0.65771914,0.18250088,-0.36299148,0.33271846,-0.60936797,1.339671,-0.12343848,1.0515145,0.25372154,-0.4882932,-0.06304815,-0.17309606,0.30466563,0.08878487,-0.5156037,-0.33687803,0.5997892,0.1856752,0.5528808,0.5089958,-0.77673227,-0.43565074,-3.0061991,-0.22721101,1.1010107,1.285386,0.39106908,0.06541425,0.18982089,-0.33513412,-0.5287475,0.2362032,0.40598843,0.62325794,-0.8586378,-0.64406306,0.09112385,0.055086333,0.28247252,0.13180174,0.5969885,0.15263812,0.39325213,-0.45515132,-0.47813737,-0.33163494,0.3496457,0.39913225,-0.5750896,0.104484975,-0.69256973,0.18164606,0.09423768,0.045548365,0.51491827,-0.854532,0.40470743,-0.12821743,-0.22693269,-0.31501335,-0.53367645,0.4482267,-0.647523,-0.48177683,-0.45763886,0.31513563,0.115996234,-0.56461513,0.112646066,0.6897073,0.30277768,1.2858859,-0.47804725,0.36647198,0.8737555,0.89239925,-0.26573908,0.13890187,0.8274238,0.6068263,0.21134727,-0.5331567,-0.40894493,-0.20580018,-0.4116305,-8.852631E-4,0.71050125,0.18508287,0.3889124,-0.38231707,-0.7620664,-0.26053023,0.5970558,0.12187529,-0.35657012,-0.20820218,-0.4604194,-0.36569384,0.08593381,0.31331047,-0.12055105,0.12107455,0.45513165,-0.0023682665,0.3980597,0.1632141,0.33392814,-0.050680444,0.13038154,0.13847367,1.3034295,0.2744402,0.124595374,-0.53905815,0.35776177,-0.13275874,0.018385967,-0.16347891,-0.19726902,-0.4961019,0.07333634,0.52688265,0.20600657,0.14087541,-0.21112329,0.5135732,0.4315345,-0.020893982,0.41702232,0.07464256,-0.2845659,-0.13949631,-0.013015106,0.40522456,0.041121256,1.1591781,0.043223336,0.39874303,-0.30360883,-0.27349794,-0.04964445,-0.5729313,0.14069785,-0.33394682,0.9343837,-0.9350466,-0.8602237,-0.29167423,-1.5445148,0.040775828,-0.018334387,0.42681727,-0.1458294,0.49805698,0.27569738,-0.10979689,-0.5429115,-0.020050785,0.31627277,-0.6220587,0.061334964,0.13807471,0.34479836,0.11312207,0.796787,-0.48607716,-0.14010525,-0.61075526,0.6672587,1.6726193,0.4950563,0.41054416,0.2522166,-0.95681673,-1.0119423,-0.4870938,0.48430035,-0.38044646,0.59904295,0.23687701,0.93788594,-0.22263275,0.07065558,-0.4562675,-0.08655307,0.69646436,-0.5812278,-0.30097616,-0.9931224,0.23148312,0.13233739,0.74470276,0.014401446,0.08970716,0.23342986,-0.4664313,0.054693338,0.2778716,0.4019597,-0.97575116,0.058492173,-0.5017976,0.09864781,-0.3473021,-0.4823157,0.13100009,0.4488969,-0.1672722,-1.0326554,-0.18146743,-0.349057,0.61888343,-0.18784277,-0.53318554,-0.2559318,-0.6182706,0.24759102,-0.71451706,0.7098109,0.53027123,0.36208203,0.23936976,-0.10664157,0.3863506,0.42203853,0.1808002,-0.22444467,1.0584201,-0.29779503,0.43873647,0.2835624,-0.76047546,0.37234184,0.2044634,0.5471627,0.4247879,0.4856389,-0.7011678,0.40757468,0.1918484,0.54874665,-0.1279645,0.5678282,0.02382508,0.18181127,-0.42465243,0.20996757,-0.48161522,-0.53782755,-0.1985728,-0.12512843,-0.47403875,0.44477355,-0.05491643,0.81035346,0.3633467,0.5024226,-0.07482373,-0.7337039,0.0675221,0.3952929,0.45677915,-0.29935122,-0.42017272,-0.80567694,0.2868536,-0.40015128,-0.060443062,-0.21671702,0.18321274,-0.20112215,-0.50728095,0.3698101,0.56871736,-0.0904237,-0.43735075,-0.79018253,-0.0739331,0.49475756,0.014863159,-0.47107005,0.14020085,0.28982484,0.32168245,-0.56485337,0.17436136,-0.0050138533,-0.9533212,0.036251593,-0.33072975,-0.87562966,-0.4379939,0.8052754,0.30534235,0.20573251]", > >>>>>> "querystring" : "{!knn f=TEXTEMBEDDINGS_BESCHREIBUNG_768 > >>>>>> > >> > topK=6}[-0.72818565,-0.41219202,0.5225889,0.53035444,0.76394194,0.7767894,0.82073385,0.95900697,-0.36803666,-0.21959017,0.7140104,-0.13439204,0.06747756,1.3333143,0.2797215,1.3020324,0.13869889,-0.10991136,0.21142228,-0.36915645,-0.15180133,-0.62384933,-0.51568866,0.23904686,-0.61362225,0.026911864,0.36433217,-0.6443799,0.86989623,-0.0058394694,0.360967,-0.39565668,0.4483029,-0.26269752,0.24516392,0.44815013,0.43040967,0.3587644,0.873287,-0.10215642,-0.25633913,-0.21385787,-0.5121543,-0.103228725,0.3895751,-0.49162507,-0.9432531,-0.5980949,0.63356507,-0.40455475,0.9550473,0.7282631,-0.031743232,0.439844,0.21040638,-0.36207297,0.22014715,-0.5069794,0.4957589,-0.30554834,-0.8615413,-0.4926672,-0.013558676,-0.041132603,0.65787005,0.30416572,0.18375356,0.6744972,-0.6142544,-0.18229552,-0.11830064,1.0248266,0.2171961,1.0895737,-0.027262354,-0.17439449,0.74050045,-0.8744221,-0.16733684,0.93963474,-0.4188998,-1.1815201,-0.4237022,-1.4382845,0.9899152,-0.17618449,0.8811465,-0.1435171,0.1467631,0.20906292,-0.19986387,-0.07172442,-0.5412453,-0.38141707,-0.8519122,0.57226634,0.27108946,0.47601715,-0.50961906,0.7443681,-0.46236074,0.44012043,-0.64782083,0.12072155,0.17669038,0.8894314,-0.26589298,-1.0005188,-0.38552156,-0.17849512,-0.41564974,0.46785104,0.07669882,0.098340474,-0.40940204,-0.22534837,-0.46240783,-0.6026122,-0.150852,-0.056922164,-0.5217692,0.2522323,-0.05158736,0.4368061,0.07538188,-0.15538858,-0.27512375,-7.942749,-0.5104796,0.3684156,0.17436846,0.33592805,0.30256566,0.3045547,-0.8253086,0.39595905,-0.63918144,0.023157855,-0.19350462,-1.1046336,0.24591433,-0.35902688,-0.80178124,0.60646725,-0.30873522,0.3227124,0.14232452,-0.11611875,0.36796737,-0.53184104,0.30364224,0.28488177,-1.0550518,-0.42094457,-0.12320968,-0.9765126,-0.15129066,0.001331389,-0.27246574,0.4749163,0.4177817,0.2973763,-0.29119635,-0.3972071,0.59532064,0.10756115,0.23748569,-0.70517355,1.201706,0.502714,-0.26869884,-0.053134415,-0.66120774,-0.0039092028,-0.64522946,-0.041144915,-0.05124693,-0.9726303,-0.99400455,-0.07065071,-0.78489596,0.80071807,-0.13281979,-0.006903738,0.5823942,-0.2669629,1.6502038,0.7482211,-0.6007372,0.66805166,-0.3942953,-0.5034327,0.27077916,0.15055649,0.10765681,-0.12882604,0.24207312,0.18053184,-0.21453774,0.45982376,-0.5745059,0.2006089,0.50371104,0.8538735,0.97700334,0.05624181,-0.132805,0.033950403,-0.18651533,-0.06351512,-0.7937849,-0.6627913,-0.3048031,0.53663576,0.17241986,0.36505452,-0.09964923,0.13906284,0.5434706,-0.36837682,-0.35080168,0.46206424,-0.20238751,-0.27848598,0.2480285,0.5826801,-0.48960462,0.7367855,1.0053717,0.17819782,0.38643655,-0.16025475,-0.35159552,-0.0024995431,-0.022663444,0.05895573,-0.19539028,-0.36216947,-0.06652315,-0.13567717,-0.68101376,-0.3490555,0.55224353,0.045317963,-0.8309715,-0.24363773,0.4269611,0.24411823,-0.06477823,0.80422884,0.35045695,0.20747428,-0.3475987,0.1297931,0.09917005,0.007867369,0.38570535,0.13428706,1.3826305,-0.8623333,-1.0942854,0.3748387,-0.5336166,-0.35972998,-0.10899716,0.278379,0.94615966,-0.93360096,-0.9554229,-0.28387132,-0.02770327,-0.9298182,0.45993057,-0.65919405,0.1475371,-0.34568012,1.209977,0.2802038,0.45577803,0.12152732,0.87355286,1.5883585,0.6779906,0.17098032,1.0125086,-0.55774075,-0.38352862,0.16628654,0.15557681,0.1861148,0.04379474,-0.7305415,-0.030636007,0.024982145,0.67828315,0.42840484,-0.053252134,-0.4504099,-0.20452605,-0.1081517,-0.14191566,0.07114916,-0.078530826,0.17243765,-0.46307424,-0.008914786,0.26029265,0.52127206,-0.35917446,-0.015462582,-0.28368863,-0.7689151,-0.263665,0.49229744,0.7343957,0.24539919,-0.2738248,-0.12916541,1.5979353,0.2385032,-0.5589315,0.1753772,-0.57815063,0.13694653,-0.20679545,0.78396624,-0.26194802,0.16952795,0.009313789,0.053571712,-0.50364834,0.19213165,0.020542603,0.3242613,-0.21481745,0.46329287,0.1883949,-0.53527766,0.19803612,-0.10915735,-0.53217936,0.5851483,0.8633852,-0.22306518,0.19917397,-0.48938546,-0.3489624,0.26114023,0.5178694,-0.49423835,0.22051883,0.9698861,-0.023483349,0.5058909,-0.19136174,-0.06710764,-0.50730103,-0.5724341,-0.07458467,0.72695184,0.36962357,0.12050099,0.16827017,-0.36094713,-0.22090997,0.5247373,0.5981514,0.11581371,-0.14274465,0.110738344,0.5218388,-0.38720345,0.9173073,-0.8510778,-0.22613077,-0.2748853,0.2863147,-0.3778913,-0.18058139,0.37012923,-0.5144021,-0.5551922,0.16914673,0.78963786,1.1504575,-0.7211657,-0.63688916,-0.075630635,-0.89228654,0.8719732,-0.16148148,-0.093216084,-0.0649338,0.40419587,-0.5091749,0.3257159,-0.38114238,-0.7489807,-0.14795141,0.070623375,1.0296395,-0.18232572,0.045388192,-0.5071435,-0.23788519,-0.60791194,-0.25087252,0.009456612,0.8095627,-0.43995717,-0.21105917,-0.83442384,0.7455656,-0.9289396,-0.26139355,0.33529952,-0.7209875,0.44793245,-0.11075375,0.35380912,-0.49433497,1.061043,0.24260058,1.277706,-1.0634571,-0.028423825,-0.6881516,-0.2748189,0.32650438,0.22231913,-0.6346664,-0.32749617,0.6449113,0.4030831,0.086880766,-0.34941757,-0.32754892,0.44228384,-1.0537914,-0.4920558,0.11018584,-0.4000114,0.19572248,-0.006614799,-0.1560454,-0.6138168,-0.2874727,0.08457174,0.852747,0.020211289,-0.46017483,0.05079806,0.61822194,0.16241436,-0.418595,-0.44193658,-0.23446727,0.08835063,-0.86544657,0.039239086,0.47766128,-0.31095138,-0.545759,-0.09576557,-0.5694708,-0.10043368,-0.2653149,-0.1599367,0.13423382,-0.1830842,0.046266567,-0.08192452,-0.75070643,-0.65771914,0.18250088,-0.36299148,0.33271846,-0.60936797,1.339671,-0.12343848,1.0515145,0.25372154,-0.4882932,-0.06304815,-0.17309606,0.30466563,0.08878487,-0.5156037,-0.33687803,0.5997892,0.1856752,0.5528808,0.5089958,-0.77673227,-0.43565074,-3.0061991,-0.22721101,1.1010107,1.285386,0.39106908,0.06541425,0.18982089,-0.33513412,-0.5287475,0.2362032,0.40598843,0.62325794,-0.8586378,-0.64406306,0.09112385,0.055086333,0.28247252,0.13180174,0.5969885,0.15263812,0.39325213,-0.45515132,-0.47813737,-0.33163494,0.3496457,0.39913225,-0.5750896,0.104484975,-0.69256973,0.18164606,0.09423768,0.045548365,0.51491827,-0.854532,0.40470743,-0.12821743,-0.22693269,-0.31501335,-0.53367645,0.4482267,-0.647523,-0.48177683,-0.45763886,0.31513563,0.115996234,-0.56461513,0.112646066,0.6897073,0.30277768,1.2858859,-0.47804725,0.36647198,0.8737555,0.89239925,-0.26573908,0.13890187,0.8274238,0.6068263,0.21134727,-0.5331567,-0.40894493,-0.20580018,-0.4116305,-8.852631E-4,0.71050125,0.18508287,0.3889124,-0.38231707,-0.7620664,-0.26053023,0.5970558,0.12187529,-0.35657012,-0.20820218,-0.4604194,-0.36569384,0.08593381,0.31331047,-0.12055105,0.12107455,0.45513165,-0.0023682665,0.3980597,0.1632141,0.33392814,-0.050680444,0.13038154,0.13847367,1.3034295,0.2744402,0.124595374,-0.53905815,0.35776177,-0.13275874,0.018385967,-0.16347891,-0.19726902,-0.4961019,0.07333634,0.52688265,0.20600657,0.14087541,-0.21112329,0.5135732,0.4315345,-0.020893982,0.41702232,0.07464256,-0.2845659,-0.13949631,-0.013015106,0.40522456,0.041121256,1.1591781,0.043223336,0.39874303,-0.30360883,-0.27349794,-0.04964445,-0.5729313,0.14069785,-0.33394682,0.9343837,-0.9350466,-0.8602237,-0.29167423,-1.5445148,0.040775828,-0.018334387,0.42681727,-0.1458294,0.49805698,0.27569738,-0.10979689,-0.5429115,-0.020050785,0.31627277,-0.6220587,0.061334964,0.13807471,0.34479836,0.11312207,0.796787,-0.48607716,-0.14010525,-0.61075526,0.6672587,1.6726193,0.4950563,0.41054416,0.2522166,-0.95681673,-1.0119423,-0.4870938,0.48430035,-0.38044646,0.59904295,0.23687701,0.93788594,-0.22263275,0.07065558,-0.4562675,-0.08655307,0.69646436,-0.5812278,-0.30097616,-0.9931224,0.23148312,0.13233739,0.74470276,0.014401446,0.08970716,0.23342986,-0.4664313,0.054693338,0.2778716,0.4019597,-0.97575116,0.058492173,-0.5017976,0.09864781,-0.3473021,-0.4823157,0.13100009,0.4488969,-0.1672722,-1.0326554,-0.18146743,-0.349057,0.61888343,-0.18784277,-0.53318554,-0.2559318,-0.6182706,0.24759102,-0.71451706,0.7098109,0.53027123,0.36208203,0.23936976,-0.10664157,0.3863506,0.42203853,0.1808002,-0.22444467,1.0584201,-0.29779503,0.43873647,0.2835624,-0.76047546,0.37234184,0.2044634,0.5471627,0.4247879,0.4856389,-0.7011678,0.40757468,0.1918484,0.54874665,-0.1279645,0.5678282,0.02382508,0.18181127,-0.42465243,0.20996757,-0.48161522,-0.53782755,-0.1985728,-0.12512843,-0.47403875,0.44477355,-0.05491643,0.81035346,0.3633467,0.5024226,-0.07482373,-0.7337039,0.0675221,0.3952929,0.45677915,-0.29935122,-0.42017272,-0.80567694,0.2868536,-0.40015128,-0.060443062,-0.21671702,0.18321274,-0.20112215,-0.50728095,0.3698101,0.56871736,-0.0904237,-0.43735075,-0.79018253,-0.0739331,0.49475756,0.014863159,-0.47107005,0.14020085,0.28982484,0.32168245,-0.56485337,0.17436136,-0.0050138533,-0.9533212,0.036251593,-0.33072975,-0.87562966,-0.4379939,0.8052754,0.30534235,0.20573251]", > >>>>>> "parsedquery" : > >>>>>> > >> > "KnnVectorQuery(KnnVectorQuery:TEXTEMBEDDINGS_BESCHREIBUNG_768[-0.72818565,...][6])", > >>>>>> "parsedquery_toString" : > >>>>>> > "KnnVectorQuery:TEXTEMBEDDINGS_BESCHREIBUNG_768[-0.72818565,...][6]", > >>>>>> "QParser" : "KnnQParser", > >>>>>> > >>>>>> Am 17.10.2023 um 10:05 schrieb Alessandro Benedetti: > >>>>>>> What's your full Solr query? > >>>>>>> Are you on SolrCloud or single Solr node? > >>>>>>> -------------------------- > >>>>>>> *Alessandro Benedetti* > >>>>>>> Director @ Sease Ltd. > >>>>>>> *Apache Lucene/Solr Committer* > >>>>>>> *Apache Solr PMC Member* > >>>>>>> > >>>>>>> e-mail:a.benede...@sease.io > >>>>>>> > >>>>>>> > >>>>>>> *Sease* - Information Retrieval Applied > >>>>>>> Consulting | Training | Open Source > >>>>>>> > >>>>>>> Website: Sease.io<http://sease.io/> > >>>>>>> LinkedIn<https://linkedin.com/company/sease-ltd> | Twitter > >>>>>>> <https://twitter.com/seaseltd> | Youtube > >>>>>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | > >> Github > >>>>>>> <https://github.com/seaseltd> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, 17 Oct 2023 at 09:45, Mirko Sertic<mirko.ser...@web.de> > >>>> wrote: > >>>>>>>> To correct me, there was a typo. I meant: > >>>>>>>> > >>>>>>>> If I specify topK=6, I get numFound=12, but only some of them > match > >>>> the > >>>>>>>> top 6 > >>>>>>>> > >>>>>>>> > >>>>>>>> Am 17.10.2023 um 09:31 schrieb Mirko Sertic: > >>>>>>>>> Hi! > >>>>>>>>> > >>>>>>>>> To keep you updated, here are some observations regarding the > >>>>>>>>> numFound/resultset size and DenseVectorQueries: > >>>>>>>>> > >>>>>>>>> If I specity topK=10, I get numFound=20, but only some of them > >> match > >>>>>> the > >>>>>>>>> top 10 > >>>>>>>>> > >>>>>>>>> If I specify topK=8, I get numFound=16, but only some of them > match > >>>> the > >>>>>>>>> top 8 > >>>>>>>>> > >>>>>>>>> If I specify topK=6, I get numFound=8, but only some of them > match > >>>> the > >>>>>>>>> top 6 > >>>>>>>>> > >>>>>>>>> So the numFound seems always to be the double of topK. Might > there > >>>> be a > >>>>>>>>> correlation with sharding? Our collection has two shards, so does > >>>> this > >>>>>>>>> double the results? I would't expect that, but that might be the > >> only > >>>>>>>>> thing relating to a constant 2 in our setup. > >>>>>>>>> > >>>>>>>>> Mirko > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Am 16.10.2023 um 14:46 schrieb Mirko Sertic: > >>>>>>>>>> Hi@all > >>>>>>>>>> > >>>>>>>>>> We are using Solr 9.1.1, and are trying usecases with > DenseVector > >>>>>> Fields > >>>>>>>>>> and knnQueries in mind. > >>>>>>>>>> > >>>>>>>>>> During our tests, we see the following results and are trying to > >>>>>> figure > >>>>>>>>>> out what is going on: > >>>>>>>>>> > >>>>>>>>>> a) We use the following main query : {!knn f=VECTOR_FIELD > >>>>>>>>>> topK=10}[VECTOR DATA]. We use it as a main query because we want > >> to > >>>>>>>>>> apply the distance function to the document score. However, > when I > >>>> try > >>>>>>>>>> do debug and explain the search results, I am getting more than > >>>>>> topK=10 > >>>>>>>>>> result documents, some are marked as match = true with "within > top > >>>>>> 10", > >>>>>>>>>> others are marked as match = false with "not in top 10". I'd > >> expect > >>>>>> that > >>>>>>>>>> only matched documents are part of the search result, but there > >> are > >>>> 20 > >>>>>>>>>> result documents, but only 5 of them are matched. Did I miss > >>>>>> something? > >>>>>>>>>> b) The knn query results are the approximate nearest neighbors, > >> but > >>>>>> they > >>>>>>>>>> might not be the best. We'd like to define some kind of cut-off > >>>> value > >>>>>>>>>> for knn document scores. Is this possible, and what would be a > >> good > >>>>>> day > >>>>>>>>>> to do so? Implement a post-processing filter query with an > frange > >> on > >>>>>> the > >>>>>>>>>> score field? > >>>>>>>>>> > >>>>>>>>>> Thank you all, > >>>>>>>>>> > >>>>>>>>>> Mirko > >>>>>>>>>> >
