I've prepared a testcase. Given the following documents with
TESTEMBEDDING_EU_3 is a DenseVectorField with length 3 and euclidean
distance function. They are written to a collection made of two shards
with no further routing strategy, so they should be more or less evenly
distributed between the two shards:
{
id: 'Position1',
TESTEMBEDDING_EU_3: [0, 0, 0]
}
{
id: 'Position2',
TESTEMBEDDING_EU_3: [0.1, 0.1, 0.1]
}
{
id: 'Position3',
TESTEMBEDDING_EU_3: [0.2, 0.2, 0.2]
}
{
id: 'Position4',
TESTEMBEDDING_EU_3: [0.3, 0.3, 0.3]
}
{
id: 'Position5',
TESTEMBEDDING_EU_3: [0.4, 0.4, 0.4]
}
{
id: 'Position6',
TESTEMBEDDING_EU_3: [0.5, 0.5, 0.5]
}
{
id: 'Position7',
TESTEMBEDDING_EU_3: [0.6, 0.6, 0.6]
}
{
id: 'Position8',
TESTEMBEDDING_EU_3: [0.7, 0.7, 0.7]
}
{
id: 'Position9',
TESTEMBEDDING_EU_3: [0.8, 0.8, 0.8]
}
{
id: 'Position10',
TESTEMBEDDING_EU_3: [0.9, 0.9, 0.9]
}
{
id: 'Position11',
TESTEMBEDDING_EU_3: [1.0, 1.0, 1.0]
}
How I'll do a {!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0] query.
I'd expect a result with 3 documents, id:Position11 should be an exact
macht, and the nearst neighbors should be id:Position10 and
id:Position9. I'd also expect that the explain logging should mark these
tree as part of the topK=3. I get the following search result:
{
"responseHeader": {
"zkConnected": true,
"status": 0,
"QTime": 35
},
"response": {
"numFound": 6,
"start": 0,
"maxScore": 1.0,
"numFoundExact": true,
"docs": [
{
"id": "Position11",
"TESTEMBEDDING_3": [
"1.0",
"1.0",
"1.0"
],
"[shard]":
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/
",
"[explain]": "0.0 = not in top 3\n",
"score": 1.0
},
{
"id": "Position10",
"TESTEMBEDDING_3": [
"0.9",
"0.9",
"0.9"
],
"[shard]":
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/
",
"[explain]": "0.0 = not in top 3\n",
"score": 0.97087383
},
{
"id": "Position9",
"TESTEMBEDDING_3": [
"0.8",
"0.8",
"0.8"
],
"[shard]":
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/
",
"[explain]": "0.0 = not in top 3\n",
"score": 0.89285713
},
{
"id": "Position8",
"TESTEMBEDDING_3": [
"0.7",
"0.7",
"0.7"
],
"[shard]":
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/
",
"[explain]": "0.0 = not in top 3\n",
"score": 0.78740156
},
{
"id": "Position7",
"TESTEMBEDDING_3": [
"0.6",
"0.6",
"0.6"
],
"[shard]":
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/
",
"[explain]": "0.0 = not in top 3\n",
"score": 0.6756757
},
{
"id": "Position6",
"TESTEMBEDDING_3": [
"0.5",
"0.5",
"0.5"
],
"[shard]":
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/
",
"[explain]": "0.0 = not in top 3\n",
"score": 0.5714286
}
]
},
"spellcheck": {
"suggestions": [],
"collations": []
},
"debug": {
"track": {
"rid":
"fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless-465214",
"EXECUTE_QUERY": {
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/":
{
"QTime": "23",
"ElapsedTime": "27",
"RequestPurpose": "GET_TOP_IDS,SET_TERM_STATS",
"NumFound": "3",
"Response": "{responseHeader={zkConnected=true, status=0,
QTime=23},
response={numFound=3,numFoundExact=true,start=0,maxScore=1.0,docs=[SolrDocument{id=Position11,
score=1.0}, SolrDocument{id=Position10, score=0.97087383},
SolrDocument{id=Position7, score=0.6756757}]}, sort_values={},
spellcheck={suggestions={!knn={numFound=0, startOffset=1, endOffset=5,
suggestion=[]}, =testembedding_eu_3={numFound=0, startOffset=7,
endOffset=26, suggestion=[]}, topk={numFound=16, startOffset=28,
endOffset=32, suggestion=[topp, topf, took, tonk, tops, topik, topo,
tock, tope, topa, topi, to k, tork, tolk, topb, topz]}, =3={numFound=0,
startOffset=32, endOffset=34, suggestion=[]}, [1={numFound=0,
startOffset=35, endOffset=37, suggestion=[]}, ,1={numFound=0,
startOffset=39, endOffset=41, suggestion=[]}, ,1={numFound=0,
startOffset=43, endOffset=45, suggestion=[]}},
collations={collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topp=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topp},
collationInternalRank=0}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 topf=3}[1.0,1.0,1.0], hits=6,
misspellingsAndCorrections={topk=topf}, collationInternalRank=1000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
took=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=took},
collationInternalRank=2000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 tonk=3}[1.0,1.0,1.0], hits=6,
misspellingsAndCorrections={topk=tonk}, collationInternalRank=3000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
tops=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tops},
collationInternalRank=4000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 topik=3}[1.0,1.0,1.0], hits=6,
misspellingsAndCorrections={topk=topik}, collationInternalRank=5000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topo=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topo},
collationInternalRank=6000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 tock=3}[1.0,1.0,1.0], hits=6,
misspellingsAndCorrections={topk=tock}, collationInternalRank=7000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
tope=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tope},
collationInternalRank=8000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 topa=3}[1.0,1.0,1.0], hits=6,
misspellingsAndCorrections={topk=topa}, collationInternalRank=9000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topi=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topi},
collationInternalRank=10000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 to k=3}[1.0,1.0,1.0], hits=6,
misspellingsAndCorrections={topk=to k}, collationInternalRank=11000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
tork=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tork},
collationInternalRank=12000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 tolk=3}[1.0,1.0,1.0], hits=6,
misspellingsAndCorrections={topk=tolk}, collationInternalRank=13000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topb=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topb},
collationInternalRank=14000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 topz=3}[1.0,1.0,1.0], hits=6,
misspellingsAndCorrections={topk=topz}, collationInternalRank=15000}},
originalTerms=[!knn, =testembedding_eu_3, topk, =3, [1, ,1, ,1]},
debug={timing={time=23.0, prepare={time=0.0, query={time=0.0},
facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0},
highlight={time=0.0}, stats={time=0.0}, expand={time=0.0},
terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}},
process={time=23.0, query={time=0.0}, facet={time=0.0},
facet_module={time=0.0}, mlt={time=0.0}, highlight={time=0.0},
stats={time=0.0}, expand={time=0.0}, terms={time=0.0},
spellcheck={time=23.0}, debug={time=0.0}}}}}"
},
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/":
{
"QTime": "24",
"ElapsedTime": "29",
"RequestPurpose": "GET_TOP_IDS,SET_TERM_STATS",
"NumFound": "3",
"Response": "{responseHeader={zkConnected=true, status=0,
QTime=24},
response={numFound=3,numFoundExact=true,start=0,maxScore=0.89285713,docs=[SolrDocument{id=Position9,
score=0.89285713}, SolrDocument{id=Position8, score=0.78740156},
SolrDocument{id=Position6, score=0.5714286}]}, sort_values={},
spellcheck={suggestions={!knn={numFound=0, startOffset=1, endOffset=5,
suggestion=[]}, =testembedding_eu_3={numFound=0, startOffset=7,
endOffset=26, suggestion=[]}, topk={numFound=16, startOffset=28,
endOffset=32, suggestion=[topp, took, topf, tops, topo, topik, tonk,
tope, tock, to k, topi, tork, tolk, top5, topc, topka]}, =3={numFound=0,
startOffset=32, endOffset=34, suggestion=[]}, [1={numFound=0,
startOffset=35, endOffset=37, suggestion=[]}, ,1={numFound=0,
startOffset=39, endOffset=41, suggestion=[]}, ,1={numFound=0,
startOffset=43, endOffset=45, suggestion=[]}},
collations={collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topp=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topp},
collationInternalRank=0}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 took=3}[1.0,1.0,1.0], hits=5,
misspellingsAndCorrections={topk=took}, collationInternalRank=1000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topf=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topf},
collationInternalRank=2000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 tops=3}[1.0,1.0,1.0], hits=5,
misspellingsAndCorrections={topk=tops}, collationInternalRank=3000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topo=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topo},
collationInternalRank=4000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 topik=3}[1.0,1.0,1.0], hits=5,
misspellingsAndCorrections={topk=topik}, collationInternalRank=5000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
tonk=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tonk},
collationInternalRank=6000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 tope=3}[1.0,1.0,1.0], hits=5,
misspellingsAndCorrections={topk=tope}, collationInternalRank=7000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
tock=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tock},
collationInternalRank=8000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 to k=3}[1.0,1.0,1.0], hits=5,
misspellingsAndCorrections={topk=to k}, collationInternalRank=9000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topi=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topi},
collationInternalRank=10000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 tork=3}[1.0,1.0,1.0], hits=5,
misspellingsAndCorrections={topk=tork}, collationInternalRank=11000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
tolk=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tolk},
collationInternalRank=12000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 top5=3}[1.0,1.0,1.0], hits=5,
misspellingsAndCorrections={topk=top5}, collationInternalRank=13000},
collation={collationQuery={!knn f=TESTEMBEDDING_EU_3
topc=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topc},
collationInternalRank=14000}, collation={collationQuery={!knn
f=TESTEMBEDDING_EU_3 topka=3}[1.0,1.0,1.0], hits=5,
misspellingsAndCorrections={topk=topka}, collationInternalRank=15000}},
originalTerms=[!knn, =testembedding_eu_3, topk, =3, [1, ,1, ,1]},
debug={timing={time=24.0, prepare={time=0.0, query={time=0.0},
facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0},
highlight={time=0.0}, stats={time=0.0}, expand={time=0.0},
terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}},
process={time=24.0, query={time=0.0}, facet={time=0.0},
facet_module={time=0.0}, mlt={time=0.0}, highlight={time=0.0},
stats={time=0.0}, expand={time=0.0}, terms={time=0.0},
spellcheck={time=24.0}, debug={time=0.0}}}}}"
}
},
"GET_FIELDS": {
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/":
{
"QTime": "1",
"ElapsedTime": "3",
"RequestPurpose": "GET_FIELDS,GET_DEBUG,SET_TERM_STATS",
"NumFound": "3",
"Response": "{responseHeader={zkConnected=true, status=0,
QTime=1},
response={numFound=3,numFoundExact=true,start=0,docs=[SolrDocument{id=Position9,
TESTEMBEDDING_3=[0.8, 0.8, 0.8],
[shard]=
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/,
[explain]=0.0 = not in top 3\n, score=0.89285713},
SolrDocument{id=Position8, TESTEMBEDDING_3=[0.7, 0.7, 0.7],
[shard]=
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/,
[explain]=0.0 = not in top 3\n, score=0.78740156},
SolrDocument{id=Position6, TESTEMBEDDING_3=[0.5, 0.5, 0.5],
[shard]=
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/,
[explain]=0.0 = not in top 3\n, score=0.5714286}]},
debug={rawquerystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0],
querystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0],
parsedquery=KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]),
parsedquery_toString=KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3],
explain={Position9={match=false, value=0.0, description=not in top 3},
Position8={match=false, value=0.0, description=not in top 3},
Position6={match=false, value=0.0, description=not in top 3}},
QParser=KnnQParser, timing={time=1.0, prepare={time=0.0,
query={time=0.0}, facet={time=0.0}, facet_module={time=0.0},
mlt={time=0.0}, highlight={time=0.0}, stats={time=0.0},
expand={time=0.0}, terms={time=0.0}, spellcheck={time=0.0},
debug={time=0.0}}, process={time=0.0, query={time=0.0},
facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0},
highlight={time=0.0}, stats={time=0.0}, expand={time=0.0},
terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}}}}}"
},
"
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/":
{
"QTime": "1",
"ElapsedTime": "3",
"RequestPurpose": "GET_FIELDS,GET_DEBUG,SET_TERM_STATS",
"NumFound": "3",
"Response": "{responseHeader={zkConnected=true, status=0,
QTime=1},
response={numFound=3,numFoundExact=true,start=0,docs=[SolrDocument{id=Position10,
TESTEMBEDDING_3=[0.9, 0.9, 0.9],
[shard]=
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/,
[explain]=0.0 = not in top 3\n, score=0.97087383},
SolrDocument{id=Position11, TESTEMBEDDING_3=[1.0, 1.0, 1.0],
[shard]=
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/,
[explain]=0.0 = not in top 3\n, score=1.0}, SolrDocument{id=Position7,
TESTEMBEDDING_3=[0.6, 0.6, 0.6],
[shard]=
http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/,
[explain]=0.0 = not in top 3\n, score=0.6756757}]},
debug={rawquerystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0],
querystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0],
parsedquery=KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]),
parsedquery_toString=KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3],
explain={Position10={match=false, value=0.0, description=not in top 3},
Position11={match=false, value=0.0, description=not in top 3},
Position7={match=false, value=0.0, description=not in top 3}},
QParser=KnnQParser, timing={time=1.0, prepare={time=0.0,
query={time=0.0}, facet={time=0.0}, facet_module={time=0.0},
mlt={time=0.0}, highlight={time=0.0}, stats={time=0.0},
expand={time=0.0}, terms={time=0.0}, spellcheck={time=0.0},
debug={time=0.0}}, process={time=0.0, query={time=0.0},
facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0},
highlight={time=0.0}, stats={time=0.0}, expand={time=0.0},
terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}}}}}"
}
}
},
"timing": {
"time": 49.0,
"prepare": {
"time": 0.0,
"query": {
"time": 0.0
},
"facet": {
"time": 0.0
},
"facet_module": {
"time": 0.0
},
"mlt": {
"time": 0.0
},
"highlight": {
"time": 0.0
},
"stats": {
"time": 0.0
},
"expand": {
"time": 0.0
},
"terms": {
"time": 0.0
},
"spellcheck": {
"time": 0.0
},
"debug": {
"time": 0.0
}
},
"process": {
"time": 47.0,
"query": {
"time": 0.0
},
"facet": {
"time": 0.0
},
"facet_module": {
"time": 0.0
},
"mlt": {
"time": 0.0
},
"highlight": {
"time": 0.0
},
"stats": {
"time": 0.0
},
"expand": {
"time": 0.0
},
"terms": {
"time": 0.0
},
"spellcheck": {
"time": 47.0
},
"debug": {
"time": 0.0
}
}
},
"rawquerystring": "{!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0]",
"querystring": "{!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0]",
"parsedquery":
"KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3])",
"parsedquery_toString":
"KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]",
"QParser": "KnnQParser",
"explain": {
"Position11": {
"match": false,
"value": 0.0,
"description": "not in top 3"
},
"Position10": {
"match": false,
"value": 0.0,
"description": "not in top 3"
},
"Position9": {
"match": false,
"value": 0.0,
"description": "not in top 3"
},
"Position8": {
"match": false,
"value": 0.0,
"description": "not in top 3"
},
"Position7": {
"match": false,
"value": 0.0,
"description": "not in top 3"
},
"Position6": {
"match": false,
"value": 0.0,
"description": "not in top 3"
}
}
}
}
I got 6 search results. The top 3 of them are correct, and it seems that
two shards were queried, the results were combined and sorted correctly
by score. From my point of view is the result set size a functional bug,
as I requested the topK=3, but I got 6.
The explain data is completely random. The description is wrong, the
match flag is wrong, and if value should be the score, it is also wrong.
This is a bug in the explain implementation, and regarding to this a
debug bug.
Mirko
Am 18.10.2023 um 16:56 schrieb Alessandro Benedetti:
Let's distinguish if you are talking about a "debug" bug (the explain
message) or a functional bug.
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*
e-mail: a.benede...@sease.io
*Sease* - Information Retrieval Applied
Consulting | Training | Open Source
Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>
On Wed, 18 Oct 2023 at 16:17, Mirko Sertic <mirko.ser...@web.de> wrote:
Am 18.10.2023 um 15:44 schrieb Alessandro Benedetti:
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*
e-mail: a.benede...@sease.io
*Sease* - Information Retrieval Applied
Consulting | Training | Open Source
Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>
On Wed, 18 Oct 2023 at 15:32, Mirko Sertic <mirko.ser...@web.de>
wrote:
Hi Alessandro,
thank you for clarification! I havent't read something about "topK is
per shard" in the documentation, but maybe I've missed that somewhere.
From a performance and implementation perspective, this might make
sense. From a users perspective, I'd expect that "topK" is per
collection, and not per shard, so the same behavior as "rows". I
should't have to think about the collection data layout while
formulating queries and interpreting the layouts.
Now, if "topK" is per shard, we assume that good matching results are
evenly distributed among the shards. This might not be always the
case.
If we use features like collapse & expand, block/join or explicit
document routing strategies, this might lead to strange results. In
this
case, it might be better to retrieve the topK per shard, join them all
together and sort them by score, and take topK from that sorted list
to
get the correct overall topK. What do you think about that?
That is what currently *should* happen.
You can select the number of rows you like and get a subset of the
NShards*K results.
Are you observing something different?
Yes, if this does work correct, there should be no "not in top x" in
explain logs, but I see them, so I assume there is a bug somewhere.
We've also found https://issues.apache.org/jira/browse/SOLR-16952,
which
might be correlated.
That seems to me like a bug for the stored fields functionality.
I also tried to understand
https://github.com/apache/lucene/blob/723c05b047b4a66105636162bb3e0daaf2cc1de5/lucene/core/src/java/org/apache/lucene/search/AbstractKnnVectorQuery.java#L337
,
but failed to get the details.
I oberved the explain data further. Every document gets a score, which
is the result of the similarity function. However, some of them are
marked with "not in top 6 docs". So from my point of view those whould
not be part of the search result.
Can you elaborate on this? maybe through a full query and response with
the
debug?
Anyway, the debug score for vector-based search is currently almost
useless, the score is not really explained.
We know this and it's a Lucene level 'problem' (we haven't worked yet
on
a
better way for doing it, is your company interested in sponsoring maybe
:) )
Mirko
Am 17.10.2023 um 18:16 schrieb Alessandro Benedetti:
Hi Mirko,
the topK is per shard.
Then shards * k results are aggregated.
Does it make sense?
In regards to the debugging, it seems a bug, they all should be with
a
score and within top-k
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*
e-mail:a.benede...@sease.io
*Sease* - Information Retrieval Applied
Consulting | Training | Open Source
Website: Sease.io<http://sease.io/>
LinkedIn<https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>
On Tue, 17 Oct 2023 at 10:41, Mirko Sertic<mirko.ser...@web.de>
wrote:
Hey!
Thank you for your help!
We are running in cloud mode on GKE. Our index has 2 shards, and
every
shard has 2 replicas. The leader is a TLOG, the other replica is a
PULL.
Our main query is basically {!knn f=VECTOR_FIELD topK=10}[VECTOR
DATA].
Thats it.
I am really unsure how to debug this issue, Solr explain only gives
some
basic information. I really do not understand why I am getting more
than
topK=10 search results, and some of them do not match according to
the
explain data.
Here is an example debug log of the parsed query:
"rawquerystring" : "{!knn f=TEXTEMBEDDINGS_BESCHREIBUNG_768
topK=6}[-0.72818565,-0.41219202,0.5225889,0.53035444,0.76394194,0.7767894,0.82073385,0.95900697,-0.36803666,-0.21959017,0.7140104,-0.13439204,0.06747756,1.3333143,0.2797215,1.3020324,0.13869889,-0.10991136,0.21142228,-0.36915645,-0.15180133,-0.62384933,-0.51568866,0.23904686,-0.61362225,0.026911864,0.36433217,-0.6443799,0.86989623,-0.0058394694,0.360967,-0.39565668,0.4483029,-0.26269752,0.24516392,0.44815013,0.43040967,0.3587644,0.873287,-0.10215642,-0.25633913,-0.21385787,-0.5121543,-0.103228725,0.3895751,-0.49162507,-0.9432531,-0.5980949,0.63356507,-0.40455475,0.9550473,0.7282631,-0.031743232,0.439844,0.21040638,-0.36207297,0.22014715,-0.5069794,0.4957589,-0.30554834,-0.8615413,-0.4926672,-0.013558676,-0.041132603,0.65787005,0.30416572,0.18375356,0.6744972,-0.6142544,-0.18229552,-0.11830064,1.0248266,0.2171961,1.0895737,-0.027262354,-0.17439449,0.74050045,-0.8744221,-0.16733684,0.93963474,-0.4188998,-1.1815201,-0.4237022,-1.4382845,0.9899152,-0.17618449,0.8811465,-0.1435171,0.1467631,0.20906292,-0.19986387,-0.07172442,-0.5412453,-0.38141707,-0.8519122,0.57226634,0.27108946,0.47601715,-0.50961906,0.7443681,-0.46236074,0.44012043,-0.64782083,0.12072155,0.17669038,0.8894314,-0.26589298,-1.0005188,-0.38552156,-0.17849512,-0.41564974,0.46785104,0.07669882,0.098340474,-0.40940204,-0.22534837,-0.46240783,-0.6026122,-0.150852,-0.056922164,-0.5217692,0.2522323,-0.05158736,0.4368061,0.07538188,-0.15538858,-0.27512375,-7.942749,-0.5104796,0.3684156,0.17436846,0.33592805,0.30256566,0.3045547,-0.8253086,0.39595905,-0.63918144,0.023157855,-0.19350462,-1.1046336,0.24591433,-0.35902688,-0.80178124,0.60646725,-0.30873522,0.3227124,0.14232452,-0.11611875,0.36796737,-0.53184104,0.30364224,0.28488177,-1.0550518,-0.42094457,-0.12320968,-0.9765126,-0.15129066,0.001331389,-0.27246574,0.4749163,0.4177817,0.2973763,-0.29119635,-0.3972071,0.59532064,0.10756115,0.23748569,-0.70517355,1.201706,0.502714,-0.26869884,-0.053134415,-0.66120774,-0.0039092028,-0.64522946,-0.041144915,-0.05124693,-0.9726303,-0.99400455,-0.07065071,-0.78489596,0.80071807,-0.13281979,-0.006903738,0.5823942,-0.2669629,1.6502038,0.7482211,-0.6007372,0.66805166,-0.3942953,-0.5034327,0.27077916,0.15055649,0.10765681,-0.12882604,0.24207312,0.18053184,-0.21453774,0.45982376,-0.5745059,0.2006089,0.50371104,0.8538735,0.97700334,0.05624181,-0.132805,0.033950403,-0.18651533,-0.06351512,-0.7937849,-0.6627913,-0.3048031,0.53663576,0.17241986,0.36505452,-0.09964923,0.13906284,0.5434706,-0.36837682,-0.35080168,0.46206424,-0.20238751,-0.27848598,0.2480285,0.5826801,-0.48960462,0.7367855,1.0053717,0.17819782,0.38643655,-0.16025475,-0.35159552,-0.0024995431,-0.022663444,0.05895573,-0.19539028,-0.36216947,-0.06652315,-0.13567717,-0.68101376,-0.3490555,0.55224353,0.045317963,-0.8309715,-0.24363773,0.4269611,0.24411823,-0.06477823,0.80422884,0.35045695,0.20747428,-0.3475987,0.1297931,0.09917005,0.007867369,0.38570535,0.13428706,1.3826305,-0.8623333,-1.0942854,0.3748387,-0.5336166,-0.35972998,-0.10899716,0.278379,0.94615966,-0.93360096,-0.9554229,-0.28387132,-0.02770327,-0.9298182,0.45993057,-0.65919405,0.1475371,-0.34568012,1.209977,0.2802038,0.45577803,0.12152732,0.87355286,1.5883585,0.6779906,0.17098032,1.0125086,-0.55774075,-0.38352862,0.16628654,0.15557681,0.1861148,0.04379474,-0.7305415,-0.030636007,0.024982145,0.67828315,0.42840484,-0.053252134,-0.4504099,-0.20452605,-0.1081517,-0.14191566,0.07114916,-0.078530826,0.17243765,-0.46307424,-0.008914786,0.26029265,0.52127206,-0.35917446,-0.015462582,-0.28368863,-0.7689151,-0.263665,0.49229744,0.7343957,0.24539919,-0.2738248,-0.12916541,1.5979353,0.2385032,-0.5589315,0.1753772,-0.57815063,0.13694653,-0.20679545,0.78396624,-0.26194802,0.16952795,0.009313789,0.053571712,-0.50364834,0.19213165,0.020542603,0.3242613,-0.21481745,0.46329287,0.1883949,-0.53527766,0.19803612,-0.10915735,-0.53217936,0.5851483,0.8633852,-0.22306518,0.19917397,-0.48938546,-0.3489624,0.26114023,0.5178694,-0.49423835,0.22051883,0.9698861,-0.023483349,0.5058909,-0.19136174,-0.06710764,-0.50730103,-0.5724341,-0.07458467,0.72695184,0.36962357,0.12050099,0.16827017,-0.36094713,-0.22090997,0.5247373,0.5981514,0.11581371,-0.14274465,0.110738344,0.5218388,-0.38720345,0.9173073,-0.8510778,-0.22613077,-0.2748853,0.2863147,-0.3778913,-0.18058139,0.37012923,-0.5144021,-0.5551922,0.16914673,0.78963786,1.1504575,-0.7211657,-0.63688916,-0.075630635,-0.89228654,0.8719732,-0.16148148,-0.093216084,-0.0649338,0.40419587,-0.5091749,0.3257159,-0.38114238,-0.7489807,-0.14795141,0.070623375,1.0296395,-0.18232572,0.045388192,-0.5071435,-0.23788519,-0.60791194,-0.25087252,0.009456612,0.8095627,-0.43995717,-0.21105917,-0.83442384,0.7455656,-0.9289396,-0.26139355,0.33529952,-0.7209875,0.44793245,-0.11075375,0.35380912,-0.49433497,1.061043,0.24260058,1.277706,-1.0634571,-0.028423825,-0.6881516,-0.2748189,0.32650438,0.22231913,-0.6346664,-0.32749617,0.6449113,0.4030831,0.086880766,-0.34941757,-0.32754892,0.44228384,-1.0537914,-0.4920558,0.11018584,-0.4000114,0.19572248,-0.006614799,-0.1560454,-0.6138168,-0.2874727,0.08457174,0.852747,0.020211289,-0.46017483,0.05079806,0.61822194,0.16241436,-0.418595,-0.44193658,-0.23446727,0.08835063,-0.86544657,0.039239086,0.47766128,-0.31095138,-0.545759,-0.09576557,-0.5694708,-0.10043368,-0.2653149,-0.1599367,0.13423382,-0.1830842,0.046266567,-0.08192452,-0.75070643,-0.65771914,0.18250088,-0.36299148,0.33271846,-0.60936797,1.339671,-0.12343848,1.0515145,0.25372154,-0.4882932,-0.06304815,-0.17309606,0.30466563,0.08878487,-0.5156037,-0.33687803,0.5997892,0.1856752,0.5528808,0.5089958,-0.77673227,-0.43565074,-3.0061991,-0.22721101,1.1010107,1.285386,0.39106908,0.06541425,0.18982089,-0.33513412,-0.5287475,0.2362032,0.40598843,0.62325794,-0.8586378,-0.64406306,0.09112385,0.055086333,0.28247252,0.13180174,0.5969885,0.15263812,0.39325213,-0.45515132,-0.47813737,-0.33163494,0.3496457,0.39913225,-0.5750896,0.104484975,-0.69256973,0.18164606,0.09423768,0.045548365,0.51491827,-0.854532,0.40470743,-0.12821743,-0.22693269,-0.31501335,-0.53367645,0.4482267,-0.647523,-0.48177683,-0.45763886,0.31513563,0.115996234,-0.56461513,0.112646066,0.6897073,0.30277768,1.2858859,-0.47804725,0.36647198,0.8737555,0.89239925,-0.26573908,0.13890187,0.8274238,0.6068263,0.21134727,-0.5331567,-0.40894493,-0.20580018,-0.4116305,-8.852631E-4,0.71050125,0.18508287,0.3889124,-0.38231707,-0.7620664,-0.26053023,0.5970558,0.12187529,-0.35657012,-0.20820218,-0.4604194,-0.36569384,0.08593381,0.31331047,-0.12055105,0.12107455,0.45513165,-0.0023682665,0.3980597,0.1632141,0.33392814,-0.050680444,0.13038154,0.13847367,1.3034295,0.2744402,0.124595374,-0.53905815,0.35776177,-0.13275874,0.018385967,-0.16347891,-0.19726902,-0.4961019,0.07333634,0.52688265,0.20600657,0.14087541,-0.21112329,0.5135732,0.4315345,-0.020893982,0.41702232,0.07464256,-0.2845659,-0.13949631,-0.013015106,0.40522456,0.041121256,1.1591781,0.043223336,0.39874303,-0.30360883,-0.27349794,-0.04964445,-0.5729313,0.14069785,-0.33394682,0.9343837,-0.9350466,-0.8602237,-0.29167423,-1.5445148,0.040775828,-0.018334387,0.42681727,-0.1458294,0.49805698,0.27569738,-0.10979689,-0.5429115,-0.020050785,0.31627277,-0.6220587,0.061334964,0.13807471,0.34479836,0.11312207,0.796787,-0.48607716,-0.14010525,-0.61075526,0.6672587,1.6726193,0.4950563,0.41054416,0.2522166,-0.95681673,-1.0119423,-0.4870938,0.48430035,-0.38044646,0.59904295,0.23687701,0.93788594,-0.22263275,0.07065558,-0.4562675,-0.08655307,0.69646436,-0.5812278,-0.30097616,-0.9931224,0.23148312,0.13233739,0.74470276,0.014401446,0.08970716,0.23342986,-0.4664313,0.054693338,0.2778716,0.4019597,-0.97575116,0.058492173,-0.5017976,0.09864781,-0.3473021,-0.4823157,0.13100009,0.4488969,-0.1672722,-1.0326554,-0.18146743,-0.349057,0.61888343,-0.18784277,-0.53318554,-0.2559318,-0.6182706,0.24759102,-0.71451706,0.7098109,0.53027123,0.36208203,0.23936976,-0.10664157,0.3863506,0.42203853,0.1808002,-0.22444467,1.0584201,-0.29779503,0.43873647,0.2835624,-0.76047546,0.37234184,0.2044634,0.5471627,0.4247879,0.4856389,-0.7011678,0.40757468,0.1918484,0.54874665,-0.1279645,0.5678282,0.02382508,0.18181127,-0.42465243,0.20996757,-0.48161522,-0.53782755,-0.1985728,-0.12512843,-0.47403875,0.44477355,-0.05491643,0.81035346,0.3633467,0.5024226,-0.07482373,-0.7337039,0.0675221,0.3952929,0.45677915,-0.29935122,-0.42017272,-0.80567694,0.2868536,-0.40015128,-0.060443062,-0.21671702,0.18321274,-0.20112215,-0.50728095,0.3698101,0.56871736,-0.0904237,-0.43735075,-0.79018253,-0.0739331,0.49475756,0.014863159,-0.47107005,0.14020085,0.28982484,0.32168245,-0.56485337,0.17436136,-0.0050138533,-0.9533212,0.036251593,-0.33072975,-0.87562966,-0.4379939,0.8052754,0.30534235,0.20573251]",
"querystring" : "{!knn f=TEXTEMBEDDINGS_BESCHREIBUNG_768
topK=6}[-0.72818565,-0.41219202,0.5225889,0.53035444,0.76394194,0.7767894,0.82073385,0.95900697,-0.36803666,-0.21959017,0.7140104,-0.13439204,0.06747756,1.3333143,0.2797215,1.3020324,0.13869889,-0.10991136,0.21142228,-0.36915645,-0.15180133,-0.62384933,-0.51568866,0.23904686,-0.61362225,0.026911864,0.36433217,-0.6443799,0.86989623,-0.0058394694,0.360967,-0.39565668,0.4483029,-0.26269752,0.24516392,0.44815013,0.43040967,0.3587644,0.873287,-0.10215642,-0.25633913,-0.21385787,-0.5121543,-0.103228725,0.3895751,-0.49162507,-0.9432531,-0.5980949,0.63356507,-0.40455475,0.9550473,0.7282631,-0.031743232,0.439844,0.21040638,-0.36207297,0.22014715,-0.5069794,0.4957589,-0.30554834,-0.8615413,-0.4926672,-0.013558676,-0.041132603,0.65787005,0.30416572,0.18375356,0.6744972,-0.6142544,-0.18229552,-0.11830064,1.0248266,0.2171961,1.0895737,-0.027262354,-0.17439449,0.74050045,-0.8744221,-0.16733684,0.93963474,-0.4188998,-1.1815201,-0.4237022,-1.4382845,0.9899152,-0.17618449,0.8811465,-0.1435171,0.1467631,0.20906292,-0.19986387,-0.07172442,-0.5412453,-0.38141707,-0.8519122,0.57226634,0.27108946,0.47601715,-0.50961906,0.7443681,-0.46236074,0.44012043,-0.64782083,0.12072155,0.17669038,0.8894314,-0.26589298,-1.0005188,-0.38552156,-0.17849512,-0.41564974,0.46785104,0.07669882,0.098340474,-0.40940204,-0.22534837,-0.46240783,-0.6026122,-0.150852,-0.056922164,-0.5217692,0.2522323,-0.05158736,0.4368061,0.07538188,-0.15538858,-0.27512375,-7.942749,-0.5104796,0.3684156,0.17436846,0.33592805,0.30256566,0.3045547,-0.8253086,0.39595905,-0.63918144,0.023157855,-0.19350462,-1.1046336,0.24591433,-0.35902688,-0.80178124,0.60646725,-0.30873522,0.3227124,0.14232452,-0.11611875,0.36796737,-0.53184104,0.30364224,0.28488177,-1.0550518,-0.42094457,-0.12320968,-0.9765126,-0.15129066,0.001331389,-0.27246574,0.4749163,0.4177817,0.2973763,-0.29119635,-0.3972071,0.59532064,0.10756115,0.23748569,-0.70517355,1.201706,0.502714,-0.26869884,-0.053134415,-0.66120774,-0.0039092028,-0.64522946,-0.041144915,-0.05124693,-0.9726303,-0.99400455,-0.07065071,-0.78489596,0.80071807,-0.13281979,-0.006903738,0.5823942,-0.2669629,1.6502038,0.7482211,-0.6007372,0.66805166,-0.3942953,-0.5034327,0.27077916,0.15055649,0.10765681,-0.12882604,0.24207312,0.18053184,-0.21453774,0.45982376,-0.5745059,0.2006089,0.50371104,0.8538735,0.97700334,0.05624181,-0.132805,0.033950403,-0.18651533,-0.06351512,-0.7937849,-0.6627913,-0.3048031,0.53663576,0.17241986,0.36505452,-0.09964923,0.13906284,0.5434706,-0.36837682,-0.35080168,0.46206424,-0.20238751,-0.27848598,0.2480285,0.5826801,-0.48960462,0.7367855,1.0053717,0.17819782,0.38643655,-0.16025475,-0.35159552,-0.0024995431,-0.022663444,0.05895573,-0.19539028,-0.36216947,-0.06652315,-0.13567717,-0.68101376,-0.3490555,0.55224353,0.045317963,-0.8309715,-0.24363773,0.4269611,0.24411823,-0.06477823,0.80422884,0.35045695,0.20747428,-0.3475987,0.1297931,0.09917005,0.007867369,0.38570535,0.13428706,1.3826305,-0.8623333,-1.0942854,0.3748387,-0.5336166,-0.35972998,-0.10899716,0.278379,0.94615966,-0.93360096,-0.9554229,-0.28387132,-0.02770327,-0.9298182,0.45993057,-0.65919405,0.1475371,-0.34568012,1.209977,0.2802038,0.45577803,0.12152732,0.87355286,1.5883585,0.6779906,0.17098032,1.0125086,-0.55774075,-0.38352862,0.16628654,0.15557681,0.1861148,0.04379474,-0.7305415,-0.030636007,0.024982145,0.67828315,0.42840484,-0.053252134,-0.4504099,-0.20452605,-0.1081517,-0.14191566,0.07114916,-0.078530826,0.17243765,-0.46307424,-0.008914786,0.26029265,0.52127206,-0.35917446,-0.015462582,-0.28368863,-0.7689151,-0.263665,0.49229744,0.7343957,0.24539919,-0.2738248,-0.12916541,1.5979353,0.2385032,-0.5589315,0.1753772,-0.57815063,0.13694653,-0.20679545,0.78396624,-0.26194802,0.16952795,0.009313789,0.053571712,-0.50364834,0.19213165,0.020542603,0.3242613,-0.21481745,0.46329287,0.1883949,-0.53527766,0.19803612,-0.10915735,-0.53217936,0.5851483,0.8633852,-0.22306518,0.19917397,-0.48938546,-0.3489624,0.26114023,0.5178694,-0.49423835,0.22051883,0.9698861,-0.023483349,0.5058909,-0.19136174,-0.06710764,-0.50730103,-0.5724341,-0.07458467,0.72695184,0.36962357,0.12050099,0.16827017,-0.36094713,-0.22090997,0.5247373,0.5981514,0.11581371,-0.14274465,0.110738344,0.5218388,-0.38720345,0.9173073,-0.8510778,-0.22613077,-0.2748853,0.2863147,-0.3778913,-0.18058139,0.37012923,-0.5144021,-0.5551922,0.16914673,0.78963786,1.1504575,-0.7211657,-0.63688916,-0.075630635,-0.89228654,0.8719732,-0.16148148,-0.093216084,-0.0649338,0.40419587,-0.5091749,0.3257159,-0.38114238,-0.7489807,-0.14795141,0.070623375,1.0296395,-0.18232572,0.045388192,-0.5071435,-0.23788519,-0.60791194,-0.25087252,0.009456612,0.8095627,-0.43995717,-0.21105917,-0.83442384,0.7455656,-0.9289396,-0.26139355,0.33529952,-0.7209875,0.44793245,-0.11075375,0.35380912,-0.49433497,1.061043,0.24260058,1.277706,-1.0634571,-0.028423825,-0.6881516,-0.2748189,0.32650438,0.22231913,-0.6346664,-0.32749617,0.6449113,0.4030831,0.086880766,-0.34941757,-0.32754892,0.44228384,-1.0537914,-0.4920558,0.11018584,-0.4000114,0.19572248,-0.006614799,-0.1560454,-0.6138168,-0.2874727,0.08457174,0.852747,0.020211289,-0.46017483,0.05079806,0.61822194,0.16241436,-0.418595,-0.44193658,-0.23446727,0.08835063,-0.86544657,0.039239086,0.47766128,-0.31095138,-0.545759,-0.09576557,-0.5694708,-0.10043368,-0.2653149,-0.1599367,0.13423382,-0.1830842,0.046266567,-0.08192452,-0.75070643,-0.65771914,0.18250088,-0.36299148,0.33271846,-0.60936797,1.339671,-0.12343848,1.0515145,0.25372154,-0.4882932,-0.06304815,-0.17309606,0.30466563,0.08878487,-0.5156037,-0.33687803,0.5997892,0.1856752,0.5528808,0.5089958,-0.77673227,-0.43565074,-3.0061991,-0.22721101,1.1010107,1.285386,0.39106908,0.06541425,0.18982089,-0.33513412,-0.5287475,0.2362032,0.40598843,0.62325794,-0.8586378,-0.64406306,0.09112385,0.055086333,0.28247252,0.13180174,0.5969885,0.15263812,0.39325213,-0.45515132,-0.47813737,-0.33163494,0.3496457,0.39913225,-0.5750896,0.104484975,-0.69256973,0.18164606,0.09423768,0.045548365,0.51491827,-0.854532,0.40470743,-0.12821743,-0.22693269,-0.31501335,-0.53367645,0.4482267,-0.647523,-0.48177683,-0.45763886,0.31513563,0.115996234,-0.56461513,0.112646066,0.6897073,0.30277768,1.2858859,-0.47804725,0.36647198,0.8737555,0.89239925,-0.26573908,0.13890187,0.8274238,0.6068263,0.21134727,-0.5331567,-0.40894493,-0.20580018,-0.4116305,-8.852631E-4,0.71050125,0.18508287,0.3889124,-0.38231707,-0.7620664,-0.26053023,0.5970558,0.12187529,-0.35657012,-0.20820218,-0.4604194,-0.36569384,0.08593381,0.31331047,-0.12055105,0.12107455,0.45513165,-0.0023682665,0.3980597,0.1632141,0.33392814,-0.050680444,0.13038154,0.13847367,1.3034295,0.2744402,0.124595374,-0.53905815,0.35776177,-0.13275874,0.018385967,-0.16347891,-0.19726902,-0.4961019,0.07333634,0.52688265,0.20600657,0.14087541,-0.21112329,0.5135732,0.4315345,-0.020893982,0.41702232,0.07464256,-0.2845659,-0.13949631,-0.013015106,0.40522456,0.041121256,1.1591781,0.043223336,0.39874303,-0.30360883,-0.27349794,-0.04964445,-0.5729313,0.14069785,-0.33394682,0.9343837,-0.9350466,-0.8602237,-0.29167423,-1.5445148,0.040775828,-0.018334387,0.42681727,-0.1458294,0.49805698,0.27569738,-0.10979689,-0.5429115,-0.020050785,0.31627277,-0.6220587,0.061334964,0.13807471,0.34479836,0.11312207,0.796787,-0.48607716,-0.14010525,-0.61075526,0.6672587,1.6726193,0.4950563,0.41054416,0.2522166,-0.95681673,-1.0119423,-0.4870938,0.48430035,-0.38044646,0.59904295,0.23687701,0.93788594,-0.22263275,0.07065558,-0.4562675,-0.08655307,0.69646436,-0.5812278,-0.30097616,-0.9931224,0.23148312,0.13233739,0.74470276,0.014401446,0.08970716,0.23342986,-0.4664313,0.054693338,0.2778716,0.4019597,-0.97575116,0.058492173,-0.5017976,0.09864781,-0.3473021,-0.4823157,0.13100009,0.4488969,-0.1672722,-1.0326554,-0.18146743,-0.349057,0.61888343,-0.18784277,-0.53318554,-0.2559318,-0.6182706,0.24759102,-0.71451706,0.7098109,0.53027123,0.36208203,0.23936976,-0.10664157,0.3863506,0.42203853,0.1808002,-0.22444467,1.0584201,-0.29779503,0.43873647,0.2835624,-0.76047546,0.37234184,0.2044634,0.5471627,0.4247879,0.4856389,-0.7011678,0.40757468,0.1918484,0.54874665,-0.1279645,0.5678282,0.02382508,0.18181127,-0.42465243,0.20996757,-0.48161522,-0.53782755,-0.1985728,-0.12512843,-0.47403875,0.44477355,-0.05491643,0.81035346,0.3633467,0.5024226,-0.07482373,-0.7337039,0.0675221,0.3952929,0.45677915,-0.29935122,-0.42017272,-0.80567694,0.2868536,-0.40015128,-0.060443062,-0.21671702,0.18321274,-0.20112215,-0.50728095,0.3698101,0.56871736,-0.0904237,-0.43735075,-0.79018253,-0.0739331,0.49475756,0.014863159,-0.47107005,0.14020085,0.28982484,0.32168245,-0.56485337,0.17436136,-0.0050138533,-0.9533212,0.036251593,-0.33072975,-0.87562966,-0.4379939,0.8052754,0.30534235,0.20573251]",
"parsedquery" :
"KnnVectorQuery(KnnVectorQuery:TEXTEMBEDDINGS_BESCHREIBUNG_768[-0.72818565,...][6])",
"parsedquery_toString" :
"KnnVectorQuery:TEXTEMBEDDINGS_BESCHREIBUNG_768[-0.72818565,...][6]",
"QParser" : "KnnQParser",
Am 17.10.2023 um 10:05 schrieb Alessandro Benedetti:
What's your full Solr query?
Are you on SolrCloud or single Solr node?
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*
e-mail:a.benede...@sease.io
*Sease* - Information Retrieval Applied
Consulting | Training | Open Source
Website: Sease.io<http://sease.io/>
LinkedIn<https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> |
Github
<https://github.com/seaseltd>
On Tue, 17 Oct 2023 at 09:45, Mirko Sertic<mirko.ser...@web.de>
wrote:
To correct me, there was a typo. I meant:
If I specify topK=6, I get numFound=12, but only some of them
match
the
top 6
Am 17.10.2023 um 09:31 schrieb Mirko Sertic:
Hi!
To keep you updated, here are some observations regarding the
numFound/resultset size and DenseVectorQueries:
If I specity topK=10, I get numFound=20, but only some of them
match
the
top 10
If I specify topK=8, I get numFound=16, but only some of them
match
the
top 8
If I specify topK=6, I get numFound=8, but only some of them
match
the
top 6
So the numFound seems always to be the double of topK. Might
there
be a
correlation with sharding? Our collection has two shards, so does
this
double the results? I would't expect that, but that might be the
only
thing relating to a constant 2 in our setup.
Mirko
Am 16.10.2023 um 14:46 schrieb Mirko Sertic:
Hi@all
We are using Solr 9.1.1, and are trying usecases with
DenseVector
Fields
and knnQueries in mind.
During our tests, we see the following results and are trying to
figure
out what is going on:
a) We use the following main query : {!knn f=VECTOR_FIELD
topK=10}[VECTOR DATA]. We use it as a main query because we want
to
apply the distance function to the document score. However,
when I
try
do debug and explain the search results, I am getting more than
topK=10
result documents, some are marked as match = true with "within
top
10",
others are marked as match = false with "not in top 10". I'd
expect
that
only matched documents are part of the search result, but there
are
20
result documents, but only 5 of them are matched. Did I miss
something?
b) The knn query results are the approximate nearest neighbors,
but
they
might not be the best. We'd like to define some kind of cut-off
value
for knn document scores. Is this possible, and what would be a
good
day
to do so? Implement a post-processing filter query with an
frange
on
the
score field?
Thank you all,
Mirko
