-------------------------- *Alessandro Benedetti* Director @ Sease Ltd. *Apache Lucene/Solr Committer* *Apache Solr PMC Member*
e-mail: a.benede...@sease.io *Sease* - Information Retrieval Applied Consulting | Training | Open Source Website: Sease.io <http://sease.io/> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter <https://twitter.com/seaseltd> | Youtube <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github <https://github.com/seaseltd> On Wed, 18 Oct 2023 at 15:32, Mirko Sertic <mirko.ser...@web.de> wrote: > Hi Alessandro, > thank you for clarification! I havent't read something about "topK is > per shard" in the documentation, but maybe I've missed that somewhere. > From a performance and implementation perspective, this might make > sense. From a users perspective, I'd expect that "topK" is per > collection, and not per shard, so the same behavior as "rows". I > should't have to think about the collection data layout while > formulating queries and interpreting the layouts. > Now, if "topK" is per shard, we assume that good matching results are > evenly distributed among the shards. This might not be always the case. > If we use features like collapse & expand, block/join or explicit > document routing strategies, this might lead to strange results. In this > case, it might be better to retrieve the topK per shard, join them all > together and sort them by score, and take topK from that sorted list to > get the correct overall topK. What do you think about that? > That is what currently *should* happen. You can select the number of rows you like and get a subset of the NShards*K results. Are you observing something different? > We've also found https://issues.apache.org/jira/browse/SOLR-16952, which > might be correlated. > That seems to me like a bug for the stored fields functionality. > > I also tried to understand > > https://github.com/apache/lucene/blob/723c05b047b4a66105636162bb3e0daaf2cc1de5/lucene/core/src/java/org/apache/lucene/search/AbstractKnnVectorQuery.java#L337, > > but failed to get the details. > > I oberved the explain data further. Every document gets a score, which > is the result of the similarity function. However, some of them are > marked with "not in top 6 docs". So from my point of view those whould > not be part of the search result. > Can you elaborate on this? maybe through a full query and response with the debug? Anyway, the debug score for vector-based search is currently almost useless, the score is not really explained. We know this and it's a Lucene level 'problem' (we haven't worked yet on a better way for doing it, is your company interested in sponsoring maybe :) ) > Mirko > > Am 17.10.2023 um 18:16 schrieb Alessandro Benedetti: > > Hi Mirko, > > the topK is per shard. > > Then shards * k results are aggregated. > > Does it make sense? > > > > In regards to the debugging, it seems a bug, they all should be with a > > score and within top-k > > -------------------------- > > *Alessandro Benedetti* > > Director @ Sease Ltd. > > *Apache Lucene/Solr Committer* > > *Apache Solr PMC Member* > > > > e-mail:a.benede...@sease.io > > > > > > *Sease* - Information Retrieval Applied > > Consulting | Training | Open Source > > > > Website: Sease.io<http://sease.io/> > > LinkedIn<https://linkedin.com/company/sease-ltd> | Twitter > > <https://twitter.com/seaseltd> | Youtube > > <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github > > <https://github.com/seaseltd> > > > > > > On Tue, 17 Oct 2023 at 10:41, Mirko Sertic<mirko.ser...@web.de> wrote: > > > >> Hey! > >> > >> Thank you for your help! > >> > >> We are running in cloud mode on GKE. Our index has 2 shards, and every > >> shard has 2 replicas. The leader is a TLOG, the other replica is a PULL. > >> > >> Our main query is basically {!knn f=VECTOR_FIELD topK=10}[VECTOR DATA]. > >> Thats it. > >> > >> I am really unsure how to debug this issue, Solr explain only gives some > >> basic information. I really do not understand why I am getting more than > >> topK=10 search results, and some of them do not match according to the > >> explain data. > >> > >> Here is an example debug log of the parsed query: > >> > >> "rawquerystring" : "{!knn f=TEXTEMBEDDINGS_BESCHREIBUNG_768 > >> > topK=6}[-0.72818565,-0.41219202,0.5225889,0.53035444,0.76394194,0.7767894,0.82073385,0.95900697,-0.36803666,-0.21959017,0.7140104,-0.13439204,0.06747756,1.3333143,0.2797215,1.3020324,0.13869889,-0.10991136,0.21142228,-0.36915645,-0.15180133,-0.62384933,-0.51568866,0.23904686,-0.61362225,0.026911864,0.36433217,-0.6443799,0.86989623,-0.0058394694,0.360967,-0.39565668,0.4483029,-0.26269752,0.24516392,0.44815013,0.43040967,0.3587644,0.873287,-0.10215642,-0.25633913,-0.21385787,-0.5121543,-0.103228725,0.3895751,-0.49162507,-0.9432531,-0.5980949,0.63356507,-0.40455475,0.9550473,0.7282631,-0.031743232,0.439844,0.21040638,-0.36207297,0.22014715,-0.5069794,0.4957589,-0.30554834,-0.8615413,-0.4926672,-0.013558676,-0.041132603,0.65787005,0.30416572,0.18375356,0.6744972,-0.6142544,-0.18229552,-0.11830064,1.0248266,0.2171961,1.0895737,-0.027262354,-0.17439449,0.74050045,-0.8744221,-0.16733684,0.93963474,-0.4188998,-1.1815201,-0.4237022,-1.4382845,0.9899152,-0.17618449,0.8811465,-0.1435171,0.1467631,0.20906292,-0.19986387,-0.07172442,-0.5412453,-0.38141707,-0.8519122,0.57226634,0.27108946,0.47601715,-0.50961906,0.7443681,-0.46236074,0.44012043,-0.64782083,0.12072155,0.17669038,0.8894314,-0.26589298,-1.0005188,-0.38552156,-0.17849512,-0.41564974,0.46785104,0.07669882,0.098340474,-0.40940204,-0.22534837,-0.46240783,-0.6026122,-0.150852,-0.056922164,-0.5217692,0.2522323,-0.05158736,0.4368061,0.07538188,-0.15538858,-0.27512375,-7.942749,-0.5104796,0.3684156,0.17436846,0.33592805,0.30256566,0.3045547,-0.8253086,0.39595905,-0.63918144,0.023157855,-0.19350462,-1.1046336,0.24591433,-0.35902688,-0.80178124,0.60646725,-0.30873522,0.3227124,0.14232452,-0.11611875,0.36796737,-0.53184104,0.30364224,0.28488177,-1.0550518,-0.42094457,-0.12320968,-0.9765126,-0.15129066,0.001331389,-0.27246574,0.4749163,0.4177817,0.2973763,-0.29119635,-0.3972071,0.59532064,0.10756115,0.23748569,-0.70517355,1.201706,0.502714,-0.26869884,-0.053134415,-0.66120774,-0.0039092028,-0.64522946,-0.041144915,-0.05124693,-0.9726303,-0.99400455,-0.07065071,-0.78489596,0.80071807,-0.13281979,-0.006903738,0.5823942,-0.2669629,1.6502038,0.7482211,-0.6007372,0.66805166,-0.3942953,-0.5034327,0.27077916,0.15055649,0.10765681,-0.12882604,0.24207312,0.18053184,-0.21453774,0.45982376,-0.5745059,0.2006089,0.50371104,0.8538735,0.97700334,0.05624181,-0.132805,0.033950403,-0.18651533,-0.06351512,-0.7937849,-0.6627913,-0.3048031,0.53663576,0.17241986,0.36505452,-0.09964923,0.13906284,0.5434706,-0.36837682,-0.35080168,0.46206424,-0.20238751,-0.27848598,0.2480285,0.5826801,-0.48960462,0.7367855,1.0053717,0.17819782,0.38643655,-0.16025475,-0.35159552,-0.0024995431,-0.022663444,0.05895573,-0.19539028,-0.36216947,-0.06652315,-0.13567717,-0.68101376,-0.3490555,0.55224353,0.045317963,-0.8309715,-0.24363773,0.4269611,0.24411823,-0.06477823,0.80422884,0.35045695,0.20747428,-0.3475987,0.1297931,0.09917005,0.007867369,0.38570535,0.13428706,1.3826305,-0.8623333,-1.0942854,0.3748387,-0.5336166,-0.35972998,-0.10899716,0.278379,0.94615966,-0.93360096,-0.9554229,-0.28387132,-0.02770327,-0.9298182,0.45993057,-0.65919405,0.1475371,-0.34568012,1.209977,0.2802038,0.45577803,0.12152732,0.87355286,1.5883585,0.6779906,0.17098032,1.0125086,-0.55774075,-0.38352862,0.16628654,0.15557681,0.1861148,0.04379474,-0.7305415,-0.030636007,0.024982145,0.67828315,0.42840484,-0.053252134,-0.4504099,-0.20452605,-0.1081517,-0.14191566,0.07114916,-0.078530826,0.17243765,-0.46307424,-0.008914786,0.26029265,0.52127206,-0.35917446,-0.015462582,-0.28368863,-0.7689151,-0.263665,0.49229744,0.7343957,0.24539919,-0.2738248,-0.12916541,1.5979353,0.2385032,-0.5589315,0.1753772,-0.57815063,0.13694653,-0.20679545,0.78396624,-0.26194802,0.16952795,0.009313789,0.053571712,-0.50364834,0.19213165,0.020542603,0.3242613,-0.21481745,0.46329287,0.1883949,-0.53527766,0.19803612,-0.10915735,-0.53217936,0.5851483,0.8633852,-0.22306518,0.19917397,-0.48938546,-0.3489624,0.26114023,0.5178694,-0.49423835,0.22051883,0.9698861,-0.023483349,0.5058909,-0.19136174,-0.06710764,-0.50730103,-0.5724341,-0.07458467,0.72695184,0.36962357,0.12050099,0.16827017,-0.36094713,-0.22090997,0.5247373,0.5981514,0.11581371,-0.14274465,0.110738344,0.5218388,-0.38720345,0.9173073,-0.8510778,-0.22613077,-0.2748853,0.2863147,-0.3778913,-0.18058139,0.37012923,-0.5144021,-0.5551922,0.16914673,0.78963786,1.1504575,-0.7211657,-0.63688916,-0.075630635,-0.89228654,0.8719732,-0.16148148,-0.093216084,-0.0649338,0.40419587,-0.5091749,0.3257159,-0.38114238,-0.7489807,-0.14795141,0.070623375,1.0296395,-0.18232572,0.045388192,-0.5071435,-0.23788519,-0.60791194,-0.25087252,0.009456612,0.8095627,-0.43995717,-0.21105917,-0.83442384,0.7455656,-0.9289396,-0.26139355,0.33529952,-0.7209875,0.44793245,-0.11075375,0.35380912,-0.49433497,1.061043,0.24260058,1.277706,-1.0634571,-0.028423825,-0.6881516,-0.2748189,0.32650438,0.22231913,-0.6346664,-0.32749617,0.6449113,0.4030831,0.086880766,-0.34941757,-0.32754892,0.44228384,-1.0537914,-0.4920558,0.11018584,-0.4000114,0.19572248,-0.006614799,-0.1560454,-0.6138168,-0.2874727,0.08457174,0.852747,0.020211289,-0.46017483,0.05079806,0.61822194,0.16241436,-0.418595,-0.44193658,-0.23446727,0.08835063,-0.86544657,0.039239086,0.47766128,-0.31095138,-0.545759,-0.09576557,-0.5694708,-0.10043368,-0.2653149,-0.1599367,0.13423382,-0.1830842,0.046266567,-0.08192452,-0.75070643,-0.65771914,0.18250088,-0.36299148,0.33271846,-0.60936797,1.339671,-0.12343848,1.0515145,0.25372154,-0.4882932,-0.06304815,-0.17309606,0.30466563,0.08878487,-0.5156037,-0.33687803,0.5997892,0.1856752,0.5528808,0.5089958,-0.77673227,-0.43565074,-3.0061991,-0.22721101,1.1010107,1.285386,0.39106908,0.06541425,0.18982089,-0.33513412,-0.5287475,0.2362032,0.40598843,0.62325794,-0.8586378,-0.64406306,0.09112385,0.055086333,0.28247252,0.13180174,0.5969885,0.15263812,0.39325213,-0.45515132,-0.47813737,-0.33163494,0.3496457,0.39913225,-0.5750896,0.104484975,-0.69256973,0.18164606,0.09423768,0.045548365,0.51491827,-0.854532,0.40470743,-0.12821743,-0.22693269,-0.31501335,-0.53367645,0.4482267,-0.647523,-0.48177683,-0.45763886,0.31513563,0.115996234,-0.56461513,0.112646066,0.6897073,0.30277768,1.2858859,-0.47804725,0.36647198,0.8737555,0.89239925,-0.26573908,0.13890187,0.8274238,0.6068263,0.21134727,-0.5331567,-0.40894493,-0.20580018,-0.4116305,-8.852631E-4,0.71050125,0.18508287,0.3889124,-0.38231707,-0.7620664,-0.26053023,0.5970558,0.12187529,-0.35657012,-0.20820218,-0.4604194,-0.36569384,0.08593381,0.31331047,-0.12055105,0.12107455,0.45513165,-0.0023682665,0.3980597,0.1632141,0.33392814,-0.050680444,0.13038154,0.13847367,1.3034295,0.2744402,0.124595374,-0.53905815,0.35776177,-0.13275874,0.018385967,-0.16347891,-0.19726902,-0.4961019,0.07333634,0.52688265,0.20600657,0.14087541,-0.21112329,0.5135732,0.4315345,-0.020893982,0.41702232,0.07464256,-0.2845659,-0.13949631,-0.013015106,0.40522456,0.041121256,1.1591781,0.043223336,0.39874303,-0.30360883,-0.27349794,-0.04964445,-0.5729313,0.14069785,-0.33394682,0.9343837,-0.9350466,-0.8602237,-0.29167423,-1.5445148,0.040775828,-0.018334387,0.42681727,-0.1458294,0.49805698,0.27569738,-0.10979689,-0.5429115,-0.020050785,0.31627277,-0.6220587,0.061334964,0.13807471,0.34479836,0.11312207,0.796787,-0.48607716,-0.14010525,-0.61075526,0.6672587,1.6726193,0.4950563,0.41054416,0.2522166,-0.95681673,-1.0119423,-0.4870938,0.48430035,-0.38044646,0.59904295,0.23687701,0.93788594,-0.22263275,0.07065558,-0.4562675,-0.08655307,0.69646436,-0.5812278,-0.30097616,-0.9931224,0.23148312,0.13233739,0.74470276,0.014401446,0.08970716,0.23342986,-0.4664313,0.054693338,0.2778716,0.4019597,-0.97575116,0.058492173,-0.5017976,0.09864781,-0.3473021,-0.4823157,0.13100009,0.4488969,-0.1672722,-1.0326554,-0.18146743,-0.349057,0.61888343,-0.18784277,-0.53318554,-0.2559318,-0.6182706,0.24759102,-0.71451706,0.7098109,0.53027123,0.36208203,0.23936976,-0.10664157,0.3863506,0.42203853,0.1808002,-0.22444467,1.0584201,-0.29779503,0.43873647,0.2835624,-0.76047546,0.37234184,0.2044634,0.5471627,0.4247879,0.4856389,-0.7011678,0.40757468,0.1918484,0.54874665,-0.1279645,0.5678282,0.02382508,0.18181127,-0.42465243,0.20996757,-0.48161522,-0.53782755,-0.1985728,-0.12512843,-0.47403875,0.44477355,-0.05491643,0.81035346,0.3633467,0.5024226,-0.07482373,-0.7337039,0.0675221,0.3952929,0.45677915,-0.29935122,-0.42017272,-0.80567694,0.2868536,-0.40015128,-0.060443062,-0.21671702,0.18321274,-0.20112215,-0.50728095,0.3698101,0.56871736,-0.0904237,-0.43735075,-0.79018253,-0.0739331,0.49475756,0.014863159,-0.47107005,0.14020085,0.28982484,0.32168245,-0.56485337,0.17436136,-0.0050138533,-0.9533212,0.036251593,-0.33072975,-0.87562966,-0.4379939,0.8052754,0.30534235,0.20573251]", > >> > >> "querystring" : "{!knn f=TEXTEMBEDDINGS_BESCHREIBUNG_768 > >> > topK=6}[-0.72818565,-0.41219202,0.5225889,0.53035444,0.76394194,0.7767894,0.82073385,0.95900697,-0.36803666,-0.21959017,0.7140104,-0.13439204,0.06747756,1.3333143,0.2797215,1.3020324,0.13869889,-0.10991136,0.21142228,-0.36915645,-0.15180133,-0.62384933,-0.51568866,0.23904686,-0.61362225,0.026911864,0.36433217,-0.6443799,0.86989623,-0.0058394694,0.360967,-0.39565668,0.4483029,-0.26269752,0.24516392,0.44815013,0.43040967,0.3587644,0.873287,-0.10215642,-0.25633913,-0.21385787,-0.5121543,-0.103228725,0.3895751,-0.49162507,-0.9432531,-0.5980949,0.63356507,-0.40455475,0.9550473,0.7282631,-0.031743232,0.439844,0.21040638,-0.36207297,0.22014715,-0.5069794,0.4957589,-0.30554834,-0.8615413,-0.4926672,-0.013558676,-0.041132603,0.65787005,0.30416572,0.18375356,0.6744972,-0.6142544,-0.18229552,-0.11830064,1.0248266,0.2171961,1.0895737,-0.027262354,-0.17439449,0.74050045,-0.8744221,-0.16733684,0.93963474,-0.4188998,-1.1815201,-0.4237022,-1.4382845,0.9899152,-0.17618449,0.8811465,-0.1435171,0.1467631,0.20906292,-0.19986387,-0.07172442,-0.5412453,-0.38141707,-0.8519122,0.57226634,0.27108946,0.47601715,-0.50961906,0.7443681,-0.46236074,0.44012043,-0.64782083,0.12072155,0.17669038,0.8894314,-0.26589298,-1.0005188,-0.38552156,-0.17849512,-0.41564974,0.46785104,0.07669882,0.098340474,-0.40940204,-0.22534837,-0.46240783,-0.6026122,-0.150852,-0.056922164,-0.5217692,0.2522323,-0.05158736,0.4368061,0.07538188,-0.15538858,-0.27512375,-7.942749,-0.5104796,0.3684156,0.17436846,0.33592805,0.30256566,0.3045547,-0.8253086,0.39595905,-0.63918144,0.023157855,-0.19350462,-1.1046336,0.24591433,-0.35902688,-0.80178124,0.60646725,-0.30873522,0.3227124,0.14232452,-0.11611875,0.36796737,-0.53184104,0.30364224,0.28488177,-1.0550518,-0.42094457,-0.12320968,-0.9765126,-0.15129066,0.001331389,-0.27246574,0.4749163,0.4177817,0.2973763,-0.29119635,-0.3972071,0.59532064,0.10756115,0.23748569,-0.70517355,1.201706,0.502714,-0.26869884,-0.053134415,-0.66120774,-0.0039092028,-0.64522946,-0.041144915,-0.05124693,-0.9726303,-0.99400455,-0.07065071,-0.78489596,0.80071807,-0.13281979,-0.006903738,0.5823942,-0.2669629,1.6502038,0.7482211,-0.6007372,0.66805166,-0.3942953,-0.5034327,0.27077916,0.15055649,0.10765681,-0.12882604,0.24207312,0.18053184,-0.21453774,0.45982376,-0.5745059,0.2006089,0.50371104,0.8538735,0.97700334,0.05624181,-0.132805,0.033950403,-0.18651533,-0.06351512,-0.7937849,-0.6627913,-0.3048031,0.53663576,0.17241986,0.36505452,-0.09964923,0.13906284,0.5434706,-0.36837682,-0.35080168,0.46206424,-0.20238751,-0.27848598,0.2480285,0.5826801,-0.48960462,0.7367855,1.0053717,0.17819782,0.38643655,-0.16025475,-0.35159552,-0.0024995431,-0.022663444,0.05895573,-0.19539028,-0.36216947,-0.06652315,-0.13567717,-0.68101376,-0.3490555,0.55224353,0.045317963,-0.8309715,-0.24363773,0.4269611,0.24411823,-0.06477823,0.80422884,0.35045695,0.20747428,-0.3475987,0.1297931,0.09917005,0.007867369,0.38570535,0.13428706,1.3826305,-0.8623333,-1.0942854,0.3748387,-0.5336166,-0.35972998,-0.10899716,0.278379,0.94615966,-0.93360096,-0.9554229,-0.28387132,-0.02770327,-0.9298182,0.45993057,-0.65919405,0.1475371,-0.34568012,1.209977,0.2802038,0.45577803,0.12152732,0.87355286,1.5883585,0.6779906,0.17098032,1.0125086,-0.55774075,-0.38352862,0.16628654,0.15557681,0.1861148,0.04379474,-0.7305415,-0.030636007,0.024982145,0.67828315,0.42840484,-0.053252134,-0.4504099,-0.20452605,-0.1081517,-0.14191566,0.07114916,-0.078530826,0.17243765,-0.46307424,-0.008914786,0.26029265,0.52127206,-0.35917446,-0.015462582,-0.28368863,-0.7689151,-0.263665,0.49229744,0.7343957,0.24539919,-0.2738248,-0.12916541,1.5979353,0.2385032,-0.5589315,0.1753772,-0.57815063,0.13694653,-0.20679545,0.78396624,-0.26194802,0.16952795,0.009313789,0.053571712,-0.50364834,0.19213165,0.020542603,0.3242613,-0.21481745,0.46329287,0.1883949,-0.53527766,0.19803612,-0.10915735,-0.53217936,0.5851483,0.8633852,-0.22306518,0.19917397,-0.48938546,-0.3489624,0.26114023,0.5178694,-0.49423835,0.22051883,0.9698861,-0.023483349,0.5058909,-0.19136174,-0.06710764,-0.50730103,-0.5724341,-0.07458467,0.72695184,0.36962357,0.12050099,0.16827017,-0.36094713,-0.22090997,0.5247373,0.5981514,0.11581371,-0.14274465,0.110738344,0.5218388,-0.38720345,0.9173073,-0.8510778,-0.22613077,-0.2748853,0.2863147,-0.3778913,-0.18058139,0.37012923,-0.5144021,-0.5551922,0.16914673,0.78963786,1.1504575,-0.7211657,-0.63688916,-0.075630635,-0.89228654,0.8719732,-0.16148148,-0.093216084,-0.0649338,0.40419587,-0.5091749,0.3257159,-0.38114238,-0.7489807,-0.14795141,0.070623375,1.0296395,-0.18232572,0.045388192,-0.5071435,-0.23788519,-0.60791194,-0.25087252,0.009456612,0.8095627,-0.43995717,-0.21105917,-0.83442384,0.7455656,-0.9289396,-0.26139355,0.33529952,-0.7209875,0.44793245,-0.11075375,0.35380912,-0.49433497,1.061043,0.24260058,1.277706,-1.0634571,-0.028423825,-0.6881516,-0.2748189,0.32650438,0.22231913,-0.6346664,-0.32749617,0.6449113,0.4030831,0.086880766,-0.34941757,-0.32754892,0.44228384,-1.0537914,-0.4920558,0.11018584,-0.4000114,0.19572248,-0.006614799,-0.1560454,-0.6138168,-0.2874727,0.08457174,0.852747,0.020211289,-0.46017483,0.05079806,0.61822194,0.16241436,-0.418595,-0.44193658,-0.23446727,0.08835063,-0.86544657,0.039239086,0.47766128,-0.31095138,-0.545759,-0.09576557,-0.5694708,-0.10043368,-0.2653149,-0.1599367,0.13423382,-0.1830842,0.046266567,-0.08192452,-0.75070643,-0.65771914,0.18250088,-0.36299148,0.33271846,-0.60936797,1.339671,-0.12343848,1.0515145,0.25372154,-0.4882932,-0.06304815,-0.17309606,0.30466563,0.08878487,-0.5156037,-0.33687803,0.5997892,0.1856752,0.5528808,0.5089958,-0.77673227,-0.43565074,-3.0061991,-0.22721101,1.1010107,1.285386,0.39106908,0.06541425,0.18982089,-0.33513412,-0.5287475,0.2362032,0.40598843,0.62325794,-0.8586378,-0.64406306,0.09112385,0.055086333,0.28247252,0.13180174,0.5969885,0.15263812,0.39325213,-0.45515132,-0.47813737,-0.33163494,0.3496457,0.39913225,-0.5750896,0.104484975,-0.69256973,0.18164606,0.09423768,0.045548365,0.51491827,-0.854532,0.40470743,-0.12821743,-0.22693269,-0.31501335,-0.53367645,0.4482267,-0.647523,-0.48177683,-0.45763886,0.31513563,0.115996234,-0.56461513,0.112646066,0.6897073,0.30277768,1.2858859,-0.47804725,0.36647198,0.8737555,0.89239925,-0.26573908,0.13890187,0.8274238,0.6068263,0.21134727,-0.5331567,-0.40894493,-0.20580018,-0.4116305,-8.852631E-4,0.71050125,0.18508287,0.3889124,-0.38231707,-0.7620664,-0.26053023,0.5970558,0.12187529,-0.35657012,-0.20820218,-0.4604194,-0.36569384,0.08593381,0.31331047,-0.12055105,0.12107455,0.45513165,-0.0023682665,0.3980597,0.1632141,0.33392814,-0.050680444,0.13038154,0.13847367,1.3034295,0.2744402,0.124595374,-0.53905815,0.35776177,-0.13275874,0.018385967,-0.16347891,-0.19726902,-0.4961019,0.07333634,0.52688265,0.20600657,0.14087541,-0.21112329,0.5135732,0.4315345,-0.020893982,0.41702232,0.07464256,-0.2845659,-0.13949631,-0.013015106,0.40522456,0.041121256,1.1591781,0.043223336,0.39874303,-0.30360883,-0.27349794,-0.04964445,-0.5729313,0.14069785,-0.33394682,0.9343837,-0.9350466,-0.8602237,-0.29167423,-1.5445148,0.040775828,-0.018334387,0.42681727,-0.1458294,0.49805698,0.27569738,-0.10979689,-0.5429115,-0.020050785,0.31627277,-0.6220587,0.061334964,0.13807471,0.34479836,0.11312207,0.796787,-0.48607716,-0.14010525,-0.61075526,0.6672587,1.6726193,0.4950563,0.41054416,0.2522166,-0.95681673,-1.0119423,-0.4870938,0.48430035,-0.38044646,0.59904295,0.23687701,0.93788594,-0.22263275,0.07065558,-0.4562675,-0.08655307,0.69646436,-0.5812278,-0.30097616,-0.9931224,0.23148312,0.13233739,0.74470276,0.014401446,0.08970716,0.23342986,-0.4664313,0.054693338,0.2778716,0.4019597,-0.97575116,0.058492173,-0.5017976,0.09864781,-0.3473021,-0.4823157,0.13100009,0.4488969,-0.1672722,-1.0326554,-0.18146743,-0.349057,0.61888343,-0.18784277,-0.53318554,-0.2559318,-0.6182706,0.24759102,-0.71451706,0.7098109,0.53027123,0.36208203,0.23936976,-0.10664157,0.3863506,0.42203853,0.1808002,-0.22444467,1.0584201,-0.29779503,0.43873647,0.2835624,-0.76047546,0.37234184,0.2044634,0.5471627,0.4247879,0.4856389,-0.7011678,0.40757468,0.1918484,0.54874665,-0.1279645,0.5678282,0.02382508,0.18181127,-0.42465243,0.20996757,-0.48161522,-0.53782755,-0.1985728,-0.12512843,-0.47403875,0.44477355,-0.05491643,0.81035346,0.3633467,0.5024226,-0.07482373,-0.7337039,0.0675221,0.3952929,0.45677915,-0.29935122,-0.42017272,-0.80567694,0.2868536,-0.40015128,-0.060443062,-0.21671702,0.18321274,-0.20112215,-0.50728095,0.3698101,0.56871736,-0.0904237,-0.43735075,-0.79018253,-0.0739331,0.49475756,0.014863159,-0.47107005,0.14020085,0.28982484,0.32168245,-0.56485337,0.17436136,-0.0050138533,-0.9533212,0.036251593,-0.33072975,-0.87562966,-0.4379939,0.8052754,0.30534235,0.20573251]", > >> > >> "parsedquery" : > >> > "KnnVectorQuery(KnnVectorQuery:TEXTEMBEDDINGS_BESCHREIBUNG_768[-0.72818565,...][6])", > >> > >> "parsedquery_toString" : > >> "KnnVectorQuery:TEXTEMBEDDINGS_BESCHREIBUNG_768[-0.72818565,...][6]", > >> "QParser" : "KnnQParser", > >> > >> Am 17.10.2023 um 10:05 schrieb Alessandro Benedetti: > >>> What's your full Solr query? > >>> Are you on SolrCloud or single Solr node? > >>> -------------------------- > >>> *Alessandro Benedetti* > >>> Director @ Sease Ltd. > >>> *Apache Lucene/Solr Committer* > >>> *Apache Solr PMC Member* > >>> > >>> e-mail:a.benede...@sease.io > >>> > >>> > >>> *Sease* - Information Retrieval Applied > >>> Consulting | Training | Open Source > >>> > >>> Website: Sease.io<http://sease.io/> > >>> LinkedIn<https://linkedin.com/company/sease-ltd> | Twitter > >>> <https://twitter.com/seaseltd> | Youtube > >>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github > >>> <https://github.com/seaseltd> > >>> > >>> > >>> On Tue, 17 Oct 2023 at 09:45, Mirko Sertic<mirko.ser...@web.de> > wrote: > >>> > >>>> To correct me, there was a typo. I meant: > >>>> > >>>> If I specify topK=6, I get numFound=12, but only some of them match > the > >>>> top 6 > >>>> > >>>> > >>>> Am 17.10.2023 um 09:31 schrieb Mirko Sertic: > >>>>> Hi! > >>>>> > >>>>> To keep you updated, here are some observations regarding the > >>>>> numFound/resultset size and DenseVectorQueries: > >>>>> > >>>>> If I specity topK=10, I get numFound=20, but only some of them match > >> the > >>>>> top 10 > >>>>> > >>>>> If I specify topK=8, I get numFound=16, but only some of them match > the > >>>>> top 8 > >>>>> > >>>>> If I specify topK=6, I get numFound=8, but only some of them match > the > >>>>> top 6 > >>>>> > >>>>> So the numFound seems always to be the double of topK. Might there > be a > >>>>> correlation with sharding? Our collection has two shards, so does > this > >>>>> double the results? I would't expect that, but that might be the only > >>>>> thing relating to a constant 2 in our setup. > >>>>> > >>>>> Mirko > >>>>> > >>>>> > >>>>> Am 16.10.2023 um 14:46 schrieb Mirko Sertic: > >>>>>> Hi@all > >>>>>> > >>>>>> We are using Solr 9.1.1, and are trying usecases with DenseVector > >> Fields > >>>>>> and knnQueries in mind. > >>>>>> > >>>>>> During our tests, we see the following results and are trying to > >> figure > >>>>>> out what is going on: > >>>>>> > >>>>>> a) We use the following main query : {!knn f=VECTOR_FIELD > >>>>>> topK=10}[VECTOR DATA]. We use it as a main query because we want to > >>>>>> apply the distance function to the document score. However, when I > try > >>>>>> do debug and explain the search results, I am getting more than > >> topK=10 > >>>>>> result documents, some are marked as match = true with "within top > >> 10", > >>>>>> others are marked as match = false with "not in top 10". I'd expect > >> that > >>>>>> only matched documents are part of the search result, but there are > 20 > >>>>>> result documents, but only 5 of them are matched. Did I miss > >> something? > >>>>>> b) The knn query results are the approximate nearest neighbors, but > >> they > >>>>>> might not be the best. We'd like to define some kind of cut-off > value > >>>>>> for knn document scores. Is this possible, and what would be a good > >> day > >>>>>> to do so? Implement a post-processing filter query with an frange on > >> the > >>>>>> score field? > >>>>>> > >>>>>> Thank you all, > >>>>>> > >>>>>> Mirko > >>>>>> >
