Dear Apache Solr Support Team,
I hope this message finds you well.
My name is Tatsuya Hirose, and I am currently working with Apache Solr9.2 for
vector search. I am writing to seek assistance on how to combine category
filtering and score thresholding in vector search queries.
In my current setup, I am executing a vector search using the q parameter, and
I am applying category filtering and score thresholding using the fq parameter.
However, I am experiencing an issue where even data that meets the score
threshold is being excluded.
Here is a brief overview of my query configuration:
・Vector search is performed using the q parameter.
・Category filtering and score thresholding are applied using the fq parameter.
Despite these settings, I am finding that results with scores above the
threshold are still being filtered out. Could you please provide guidance or
suggest any adjustments to ensure that only results below the score threshold
are excluded?
my query
========================================================================================
{responseHeader={status=0,QTime=36,params={q={!knn f=VECTOR_JOBS
topK=1000}[0.035584546625614166,-0.0037135074380785227,-0.09441375732421875,-0.08091471344232559,0.06334911286830902,-0.016544226557016373,-0.009578675031661987,0.02881624363362789,0.05436713621020317,0.014042883180081844,0.05005200207233429,-0.008916955441236496,0.07875142991542816,-0.04294541850686073,-0.06802155822515488,0.05358993634581566,0.04136655852198601,-0.054064180701971054,-0.02901504561305046,-0.05627920851111412,-0.005367881618440151,0.019279100000858307,-0.028985634446144104,0.05184251070022583,0.03146009147167206,0.011358415707945824,-0.005074767861515284,-0.010332686826586723,0.052167512476444244,-0.11000526696443558,-0.027527382597327232,-0.044672150164842606,0.05854484438896179,-0.07840650528669357,0.06903098523616791,0.049439240247011185,-0.054361239075660706,-0.02509576827287674,0.025177134200930595,-0.040781065821647644,-0.033132605254650116,0.014359967783093452,0.061371080577373505,0.07711537927389145,0.07437257468700409,0.03993475064635277,-0.054308004677295685,0.06438028812408447,-0.02820759452879429,-0.036603089421987534,-0.08848678320646286,0.047304052859544754,0.04506373777985573,0.05681953206658363,0.060880858451128006,-0.05436766520142555,-0.04058506712317467,-0.11434481292963028,-0.05068547651171684,0.04353500157594681,0.09206696599721909,-0.02258351258933544,0.008594350889325142,0.03408209607005119,0.039669834077358246,0.048566900193691254,0.018713651224970818,0.03889262676239014,-0.047088805586099625,-0.0567590668797493,-0.04277338832616806,0.07308576256036758,0.008183983154594898,-0.03881801292300224,0.00885037798434496,0.03731588274240494,0.03479887917637825,-0.04320204257965088,0.037571508437395096,-0.0679587572813034,-0.031020788475871086,-0.05468453839421272,-0.03124099038541317,0.025451287627220154,-0.05279769375920296,0.08654481172561646,0.058083001524209976,-0.06859175860881805,0.04565294459462166,-0.05859413370490074,0.02580122835934162,0.05507056787610054,-0.07341524213552475,-0.08186060935258865,-0.07237359881401062,-0.03577793017029762,-0.03482109680771828,0.06640908867120743,0.07913447916507721,-0.04478534683585167,0.07674097269773483,-0.027038617059588432,0.0427742563188076,-0.04823409020900726,-0.07627245783805847,0.05237693339586258,0.00942136999219656,-0.0769105777144432,0.05992821976542473,-0.07474396377801895,-0.012931128963828087,0.018789345398545265,0.06800083816051483,0.06447484344244003,-0.06841298192739487,-0.041461728513240814,-0.043237101286649704,-0.05871080234646797,0.07195127010345459,-0.04974381998181343,0.06874655932188034,0.020333755761384964,-0.07586508244276047,-0.06333187222480774,-0.06827986240386963,-0.06471151113510132,0.0454309843480587,0.06869687139987946,0.0492619164288044,0.007202852517366409,0.084236279129982,0.06838759034872055,0.03993704542517662,0.06425982713699341,0.0038626380264759064,0.03998879715800285,-0.06274054944515228,0.026097221300005913,-0.009520778432488441,-0.025847123935818672,-0.08365575969219208,0.07324723899364471,-0.038425151258707047,0.030306674540042877,0.030576184391975403,0.05077032744884491,0.03749245032668114,0.013652767054736614,0.09289221465587616,-0.015819767490029335,0.04610559716820717,-0.058255940675735474,0.03769712150096893,0.03284531086683273,0.06841275840997696,-0.04698965325951576,-0.04686743766069412,-0.03279430791735649,0.04285228252410889,0.03349098935723305,-0.05410241335630417,-0.05875148996710777,-0.0479816310107708,-0.0054592713713645935,-0.06065493822097778,-0.04231962561607361,-5.565312458202243E-4,0.04267852380871773,-0.0015223805094137788,-0.07954441756010056,-0.02864944189786911,0.03162918984889984,-0.0514996312558651,0.026252157986164093,-0.0019050539704039693,0.05508401244878769,-0.08239911496639252,0.06427738070487976,0.0693693682551384,0.026355179026722908,-0.032939378172159195,-0.02818630449473858,-0.07425744831562042,-0.05653354525566101,-0.02163071557879448,-0.06143616884946823,-0.05905051529407501,0.019536206498742104,-0.010454200208187103,-0.026982611045241356,-5.262328777462244E-4,0.037938859313726425,-0.007366125471889973,-0.08427233248949051,-0.037678882479667664,0.04340435564517975,-0.08494484424591064,0.03686724603176117,0.08409369736909866,0.04636300727725029,-0.023142173886299133,-0.013966536149382591,0.05344899371266365,0.020153826102614403,0.056393999606370926,-0.028801988810300827,-0.03857492282986641,0.02225646562874317,-0.027080746367573738,0.012224565260112286,0.06324844807386398,-0.07122324407100677,-0.05092167481780052,0.04811571538448334,-0.04632435739040375,-0.04115593060851097,0.016739698126912117,0.05709276720881462,-0.022160666063427925,0.04017924889922142,0.07590368390083313,-0.025377441197633743,0.04381873458623886,-0.06250565499067307,-0.00666261138394475,0.0628020316362381,0.04722044989466667,-0.07758677750825882,-0.04919694736599922,0.05749332159757614,-0.06679681688547134,-0.030204325914382935,-0.05486476048827171,-0.04788724333047867,-0.01270680595189333,-0.043820299208164215,0.006036344915628433,0.049858443439006805,0.03546955808997154,-0.0811769962310791,-0.03303926810622215,-0.036458633840084076,0.059668686240911484,-0.029341233894228935,0.05722843110561371,-0.037102941423654556,-0.05804651975631714,0.030349325388669968,0.010110113769769669,0.02846154384315014,-0.010385323315858841,-0.0857461467385292,-0.03958306461572647,-0.0704406201839447,-0.043374788016080856,0.05032896623015404,0.06337638199329376,0.07635693997144699,-0.053806841373443604,0.043615613132715225,0.052727892994880676,-0.050044573843479156,0.08947139233350754,0.09486859291791916,0.06327512860298157,0.04642516374588013,-0.05986626073718071,-0.040568966418504715,-0.07942715287208557,-0.05733555927872658,-0.0538201779127121,0.03840303421020508,0.08274640142917633,-0.058530423790216446,-0.03460042551159859,-0.03896292671561241,0.02606247365474701,0.09844136983156204,-0.03791554272174835,-0.0337119959294796,0.035287901759147644,0.05959693714976311,0.05371087044477463,0.04438423737883568,0.07312386482954025,-0.03839566558599472,-0.055226318538188934,0.050192445516586304,-0.04040166735649109,-0.03188833221793175,-0.043413687497377396,-0.03588910028338432,0.0401914045214653,-0.042271725833415985,0.07949542999267578,0.014143441803753376,0.019746867939829826,0.05431278049945831,-0.057717494666576385,0.039479516446590424,-2.7716034674085677E-4,-0.057753078639507294,0.042100805789232254,0.07823135703802109,-0.07230711728334427,0.030919041484594345,-0.011322765611112118,0.024991914629936218,-0.0162674468010664,0.010924405418336391,0.048790093511343,0.021293558180332184,-0.03683193773031235,-0.02963220328092575,0.09044580906629562,0.02953074499964714,-0.033115778118371964,0.04022584855556488,-0.0849866271018982,-0.061854250729084015,0.029151923954486847,-0.05089547112584114,-0.035404521971940994,-0.042780596762895584,0.06844499707221985,0.039815790951251984,-0.04816994443535805,-0.02414049208164215,0.0509430393576622,0.011589251458644867,0.022304899990558624,-0.07848770171403885,-0.061895884573459625,0.05458654463291168,-0.0720851942896843,-0.06474480777978897,-0.029861629009246826,0.07434319704771042,-0.01557699404656887,-0.0569937601685524,0.023950276896357536,0.06846567243337631,-0.04521917924284935,0.04664124548435211,-0.05920257419347763,-0.03756669908761978,0.04930774122476578,-0.06045081466436386,-0.04568680748343468,0.06075919419527054,0.051806606352329254,-0.07410642504692078,0.015194510109722614,0.04252289608120918,-0.021166639402508736,0.004609286319464445,-0.06615094840526581,-0.0023994832299649715,0.04825882986187935,0.07633655518293381,-0.05545933544635773,-0.046730510890483856,0.052121080458164215,0.032993387430906296,0.03616495430469513,0.04036552831530571,0.009766985662281513,-0.04517397657036781,0.05230315774679184,-0.06386615335941315,0.04333031550049782,0.057524193078279495,-0.02079842798411846,-0.01474597305059433,-0.030112585052847862,-0.06481742858886719,-0.0629182904958725,0.028299203142523766,-0.07313677668571472,-0.009072481654584408,0.08015219867229462,0.05580028519034386,0.04916791990399361,0.05297195166349411],fl=ITEM_UNIQUE,score,fq=[(ITEM_UNIQUE_SITE:10)
NOT (TAGT002_WSCIFL:1) AND (TAGT056_WPBFL:1), {!frange cache=false l=0.5
u=1.0}$q],sort=score
desc,rows=50,wt=javabin,version=2}},response={numFound=3,numFoundExact=true,start=0,maxScore=0.9319246,docs=[SolrDocument{ITEM_UNIQUE=3300146982,
score=0.9319246}, SolrDocument{ITEM_UNIQUE=3300148003, score=0.92536414},
SolrDocument{ITEM_UNIQUE=3000133742, score=0.92066574}]}}
========================================================================================
I have reviewed the official documentation and experimented with various query
configurations, but I have not been able to achieve the desired outcome. Could
you please provide guidance or examples on how to implement both category
filtering and score thresholding in a single vector search query?
I would like to provide an update regarding the issue I mentioned earlier.
After increasing the value of TopK, I was able to achieve the expected results.
This leads me to believe that the approximate calculation to retrieve the TopK
results is performed before the filtering conditions specified in the fq
parameter and the score threshold are applied.
Could you please confirm if this is the intended behavior of Solr? Is this how
Solr is designed to operate? If this is the expected behavior and a misuse on
my part, would increasing the number of shards or adjusting the TopK value be
necessary to achieve the desired results?
Any help or suggestions you could offer would be greatly appreciated. Thank you
very much for your time and assistance.
Best regards,
Tatsuya Hirose
Engineer
PERSOL CAREER CO., LTD.
tatsuya.hiro...@persol.co.jp<mailto:tatsuya.hiro...@persol.co.jp>
