Re: JSON boolean query syntax with edismax as default QueryParser
Hi Jane,
That change in 7.2 does look like it's correlated, and I'm the one who
implemented it in the name of security.
Could you walk me through a short series of steps to show the problem with
one of Solr's "example" setups like techproducts? Step one is run it, step
two is you sending the query via curl. Use whatever Solr version you want.
~ David
On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg wrote:
> Hi Solr colleagues,
>
> On Solr 8.4.1, we’ve noticed that the following types of JSON DSL queries
> work if our luceneMatchVersion is 7.1 or lower, or if our default query
> parser is set to lucene:
>
>
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
>
> However, if the query parser is set to edismax and the luceneMatchVersion
> is 7.2 or higher, the parsed query visible with debug=true becomes a
> complete mess, searching for the terms “bool” and “must”, rather than the
> terms we actually want to search for:
>
>
> +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> (local_subject_unstem_search:bool)^15.0 | (author_unstem_search:bool)^40.0
> […]
>
> Also while debug=true, we noticed that the JSON DSL queries get converted
> into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> suspecting these two changes in Solr 7.2 as the reason we can’t use Boolean
> JSON queries with edismax and a recent luceneMatchVersion:
> https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes.
> Does that seem correct?
>
> Also, could this be related to the question Benjamin Armintor asked on
> June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> specifically curious about whether a luceneMatchVersion of 7.1 or lower
> still works in Solr 9?
>
> Thanks for your insights,
>
> -Jane
>
> --
> Jane Sandberg (she/her)
> Library Software Engineer, Discovery and Access Services
>
Re: JSON boolean query syntax with edismax as default QueryParser
Hi David,
Thanks for looking into this, and for the security fix.
My colleague and I put together a small repository to reproduce the issue. It
has a configset, a docker-compose file, and a README with the steps to
reproduce it on solr 8.4: https://github.com/pulibrary/edismax-json-queries/
Appreciatively,
-Jane
From: David Smiley
Date: Monday, July 31, 2023 at 5:58 AM
To: users@solr.apache.org
Subject: Re: JSON boolean query syntax with edismax as default QueryParser
Hi Jane,
That change in 7.2 does look like it's correlated, and I'm the one who
implemented it in the name of security.
Could you walk me through a short series of steps to show the problem with
one of Solr's "example" setups like techproducts? Step one is run it, step
two is you sending the query via curl. Use whatever Solr version you want.
~ David
On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg wrote:
> Hi Solr colleagues,
>
> On Solr 8.4.1, we’ve noticed that the following types of JSON DSL queries
> work if our luceneMatchVersion is 7.1 or lower, or if our default query
> parser is set to lucene:
>
>
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
>
> However, if the query parser is set to edismax and the luceneMatchVersion
> is 7.2 or higher, the parsed query visible with debug=true becomes a
> complete mess, searching for the terms “bool” and “must”, rather than the
> terms we actually want to search for:
>
>
> +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> (local_subject_unstem_search:bool)^15.0 | (author_unstem_search:bool)^40.0
> […]
>
> Also while debug=true, we noticed that the JSON DSL queries get converted
> into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> suspecting these two changes in Solr 7.2 as the reason we can’t use Boolean
> JSON queries with edismax and a recent luceneMatchVersion:
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7Ce4c749a12c0e49600eaf08db91c5dec0%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C63826405126971%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QhLXzlXcEnAp86czCzwUUHeio89aiAiAA46ZTkDrWDo%3D&reserved=0.
> Does that seem correct?
>
> Also, could this be related to the question Benjamin Armintor asked on
> June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> specifically curious about whether a luceneMatchVersion of 7.1 or lower
> still works in Solr 9?
>
> Thanks for your insights,
>
> -Jane
>
> --
> Jane Sandberg (she/her)
> Library Software Engineer, Discovery and Access Services
>
Re: knn parser not working as expected
Any pointers where should I look to resolve this issue? Thanks!
On Thu, Jul 27, 2023 at 9:25 PM gnandre wrote:
> But the q parameter is still not working. I am stumped.
>
> On Fri, Jul 21, 2023 at 1:43 AM gnandre wrote:
>
>> Thanks. If I move the knn parser syntax and value to fq param and make q
>> as *:*, it works and starts giving relevant results instantly.
>>
>> curl -X POST -H "Content-Type: application/json" -d '{
>> "params": {
>> "q": "*:*", "fq": "{!knn f=dense_vector
>> topK=1}[0.06525743007659912,0.015727980062365532,0.003069591475650668,-0.016254400834441185,0.003478930564597249,-0.02475954219698906,0.020238326862454414,0.010255611501634121,0.05522076040506363,0.020635411143302917,0.05825875699520111,-0.05110647529363632,-0.04696913808584213,0.05991407483816147,-0.0003015052934642881,0.03625837340950966,-0.044656239449977875,-0.06582673639059067,-0.06842341274023056,-0.022927379235625267,0.048230838030576706,-0.12659960985183716,-0.019311215728521347,-0.04432906210422516,0.03600681200623512,0.010301047936081886,0.08415472507476807,0.04727723449468613,-0.0584205724298954,-0.045265913009643555,0.012285877950489521,0.0034233061596751213,-0.00982636958360672,-0.013216182589530945,-0.038882751017808914,-0.05872005969285965,-0.029350444674491882,0.04930287227034569,0.0022274062503129244,0.01728842593729496,-0.08762819767,-0.045831114053726196,0.072530098259449,0.03804686293005943,0.0021682181395590305,-0.05424166098237038,-0.004494055639952421,0.05843663960695267,0.058729417622089386,0.016252348199486732,0.0019551776349544525,-0.012190568260848522,-0.08235936611890793,-0.003848800901323557,0.028969185426831245,0.047798849642276764,-0.04074695333838463,-0.10175333172082901,0.06699151545763016,-0.06788542866706848,-0.01607389748096466,0.07294511049985886,0.007754810154438019,0.039606861770153046,0.07451225817203522,-0.02967959391212,0.014015864580869675,0.08055979013442993,0.0010412412229925394,0.13284511864185333,-0.013288799673318863,-0.05446619912981987,-0.03510258346796036,-0.12459734082221985,-0.017629574984312057,-0.04287091642618179,-0.019087448716163635,0.027409998700022697,-0.040427371859550476,-0.1713477075099945,-0.0035959691740572453,0.01750982739031315,-0.06452985852956772,0.10622204840183258,-0.06865541636943817,0.06022517383098602,0.03378240391612053,0.02320132404565811,0.02072194404900074,0.03390982002019882,0.0051648980006575584,0.05843415856361389,-0.07012602686882019,0.046549294143915176,0.005304296966642141,0.09183698892593384,0.060101959854364395,-0.031673040241003036,0.03126641735434532,0.10213921219110489,0.07624002546072006,-0.09995660930871964,0.03316718339920044,-0.040208760648965836,-0.016963355243206024,-0.01603076048195362,-0.00566966412588954,0.0570228286087513,0.006566803902387619,0.028397461399435997,-0.03737075999379158,-0.03357473015785217,-0.05060608312487602,0.0882791057229042,0.14182551205158234,0.01651209406554699,0.047577112913131714,-0.028357332572340965,-0.12397051602602005,0.03264006972312927,0.030581200495362282,0.025287700816988945,-0.08509892970323563,0.032361947267,-0.06732083112001419,0.0193667970597744,0.07096285372972488,-5.732041797079612e-33,0.033934514969587326,0.029480531811714172,-0.024119360372424126,0.03248802572488785,0.060654137283563614,-0.04089922457933426,-0.06845896691083908,0.015865417197346687,-0.03816983848810196,0.12768638134002686,-0.047979939728975296,0.01888129487633705,0.01966758444905281,-0.021792754530906677,-0.00209379056468606,-0.060791824012994766,0.07595516741275787,-0.05137578397989273,-0.020345840603113174,0.02730456180870533,-0.08421282470226288,0.0052170781418681145,-0.0396740548312664,0.013655638322234154,0.043763574212789536,0.0368662029504776,-0.021710995584726334,0.03603581339120865,0.04991370812058449,-0.007524373475462198,0.033250145614147186,0.0669487863779068,-0.012807670049369335,-0.08904062211513519,-0.04803512617945671,-0.0461772084236145,0.018098553642630577,0.01096352282911539,0.0617918036878109,0.014066621661186218,-0.03305654972791672,-0.08129353821277618,-0.025270603597164154,0.03537251427769661,0.06029881164431572,0.06169535592198372,0.0355769582092762,0.03534447401762009,-0.047377053648233414,0.053076375275850296,-0.019250469282269478,-0.03837420791387558,-0.00834209006279707,0.031550273299217224,0.004682184662669897,0.0590718574821949,0.0326957181096077,-0.041941817849874496,-0.04179370403289795,-0.010403091087937355,0.11914990842342377,-0.049126915633678436,0.015761952847242355,-0.012162514962255955,-0.05942496284842491,0.04794146493077278,-0.06834675371646881,-0.03294386342167854,0.02242257259786129,0.0774146020412445,-0.1095564718246,0.023828692734241486,0.054935190826654434,0.0202674251049757,-0.057155776768922806,-0.009578827768564224,-0.051850661635398865,0.09117215871810913,-0.07315851002931595,-0.0019339871359989047,-0.05835318937897682,-0.058747921139001846,-0.05519327148795128,-0.014699703082442284,-0.0020833320450037718,-0.05721793323755264,0.055632084608078,0.0064485
Re: JSON boolean query syntax with edismax as default QueryParser
Why did you set defType=edismax in the /query handler in your
solrconfig.xml?
edismax is for handling user queries -- text directly from a text box.
JSON Query DSL obviously isn't that.
~ David
On Mon, Jul 31, 2023 at 3:09 PM Jane Sandberg wrote:
> Hi David,
>
> Thanks for looking into this, and for the security fix.
>
> My colleague and I put together a small repository to reproduce the
> issue. It has a configset, a docker-compose file, and a README with the
> steps to reproduce it on solr 8.4:
> https://github.com/pulibrary/edismax-json-queries/
>
> Appreciatively,
>
> -Jane
>
> From: David Smiley
> Date: Monday, July 31, 2023 at 5:58 AM
> To: users@solr.apache.org
> Subject: Re: JSON boolean query syntax with edismax as default QueryParser
> Hi Jane,
>
> That change in 7.2 does look like it's correlated, and I'm the one who
> implemented it in the name of security.
>
> Could you walk me through a short series of steps to show the problem with
> one of Solr's "example" setups like techproducts? Step one is run it, step
> two is you sending the query via curl. Use whatever Solr version you want.
>
> ~ David
>
>
> On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg
> wrote:
>
> > Hi Solr colleagues,
> >
> > On Solr 8.4.1, we’ve noticed that the following types of JSON DSL queries
> > work if our luceneMatchVersion is 7.1 or lower, or if our default query
> > parser is set to lucene:
> >
> >
> >
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
> >
> > However, if the query parser is set to edismax and the luceneMatchVersion
> > is 7.2 or higher, the parsed query visible with debug=true becomes a
> > complete mess, searching for the terms “bool” and “must”, rather than the
> > terms we actually want to search for:
> >
> >
> > +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> > (local_subject_unstem_search:bool)^15.0 |
> (author_unstem_search:bool)^40.0
> > […]
> >
> > Also while debug=true, we noticed that the JSON DSL queries get converted
> > into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> > suspecting these two changes in Solr 7.2 as the reason we can’t use
> Boolean
> > JSON queries with edismax and a recent luceneMatchVersion:
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7Ce4c749a12c0e49600eaf08db91c5dec0%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C63826405126971%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QhLXzlXcEnAp86czCzwUUHeio89aiAiAA46ZTkDrWDo%3D&reserved=0
> <
> https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes
> >.
> > Does that seem correct?
> >
> > Also, could this be related to the question Benjamin Armintor asked on
> > June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> > specifically curious about whether a luceneMatchVersion of 7.1 or lower
> > still works in Solr 9?
> >
> > Thanks for your insights,
> >
> > -Jane
> >
> > --
> > Jane Sandberg (she/her)
> > Library Software Engineer, Discovery and Access Services
> >
>
Re: Add a new Shard to the collection
++ FYI, I can see the old shard automatically removed. On Mon, Jul 31, 2023 at 11:39 AM HariBabu kuruva wrote: > Thanks for your reply. > > I am a little bit worried about PROD. Can I go ahead and do the same steps > in PROD ? Do I need to take any backups or any steps before doing this? > > On Sat, Jul 29, 2023 at 8:51 AM Mikhail Khludnev wrote: > >> Hello Hari. >> If new shards are handling queries and updates well it's ok to have old >> shard inactive. >> You can request DELETESHARD to reclaim the disk space. >> >> On Mon, Jul 24, 2023 at 6:19 PM HariBabu kuruva < >> hari2708.kur...@gmail.com> >> wrote: >> >> > Hi All, >> > >> > I would like to add a new shard to the existing collection to have >> better >> > performance. Currently we have one shard. >> > >> > Solr - 8.11.1 >> > Nodes(servers) - 10 (Non prod - 4 nodes) >> > Zookeepers-5 >> > >> > I have tried the SPLITSHARD command in one of the non prod environments. >> > >> > * >> > >> https://solrserver.corp.company.com:8981/solr/admin/collections?action=SPLITSHARD&collection=abcStore&shard=shard1 >> > < >> > >> https://solrserver.corp.company.com:8981/solr/admin/collections?action=SPLITSHARD&collection=abcStore&shard=shard1 >> > >* >> > Now i can see total 3 shards >> > Shard1 >> > Shard1_0 >> > Shard1_1 >> > >> > But Shard1 is shown as inactive. Please let me know if we need to remove >> > this ? >> > >> > Please help me if this is the correct way of splitting the shard. >> > Are there any impacts to the data because of this ? >> > What are the measures to be taken while doing this in a PROD >> environment. >> > >> > -- >> > >> > Thanks and Regards, >> > Hari >> > Mobile:9790756568 >> > >> >> >> -- >> Sincerely yours >> Mikhail Khludnev >> > > > -- > > Thanks and Regards, > Hari > Mobile:9790756568 > -- Thanks and Regards, Hari Mobile:9790756568
