Re: Join and Sort between 2 collections in Solr 8.9.0 cloud mode
As a community how should we behave with gentlemen that didn't pay their suppliers for their services and then come here? As an Apache Lucene/Solr committer I very commonly use the funds I receive from my clients (for my services) to sponsor my and my colleague's contributions. This company has a long overdue debit with my company and stop responding. At the same time is looking for free advise on the mailing list for the same project... I find this very unrespectful... On Tue, 8 Aug 2023, 08:22 Vishal Patel, wrote: > We are using Solr 8.9.0. We have configured Solr cloud like 2 shards and > each shard has one replica. We have used 5 zoo keepers for Solr cloud. > We have created 2 collections 1) forms 2) actionscomments > Schema field of forms > required="true" multiValued="false" docValues="true"/> > docValues="true"/> > omitNorms="true" omitTermFreqAndPositions="true"/> > docValues="true"/> > other fields ... > > Schema field of actionscomments > required="true" multiValued="false" docValues="true"/> > docValues="true"/> > docValues="true"/> > > docValues="true"/> > omitNorms="true" omitTermFreqAndPositions="true"/> > > other fields ... > > Here one to many relationship between forms and actionscomments > collections. > There are 30 million records in forms collection and 700 million records > in actionscomments. We are showing columns > form_title,form_creation_date,action_name in form listing. We have given > sorting in all columns. > If any user set sort on action_name then we are getting records as below > - First get all form_ids of that user from actionscomments collection with > sorting of action_name. > - Appending above form ids with boosting in query to get the forms data. > > Please note that we set score to 1 for all documents while indexing hence > we used boost query. > > Example: > > actionscomments: > Query: user:(713073) AND entity_type:(1) AND project_id:(2076645) sort > field=action_name > > Query: ((doctype:Apps AND appType:1 AND is_formtype_active:true AND > (is_draft:false OR msg_type_id:1) AND instance_group_id:(2313622) AND > project_id:(2076645))) AND msg_id:(10130987^2 10139990^1 *)^20 > > It is taking more time to return the result. > > Is there any approach where we can get join and sort between two > collection in cloud mode? >
RE: [EXTERNAL] Re: what is SolrAuthV2 and why does it break replication
Yes, and indeed http://[REDACTED]/solr/sequence2_shard1_replica_n1/replication works without password. I have also tried http://[REDACTED]/solr/sequence2/replication which I remember working when I tried it in Solr8 but which does not work in Solr9.2 Have you any other recommendations? -Original Message- From: Jan Høydahl Sent: Saturday, August 5, 2023 11:03 AM To: users@solr.apache.org Subject: [EXTERNAL] Re: what is SolrAuthV2 and why does it break replication > nor does it work to set up security.json to allow replication without a > password Exactly how did you configure security.json for this? It could be that you succeeded disabling security for the collection but not for the core URL? Have you tried to hit http://[REDACTED]/solr/sequence2_shard1_replica_n1/replication URL directly using cURL without basicAuth? Jan > 27. jul. 2023 kl. 18:17 skrev Oakley, Craig (NIH/NLM/NCBI) [C] > : > > I have still not received any suggestions or clarifications, so I am > resending this with a different subject. > > I found that if I completely eliminate security.json, Leader/Follower > replication succeeds; but for obvious reasons, we do want security.json to be > there. > > Setting -Dsolr.pki.sendVersion=v1 -Dsolr.pki.acceptVersions=v1,v2 does not > help; nor does it work to set up security.json to allow replication without a > password and to remove httpBasicAuthUser and httpBasicAuthPassword from > solrconfig.xml on the Follower side > > Does anybody have any suggestions? > > -Original Message- > From: Oakley, Craig (NIH/NLM/NCBI) [C] > Sent: Tuesday, July 18, 2023 3:12 PM > To: users@solr.apache.org > Subject: RE: authentication for Leader/Follower replication > > I am wondering whether anyone yet has any suggestions how to proceed > > -Original Message- > From: Oakley, Craig (NIH/NLM/NCBI) [C] > Sent: Thursday, July 6, 2023 4:00 PM > To: users@solr.apache.org > Subject: authentication for Leader/Follower replication > > We are having problems transitioning Leader/Follower replication to Solr9.2.1 > > In Solr8.5 and below, what was then called Master/Slave replication had the > annoying problem that, even though we specified httpBasicAuthUser and > httpBasicAuthPassword, it would always attempt to connect first without a > password before retrying with a password. This made solr.log noisy with lots > of unnecessary login failures: but at least it worked. > > When we transitioned to Solr8.11 (with the nomenclature changed to be less > oppressive) we found that this version of Leader/Follower replication refused > to retry (and refused to do anything with the values specified > httpBasicAuthUser and httpBasicAuthPassword). We needed to open up > replication in security.json to be available without password. > > Now when we are preparing to upgrade to Solr9.2.1, we are having issues with > the following: > 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ] > o.a.s.h.IndexFetcher Last replication failed, so I'll force replication > 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ] > o.a.s.h.IndexFetcher Leader at: > http://[REDACTED]/solr/sequence2_shard1_replica_n1 is not available. Index > fetch failed by exception: > org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: > Error from server at http://[REDACTED]/solr/sequence2_shard1_replica_n1: > Expected mime type in [application/octet-stream, > application/vnd.apache.solr.javabin] but got text/html. > > > Error 401 Could not load principal from SolrAuthV2 header. > > HTTP ERROR 401 Could not load principal from SolrAuthV2 header. > > URI:/solr/sequence2_shard1_replica_n1/replication > STATUS:401 > MESSAGE:Could not load principal from SolrAuthV2 > header. > SERVLET:default > > > > > > I have added "blockUnknown":false to security.json and have confirmed that > the replication?command=indexversion command can be run without a password, > and that it can be run with the login and password specified in > httpBasicAuthUser and httpBasicAuthPassword > > I have tried tweaking security.json with forwardCredentials values, but that > has not helped > > Any suggestions? > > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
Cannot invoke "org.apache.solr.update.TransactionLog.incref()"
I am seeing this entry in our Solr logs: *Cannot invoke "org.apache.solr.update.TransactionLog.incref()" because "lookupLog" is null* We are running Solr 8.4. (not in Solr cloud mode) full stack trace: org.apache.solr.handler.RequestHandlerBase message java.lang.NullPointerException: Cannot invoke "org.apache.solr.update.TransactionLog.incref()" because "lookupLog" is null at org.apache.solr.update.UpdateLog.lookup(UpdateLog.java:1015) at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocumentFromTlog(RealTimeGetComponent.java:555) at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:628) at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:610) at org.apache.solr.update.processor.DistributedUpdateProcessor.getUpdatedDocument(DistributedUpdateProcessor.java:675) at org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:374) at org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:339) at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:339) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:225) at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:332) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:281) at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338) at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:236) at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:303) at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283) at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:196) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:127) at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:122) at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConn
Re: knn parser not working as expected
Thanks! That was it!
The config defType was taking the precedence. After removing that, it
started working fine.
Shouldn't the parser mentioned in the query take precedence over what is
mentioned in the config file?
And just one related question, the other config inside this request handler
does not matter for dense vector search right? like qf, pf, tie etc shown
below?
I might as well just use an empty request handler for the dense vector
search.
true explicit 0.01 body^0.5 title^1.2 body^0.5 title^1.2 site^1.5 100 AND
On Mon, Aug 7, 2023 at 8:35 PM Joel Bernstein wrote:
> Oh wait, I just saw your post above with the config. I suspect it's the
> hardcoded defType that is causing this problem. Try testing with this
> removed to confirm.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Mon, Aug 7, 2023 at 8:32 PM Joel Bernstein wrote:
>
> > So, the knn query is being parsed into an edismax query in the
> > attached debug output. So something is definitely not working as
> expected.
> > In your solrconfig.xml do you have a hardcoded qparser for the request
> > handler?
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Fri, Aug 4, 2023 at 12:09 PM gnandre wrote:
> >
> >> Also, here is the debug output for that workaround with fq I mentioned.
> >> This debug output is not big.
> >>
> >> "debug":{
> >> "rawquerystring":"*:*",
> >> "querystring":"*:*",
> >> "parsedquery":"+(+MatchAllDocsQuery(*:*)) ()",
> >> "parsedquery_toString":"+(+*:*) ()",
> >> "json":{"params":{
> >> "q":"*:*",
> >> "fq":"{!knn f=dense_vector
> >>
> >>
> topK=1}[0.06525743007659912,0.015727980062365532,0.003069591475650668,-0.016254400834441185,0.003478930564597249,-0.02475954219698906,0.020238326862454414,0.010255611501634121,0.05522076040506363,0.020635411143302917,0.05825875699520111,-0.05110647529363632,-0.04696913808584213,0.05991407483816147,-0.0003015052934642881,0.03625837340950966,-0.044656239449977875,-0.06582673639059067,-0.06842341274023056,-0.022927379235625267,0.048230838030576706,-0.12659960985183716,-0.019311215728521347,-0.04432906210422516,0.03600681200623512,0.010301047936081886,0.08415472507476807,0.04727723449468613,-0.0584205724298954,-0.045265913009643555,0.012285877950489521,0.0034233061596751213,-0.00982636958360672,-0.013216182589530945,-0.038882751017808914,-0.05872005969285965,-0.029350444674491882,0.04930287227034569,0.0022274062503129244,0.01728842593729496,-0.08762819767,-0.045831114053726196,0.072530098259449,0.03804686293005943,0.0021682181395590305,-0.05424166098237038,-0.004494055639952421,0.05843663960695267,0.058729417622089386,0.016252348199486732,0.0019551776349544525,-0.012190568260848522,-0.08235936611890793,-0.003848800901323557,0.028969185426831245,0.047798849642276764,-0.04074695333838463,-0.10175333172082901,0.06699151545763016,-0.06788542866706848,-0.01607389748096466,0.07294511049985886,0.007754810154438019,0.039606861770153046,0.07451225817203522,-0.02967959391212,0.014015864580869675,0.08055979013442993,0.0010412412229925394,0.13284511864185333,-0.013288799673318863,-0.05446619912981987,-0.03510258346796036,-0.12459734082221985,-0.017629574984312057,-0.04287091642618179,-0.019087448716163635,0.027409998700022697,-0.040427371859550476,-0.1713477075099945,-0.0035959691740572453,0.01750982739031315,-0.06452985852956772,0.10622204840183258,-0.06865541636943817,0.06022517383098602,0.03378240391612053,0.02320132404565811,0.02072194404900074,0.03390982002019882,0.0051648980006575584,0.05843415856361389,-0.07012602686882019,0.046549294143915176,0.005304296966642141,0.09183698892593384,0.060101959854364395,-0.031673040241003036,0.03126641735434532,0.10213921219110489,0.07624002546072006,-0.09995660930871964,0.03316718339920044,-0.040208760648965836,-0.016963355243206024,-0.01603076048195362,-0.00566966412588954,0.0570228286087513,0.006566803902387619,0.028397461399435997,-0.03737075999379158,-0.03357473015785217,-0.05060608312487602,0.0882791057229042,0.14182551205158234,0.01651209406554699,0.047577112913131714,-0.028357332572340965,-0.12397051602602005,0.03264006972312927,0.030581200495362282,0.025287700816988945,-0.08509892970323563,0.032361947267,-0.06732083112001419,0.0193667970597744,0.07096285372972488,-5.732041797079612e-33,0.033934514969587326,0.029480531811714172,-0.024119360372424126,0.03248802572488785,0.060654137283563614,-0.04089922457933426,-0.06845896691083908,0.015865417197346687,-0.03816983848810196,0.12768638134002686,-0.047979939728975296,0.01888129487633705,0.01966758444905281,-0.021792754530906677,-0.00209379056468606,-0.060791824012994766,0.07595516741275787,-0.05137578397989273,-0.020345840603113174,0.02730456180870533,-0.08421282470226288,0.0052170781418681145,-0.0396740548312664,0.013655638322234154,0.043763574212789536,0.0368662029504776,-0.021710995584726334,0.03603581339120865,0.04991370812058449,-0.007524373475462198,0.033250145614147186,0.0669487863779068,-0.01280767004936
