Re: Query on Solr and ZK ports
Hi, See https://unix.stackexchange.com/questions/684348/solr-ports-in-use for an answer. The STOP_PORT is the Jetty Servlet container's way to shutting down. It will only listen on localhost, so you need to use bin/solr stop command on the host, i.e. no need to expose this port to the outside. Wrt ZK ports, Only port 2181 needs to be open between Solr and ZK, but between the Zookeepers, also ports 2888 and 3888 needs to be open for internal communication, see https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.0/bk_reference/content/zookeeper-ports.html I'm not aware of other ports. Jan > 18. apr. 2023 kl. 05:54 skrev HariBabu kuruva : > > Can someone please help with this information? > > On Tue, Apr 4, 2023 at 10:01 PM HariBabu kuruva > wrote: > >> Hi All, >> >> I could see the solr process is Listening on 7981 port along with the >> normal solr port(8981). It is shown as DSTOP PORT , when I grep solr >> process. Could you please give more details on this port, Can we disable >> this ? >> >> With regards to Zookeeper I could see port 8080 as a ZK admin port, How >> can I use this, can i disable it if I don't want it ? >> Also I could see ZK is listening on some random port (43801) along with >> the other ports. Please throw some light on this. >> >> -- >> >> Thanks and Regards, >> Hari >> Mobile:9790756568 >> > > > -- > > Thanks and Regards, > Hari > Mobile:9790756568
SolrCoreState already closed
Hi All Getting loads of "SolrCoreState already closed" warnings... These look to be coming from the DIH... Could it be the DIH is running two processes simultaneously? Full warning below... Cheers, Paul 2023-04-18 15:21:04.520 WARN (Thread-34929) [ ] o.a.s.h.d.SolrWriter Error creating document : SolrInputDocument(fields: [_indexname=fred_index, ) => org.apache.solr.common.SolrException: SolrCoreState already closed. at org.apache.solr.update.DefaultSolrCoreState.lock(DefaultSolrCoreState.java:188) org.apache.solr.common.SolrException: SolrCoreState already closed. at org.apache.solr.update.DefaultSolrCoreState.lock(DefaultSolrCoreState.java:188) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:123) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:338) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:289) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:236) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:257) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:483) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:337) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:337) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:223) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) ~[solr-core-8.1.1.jar:8.1.1 fcbe46c28cef11bc058779afba09521de1b19bef - ab - 2019-05-22 15:20:01] at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:80) ~[?:?] at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:258) ~[?:?] at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:527) ~[?:?] at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415) ~[?:?] at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330) ~[?:?] at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233) ~[?:?] at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424) ~[?:?] at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) ~[?:?] at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466) ~[?:?] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_361]
Re: SolrJ 9.2 and Java version
The min neccessary java version use to be specified in the README, i'm not sure when/why it was removed. You can now find it in the "System Requirements" page of the ref-guide... https://solr.apache.org/guide/solr/latest/deployment-guide/system-requirements.html : Date: Thu, 13 Apr 2023 10:49:20 +0200 : From: Dominique Bejean : Reply-To: users@solr.apache.org : To: solr-user : Subject: SolrJ 9.2 and Java version : : Hi, : : I don't find in documentation which minimal version is required for SolrJ : 9.2. JAva 11 and 17 are supported for sure, but is Java 1.8 still possible ? : : Regards : : Dominique : -Hoss http://www.lucidworks.com/
Re: Suggestions to improve Star queries latencies
Hi Mikhail, Yes, 9.1.1, that should be helpful, can you please point me to the related jira(s) and/or docs? Thank you, Rajani On Mon, Apr 17, 2023 at 2:09 AM Mikhail Khludnev wrote: > Hello Rajani. > Which version are you running? IIRC 9.1.2 has some > improvement about caching short queries. > > On Sun, Apr 16, 2023 at 4:25 PM Rajani Maski > wrote: > > > Hi Solr Users, > > > > What are your suggestions to improve star queries latencies? By star > > queries I mean "*:*" or single term queries having boost formulas (such > as > > doc recency and many others) taking 10 or more seconds. It is a large > > collection with good compute resources, however I am guessing this may be > > because each shard has too many documents and I noticed per shard > response > > time also is high. > > > > Splitting shards could be an option however it is already an > > evenly distributed, composite router, 96 shards collection, I am > > concerned that more than 100 shards per collection can lead to > exhaustively > > searching too many shards and aggregation issues. What are your thoughts? > > > > Can we make use of any caches, query result cache or other caches, in > solr > > that allows warming up and persisting these queries results in ram, and > > that maybe helps reduce this query time? > > > > Thanks, > > Rajani > > > > > -- > Sincerely yours > Mikhail Khludnev > https://t.me/MUST_SEARCH > A caveat: Cyrillic! >
Re: Suggestions to improve Star queries latencies
I think there are more important questions here. What do you want with a *:* query? Do you want all the results in on return? Or do you just want the count of total documents? Or to put the results in facets? *:* should never take long unless you are requesting every single document not just the first ten. > On Apr 18, 2023, at 9:05 PM, Rajani Maski wrote: > > Hi Mikhail, > > Yes, 9.1.1, that should be helpful, can you please point me to the > related jira(s) and/or docs? > > Thank you, > Rajani > > > >> On Mon, Apr 17, 2023 at 2:09 AM Mikhail Khludnev wrote: >> >> Hello Rajani. >> Which version are you running? IIRC 9.1.2 has some >> improvement about caching short queries. >> >> On Sun, Apr 16, 2023 at 4:25 PM Rajani Maski >> wrote: >> >>> Hi Solr Users, >>> >>> What are your suggestions to improve star queries latencies? By star >>> queries I mean "*:*" or single term queries having boost formulas (such >> as >>> doc recency and many others) taking 10 or more seconds. It is a large >>> collection with good compute resources, however I am guessing this may be >>> because each shard has too many documents and I noticed per shard >> response >>> time also is high. >>> >>> Splitting shards could be an option however it is already an >>> evenly distributed, composite router, 96 shards collection, I am >>> concerned that more than 100 shards per collection can lead to >> exhaustively >>> searching too many shards and aggregation issues. What are your thoughts? >>> >>> Can we make use of any caches, query result cache or other caches, in >> solr >>> that allows warming up and persisting these queries results in ram, and >>> that maybe helps reduce this query time? >>> >>> Thanks, >>> Rajani >>> >> >> >> -- >> Sincerely yours >> Mikhail Khludnev >> https://t.me/MUST_SEARCH >> A caveat: Cyrillic! >>
Re: Suggestions to improve Star queries latencies
It is a query with popularity and recency boosts, requesting the first 100 docs with 3 fields per doc. No facets. It is a query against a collection of 96 shards ~7m docs per shard. Could the cause for latency be boost queries and would it also be time spent in aggregating results from many shards? Curious to learn more about caching such/short queries that @Mikhail mentioned. On Tue, Apr 18, 2023 at 9:44 PM Dave wrote: > I think there are more important questions here. What do you want with a > *:* query? Do you want all the results in on return? Or do you just want > the count of total documents? Or to put the results in facets? *:* should > never take long unless you are requesting every single document not just > the first ten. > > > On Apr 18, 2023, at 9:05 PM, Rajani Maski wrote: > > > > Hi Mikhail, > > > > Yes, 9.1.1, that should be helpful, can you please point me to the > > related jira(s) and/or docs? > > > > Thank you, > > Rajani > > > > > > > >> On Mon, Apr 17, 2023 at 2:09 AM Mikhail Khludnev > wrote: > >> > >> Hello Rajani. > >> Which version are you running? IIRC 9.1.2 has some > >> improvement about caching short queries. > >> > >> On Sun, Apr 16, 2023 at 4:25 PM Rajani Maski > >> wrote: > >> > >>> Hi Solr Users, > >>> > >>> What are your suggestions to improve star queries latencies? By star > >>> queries I mean "*:*" or single term queries having boost formulas > (such > >> as > >>> doc recency and many others) taking 10 or more seconds. It is a large > >>> collection with good compute resources, however I am guessing this may > be > >>> because each shard has too many documents and I noticed per shard > >> response > >>> time also is high. > >>> > >>> Splitting shards could be an option however it is already an > >>> evenly distributed, composite router, 96 shards collection, I am > >>> concerned that more than 100 shards per collection can lead to > >> exhaustively > >>> searching too many shards and aggregation issues. What are your > thoughts? > >>> > >>> Can we make use of any caches, query result cache or other caches, in > >> solr > >>> that allows warming up and persisting these queries results in ram, and > >>> that maybe helps reduce this query time? > >>> > >>> Thanks, > >>> Rajani > >>> > >> > >> > >> -- > >> Sincerely yours > >> Mikhail Khludnev > >> https://t.me/MUST_SEARCH > >> A caveat: Cyrillic! > >> >
