Re: Soft commit takes 5 seconds in Solr 8.9.0
Now for the 20 questions: What OS do you have this running on? --> centos How much physical memory is in the machine? --> 370 GB Is it running anything other than Solr? --> Yes, we have 3 Wildfly applications (each has 50GB memory) and 2 microapps (each has 20GB memory). What is the Solr heap size? --> 100 GB memory assign to solr Are you running more than one Solr instance on a single machine? --> Yes, we are using two instances of Solr in one machine. We have 2 shard 2 replicas so one shard and the other shard replica are running on this server. How big are all the index cores on a single machine? --> 700 GB(solr instance one 303GB & second instance 379 GB) Looking for Max Doc, Num Docs, and on-disk size Max Doc --> 5913435 Num Docs --> 4330880 on-disk size --> 84.32 GB On Sat, Oct 7, 2023 at 9:22 PM Shawn Heisey wrote: > On 10/5/23 23:02, John Jackson wrote: > > Hello, > > > > Kindly guide/help us on this. I already shared the requested files. > > There's nothing that jumps out at me in the config. > > You should configure caches ... it won't help with commit time but it is > likely to speed up queries unless every query your Solr receives is > entirely unique. Be careful that you don't set autowarmCount too high > ... that slows down commits. On a big install I managed, I set > autowarmCount on the filterCache to 4 ... and commits could still > sometimes take 15 seconds to complete. > > As I said before, 100 milliseconds is COMPLETELY unrealistic. I would > personally set autoSoftCommit maxTime to at LEAST 3. Jan's > recommendation is at least 1000 ... I think that's also unrealistic. > > Depending on a bunch of things including how big your index is, 5 > seconds may not be all that bad for a commit time. > > Now for the 20 questions: > > What OS do you have this running on? How much physical memory is in the > machine? Is it running anything other than Solr? What is the Solr heap > size? Are you running more than one Solr instance on a single machine? > How big are all the index cores on a single machine? Looking for Max > Doc, Num Docs, and on-disk size ... all of these are found on the core > overview in the admin UI. > > There may be more questions depending on what the answers to those > questions reveal. > > I am working my way towards getting a screenshot of the process list. > See this URL: > > > https://cwiki.apache.org/confluence/display/solr/solrperformanceproblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue > > Thanks, > Shawn > >
Solr 9.2.1: Creation of Core "Hangs"
Hello,
We experienced a problem when creating a core via a request on Solr 9.2.1. It
seems that the core creation for “blueprint_helios_comments“ just “hangs”.
Consequently, with subsequent external requests to create the same core (since
it does not seem to be there yet), Solr detects that the core creation is
already in progress and returns an error.
This is what Solr’s log tells me:
[ blueprint_helios_comments] o.a.s.s.HttpSolrCall [admin] webapp=null
path=/admin/cores
params={core=blueprint_helios_comments&action=STATUS&indexInfo=false&wt=javabin&version=2}
status=0 QTime=1
[ blueprint_helios_comments] o.a.s.h.a.CoreAdminOperation core create command
name=blueprint_helios_comments&action=CREATE&configSet=elastic&wt=javabin&version=2&dataDir=data
[ blueprint_helios_comments] o.a.s.c.SolrConfig Using Lucene MatchVersion:
9.4.0
[ blueprint_helios_comments] o.a.s.s.IndexSchema Schema name=elastic
[ blueprint_helios_comments] o.a.s.s.IndexSchema Loaded schema elastic/1.6
with uniqueid field id
[ blueprint_helios_comments] o.a.s.c.CoreContainer Creating SolrCore
'blueprint_helios_comments' using configuration from configset
/opt/solr-9.2.1/server/solr/configsets/elastic, trusted=true
[ blueprint_helios_comments] o.a.s.c.CachingDirectoryFactory solr.data.home =
/var/solr/data
[ blueprint_helios_comments] o.a.s.c.SolrCore Opening new SolrCore at
[/var/solr/data/blueprint_helios_comments],
dataDir=[/var/solr/data/blueprint_helios_comments/data/]
[ blueprint_helios_comments] o.a.s.j.SolrRequestAuthorizer Creating a new
SolrRequestAuthorizer
[ blueprint_helios_comments] o.a.s.u.UpdateHandler Using UpdateLog
implementation: org.apache.solr.update.UpdateLog
[ blueprint_helios_comments] o.a.s.u.UpdateLog Initializing UpdateLog:
dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
numVersionBuckets=65536
[ blueprint_helios_comments] o.a.s.u.CommitTracker Hard AutoCommit: if
uncommitted for 15000ms;
[ blueprint_helios_comments] o.a.s.u.CommitTracker Soft AutoCommit: if
uncommitted for 3000ms;
The creation stops after “Soft AutoCommit”. During the same run of Solr and our
test suite, other cores are created successfully.
With the next run of the same software on empty directories and attempt to
create the core, everything succeeds:
[ blueprint_helios_comments] o.a.s.s.HttpSolrCall [admin] webapp=null
path=/admin/cores
params={core=blueprint_helios_comments&action=STATUS&indexInfo=false&wt=javabin&version=2}
status=0 QTime=1
[ blueprint_helios_comments] o.a.s.h.a.CoreAdminOperation core create command
name=blueprint_helios_comments&action=CREATE&configSet=elastic&wt=javabin&version=2&dataDir=data
[ blueprint_helios_comments] o.a.s.c.SolrConfig Using Lucene MatchVersion:
9.4.0
[ blueprint_helios_comments] o.a.s.s.IndexSchema Schema name=elastic
[ blueprint_helios_comments] o.a.s.s.IndexSchema Loaded schema elastic/1.6
with uniqueid field id
[ blueprint_helios_comments] o.a.s.c.CoreContainer Creating SolrCore
'blueprint_helios_comments' using configuration from configset
/opt/solr-9.2.1/server/solr/configsets/elastic, trusted=true
[ blueprint_helios_comments] o.a.s.c.CachingDirectoryFactory solr.data.home =
/var/solr/data
[ blueprint_helios_comments] o.a.s.c.SolrCore Opening new SolrCore at
[/var/solr/data/blueprint_helios_comments],
dataDir=[/var/solr/data/blueprint_helios_comments/data/]
[ blueprint_helios_comments] o.a.s.j.SolrRequestAuthorizer Creating a new
SolrRequestAuthorizer
[ blueprint_helios_comments] o.a.s.u.UpdateHandler Using UpdateLog
implementation: org.apache.solr.update.UpdateLog
[ blueprint_helios_comments] o.a.s.u.UpdateLog Initializing UpdateLog:
dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
numVersionBuckets=65536
[ blueprint_helios_comments] o.a.s.u.CommitTracker Hard AutoCommit: if
uncommitted for 15000ms;
[ blueprint_helios_comments] o.a.s.u.CommitTracker Soft AutoCommit: if
uncommitted for 3000ms;
[ blueprint_helios_comments] o.a.s.r.ManagedResourceStorage Cannot write to
config directory /opt/solr-9.2.1/server/solr/configsets/elastic/conf ;
switching to use InMemory storage instead.
[ blueprint_helios_comments] o.a.s.s.s.Suggester init: {threshold=5.0E-4,
name=text, classname=org.apache.solr.spelling.suggest.Suggester,
lookupImpl=org.apache.solr.spelling.suggest.fst.WFSTLookupFactory, field=text}
[ blueprint_helios_comments] o.a.s.h.ReplicationHandler Commits will be
reserved for 1 ms
[ blueprint_helios_comments] o.a.s.u.UpdateLog Could not find max version in
index or recent updates, using new clock 1779334761811017728
I can’t recall having seen this issue before nor can I reproduce it, but I’d
like to ask whether this behaviour is known (couldn’t find a Jira issue or mail
thread connected to this) or anybody has an idea what might be going wrong.
Thanks!
Stefan
--
Stefan Pieper
Senior Software Engineer
[A picture containing grap
Re: Solr 9.2.1: Creation of Core "Hangs"
I don't really have a good idea, but the next log line is:
[ blueprint_helios_comments] o.a.s.r.ManagedResourceStorage Cannot write
> to config directory /opt/solr-9.2.1/server/solr/configsets/elastic/conf ;
> switching to use InMemory storage instead.
Do you have permissions on
"/opt/solr-9.2.1/server/solr/configsets/elastic/conf"
setup correctly?
Kevin Risden
On Tue, Oct 10, 2023 at 10:45 AM Pieper, Stefan
wrote:
> Hello,
>
>
>
> We experienced a problem when creating a core via a request on Solr 9.2.1.
> It seems that the core creation for “blueprint_helios_comments“ just
> “hangs”. Consequently, with subsequent external requests to create the same
> core (since it does not seem to be there yet), Solr detects that the core
> creation is already in progress and returns an error.
>
>
>
> This is what Solr’s log tells me:
>
>
>
> [ blueprint_helios_comments] o.a.s.s.HttpSolrCall [admin] webapp=null
> path=/admin/cores
> params={core=blueprint_helios_comments&action=STATUS&indexInfo=false&wt=javabin&version=2}
> status=0 QTime=1
>
> [ blueprint_helios_comments] o.a.s.h.a.CoreAdminOperation core create
> command
> name=blueprint_helios_comments&action=CREATE&configSet=elastic&wt=javabin&version=2&dataDir=data
>
> [ blueprint_helios_comments] o.a.s.c.SolrConfig Using Lucene
> MatchVersion: 9.4.0
>
> [ blueprint_helios_comments] o.a.s.s.IndexSchema Schema name=elastic
>
> [ blueprint_helios_comments] o.a.s.s.IndexSchema Loaded schema
> elastic/1.6 with uniqueid field id
>
> [ blueprint_helios_comments] o.a.s.c.CoreContainer Creating SolrCore
> 'blueprint_helios_comments' using configuration from configset
> /opt/solr-9.2.1/server/solr/configsets/elastic, trusted=true
>
> [ blueprint_helios_comments] o.a.s.c.CachingDirectoryFactory
> solr.data.home = /var/solr/data
>
> [ blueprint_helios_comments] o.a.s.c.SolrCore Opening new SolrCore at
> [/var/solr/data/blueprint_helios_comments],
> dataDir=[/var/solr/data/blueprint_helios_comments/data/]
>
> [ blueprint_helios_comments] o.a.s.j.SolrRequestAuthorizer Creating a
> new SolrRequestAuthorizer
>
> [ blueprint_helios_comments] o.a.s.u.UpdateHandler Using UpdateLog
> implementation: org.apache.solr.update.UpdateLog
>
> [ blueprint_helios_comments] o.a.s.u.UpdateLog Initializing UpdateLog:
> dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
> numVersionBuckets=65536
>
> [ blueprint_helios_comments] o.a.s.u.CommitTracker Hard AutoCommit: if
> uncommitted for 15000ms;
>
> [ blueprint_helios_comments] o.a.s.u.CommitTracker Soft AutoCommit: if
> uncommitted for 3000ms;
>
>
>
> The creation stops after “Soft AutoCommit”. During the same run of Solr
> and our test suite, other cores are created successfully.
>
>
>
> With the next run of the same software on empty directories and attempt to
> create the core, everything succeeds:
>
>
>
> [ blueprint_helios_comments] o.a.s.s.HttpSolrCall [admin] webapp=null
> path=/admin/cores
> params={core=blueprint_helios_comments&action=STATUS&indexInfo=false&wt=javabin&version=2}
> status=0 QTime=1
>
> [ blueprint_helios_comments] o.a.s.h.a.CoreAdminOperation core create
> command
> name=blueprint_helios_comments&action=CREATE&configSet=elastic&wt=javabin&version=2&dataDir=data
>
> [ blueprint_helios_comments] o.a.s.c.SolrConfig Using Lucene
> MatchVersion: 9.4.0
>
> [ blueprint_helios_comments] o.a.s.s.IndexSchema Schema name=elastic
>
> [ blueprint_helios_comments] o.a.s.s.IndexSchema Loaded schema
> elastic/1.6 with uniqueid field id
>
> [ blueprint_helios_comments] o.a.s.c.CoreContainer Creating SolrCore
> 'blueprint_helios_comments' using configuration from configset
> /opt/solr-9.2.1/server/solr/configsets/elastic, trusted=true
>
> [ blueprint_helios_comments] o.a.s.c.CachingDirectoryFactory
> solr.data.home = /var/solr/data
>
> [ blueprint_helios_comments] o.a.s.c.SolrCore Opening new SolrCore at
> [/var/solr/data/blueprint_helios_comments],
> dataDir=[/var/solr/data/blueprint_helios_comments/data/]
>
> [ blueprint_helios_comments] o.a.s.j.SolrRequestAuthorizer Creating a
> new SolrRequestAuthorizer
>
> [ blueprint_helios_comments] o.a.s.u.UpdateHandler Using UpdateLog
> implementation: org.apache.solr.update.UpdateLog
>
> [ blueprint_helios_comments] o.a.s.u.UpdateLog Initializing UpdateLog:
> dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
> numVersionBuckets=65536
>
> [ blueprint_helios_comments] o.a.s.u.CommitTracker Hard AutoCommit: if
> uncommitted for 15000ms;
>
> [ blueprint_helios_comments] o.a.s.u.CommitTracker Soft AutoCommit: if
> uncommitted for 3000ms;
>
> [ blueprint_helios_comments] o.a.s.r.ManagedResourceStorage Cannot write
> to config directory /opt/solr-9.2.1/server/solr/configsets/elastic/conf ;
> switching to use InMemory storage instead.
>
> [ blueprint_helios_comments] o.a.s.s.s.Suggester init:
> {threshold=5.0E-4, name=text,
> classname=org.apache.solr.spelling.suggest.Suggester,
> lookupImpl=
Issues creating category routed alias in SolrJ
Hello,
I've been investigating how to update our product search to be backed by a
category-routed alias. The initial steps used Postman to connect directly
to the Solr server (version 9.3.0). However, the final solution will need
to work with the SolrJ client (version 9.3.0). Yet I am unable to get SolrJ
to create the category routed alias.
While our client will create and manage the indexes, it may communicate
with servers that need different configurations. So we assume that the
server will have updated its defaults as necessary. In Postman I am using
the following requests to simulate the category creation and solr setup.
/api/cluster
{
"set-obj-property": {
"defaults" : {
"collection": {
"numShards": 3,
"nrtReplicas": 3,
"tlogReplicas": 0,
"pullReplicas": 0
}
}
}
}
/api/aliases
{
"name": "categorisedProducts",
"routers": [
{
"type": "category",
"field": "product_category_s",
"maxCardinality": 1000
}
],
"create-collection": {
"config": "ProductSearch"
}
}
When using SolrJ I am only doing the category creation, but when I omit any
details about how to shard the collection I get an error. If I only create
a collection I do not get an error.
Here is the error
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
Error from server at
http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=TestCategory&router.name=CATEGORY&router.field=product_category_s&router.maxCardinality=0&create-collection.collection.configName=TestCategory&wt=javabin&version=2:
numShards is a required param (when using CompositeId router).
The Java code that creates the category.
Create createCollection = CollectionAdminRequest.createCollection(
categoryName, configSetName, null, null, null, null);
CreateCategoryRoutedAlias createCategory = CollectionAdminRequest.
createCategoryRoutedAlias(categoryName, categoryFieldName, 0,
createCollection);
createCategory.process(client);
Regards
David Driver
Re: Soft commit takes 5 seconds in Solr 8.9.0
On 10/10/23 05:05, John Jackson wrote: Now for the 20 questions: What OS do you have this running on? --> centos How much physical memory is in the machine? --> 370 GB Is it running anything other than Solr? --> Yes, we have 3 Wildfly applications (each has 50GB memory) and 2 microapps (each has 20GB memory). What is the Solr heap size? --> 100 GB memory assign to solr Are you running more than one Solr instance on a single machine? --> Yes, we are using two instances of Solr in one machine. We have 2 shard 2 replicas so one shard and the other shard replica are running on this server. Why run two instances, especially if each one is taking 100GB of memory? One instance can handle multiple replicas with ease. 100GB seems like WAY too much heap. Do you have any evidence that you actually need a heap that big? Solr does not load most index data into the heap, it relies on the OS to cache the bulk of the index data. Have you set up custom GC tuning, or are you using Solr's default? What Java vendor and version are you running? Also, each of the replicas of a shard must run on different physical hardware, or you do not actually have redundancy. If you only have one machine, you might as well run with only one replica since the second replica is only draining resources and will not provide redundancy. I am betting that you are in a situation where there is simply not enough memory for what you are running, especially with 700GB of index data and multiple processes other than Solr on the machine. Can you get the screenshot mentioned on the wiki page that I linked in my earlier message? Note that you cannot attach an image file to an email to the list, as it will be deleted. You need to use a file/image sharing site like dropbox or imgur and give us a link. Thanks, Shawn
