Hi, This answer comes late, but maybe it's still a topic for you.
As you say you can have a lot of dynamicFields, I remember that in the past, I saw long commits due to slow segment merges related to docValues and dynamicfield. Are you intensively using DynamicFields with docValues ? Furthermore, you can enable Lucene detailed debug information (<infoStream>true</infoStream>) ? https://lucene.apache.org/solr/guide/8_5/indexconfig-in-solrconfig.html#other-indexing-settings With these Lucene debug information, are there any lines like this in your logs ? 2020-05-03 16:22:38.139 INFO (qtp1837543557-787) [ x:###] o.a.s.u.LoggingInfoStream [MS][qtp1837543557-787]: too many merges; stalling... 2020-05-03 16:24:58.318 INFO (commitScheduler-19-thread-1) [ x:###] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false} 2020-05-03 16:24:59.005 INFO (commitScheduler-19-thread-1) [ x:###] o.a.s.u.LoggingInfoStream [MS][commitScheduler-19-thread-1]: too many merges; stalling... 2020-05-03 16:31:31.402 INFO (Lucene Merge Thread #55) [ x:###] o.a.s.u.LoggingInfoStream [SM][Lucene Merge Thread #55]: 1291879 msec to merge doc values [464265 docs] Regards Dominique Le dim. 23 juil. 2023 à 13:25, Koen De Groote <kdg....@gmail.com> a écrit : > Shawn, > > After having a look at these files: No, I cannot share them. > > What I can say is that there's a couple hundred fields, dynamicFields and > copyFields(each). > > The updatehandler uses solr.DirectUpdateHandler2(the only one I can see in > the source code extending the regular updateHandler), with a max autoCommit > time of 60000 and a max autoSoftCommit time of 1000 > > Regards, > Koen > > > On Sun, Jul 23, 2023 at 1:43 AM Shawn Heisey <elyog...@elyograg.org> > wrote: > > > On 7/22/23 17:09, Koen De Groote wrote: > > > Recently, I experienced softCommits taking up to 30 seconds to return. > > The > > > application in question was creating a document per interaction and > > doing a > > > soft commit at the end of the interaction. After a period of a few > dozens > > > clients sending a continuous stream of such interactions, I could see > it > > > getting slower and slower and the source appears to be the softCommit. > > > > > > No changes in the XML config have been provided in terms of commit > > timings. > > > The JVM is given 20GB heap space, of which it seems to hang steady at > > 10GB > > > at all times throughout usage, and there's some 25M documents getting > up > > to > > > a total of 150GB of data on disk. Everything is on 1 shard, with 2 > hosts > > > each having 1 instance of the collection. The underlying disk is an SSD > > on > > > both hosts. > > > > > > Before I dive into documentation or code, I was wondering if anyone > here > > > might have immediate ideas of what could cause such behavior for soft > > > commits. > > > > > > If someone has an immediate bit of knowledge towards what causes > > > softCommits to take up to 30 seconds, that'd be appreciated. > > > > Can you share the whole config -- solrconfig.xml, the schema, and any > > file(s) referenced by either of those. The schema may be named > > managed-schema.xml, managed-schema, or schema.xml (or even something > > different) depending on Solr version and the rest of the config. > > > > Normally there isn't anything sensitive in these files, but if you do > > have something, redact it as minimally as possible, don't just delete > > the whole section with the sensitive data. > > > > Thanks, > > Shawn > > >
