“The application in question was creating a document per interaction and doing a soft commit at the end of the interaction.“
You also mentioned your autoSoftCommit interval is 1 sec. If you really need NRT, I would suggest the client stop sending a softCommit upon each insert since the (extremely) short autoSoftCommit interval is anyway taking care of making the writes available immediately . If that is not possible, try increasing the autoSoftCommit interval in solrconfig. You don’t need both. Also, do you have any autoWarm cache queries? -Rahul On Sun, Jul 23, 2023 at 1:00 PM Koen De Groote <kdg....@gmail.com> wrote: > According to monitoring, there's no increase in use of file handles or file > descriptors in the period of heavy load, on the entire system. > > On Sun, Jul 23, 2023 at 3:30 PM ufuk yılmaz <uyil...@vivaldi.net.invalid> > wrote: > > > Can it be related to file descriptor/open file handles limit? > > > > — > > > > > On 23 Jul 2023, at 14:24, Koen De Groote <kdg....@gmail.com> wrote: > > > > > > Shawn, > > > > > > After having a look at these files: No, I cannot share them. > > > > > > What I can say is that there's a couple hundred fields, dynamicFields > and > > > copyFields(each). > > > > > > The updatehandler uses solr.DirectUpdateHandler2(the only one I can see > > in > > > the source code extending the regular updateHandler), with a max > > autoCommit > > > time of 60000 and a max autoSoftCommit time of 1000 > > > > > > Regards, > > > Koen > > > > > > > > >> On Sun, Jul 23, 2023 at 1:43 AM Shawn Heisey <elyog...@elyograg.org> > > wrote: > > >> > > >>> On 7/22/23 17:09, Koen De Groote wrote: > > >>> Recently, I experienced softCommits taking up to 30 seconds to > return. > > >> The > > >>> application in question was creating a document per interaction and > > >> doing a > > >>> soft commit at the end of the interaction. After a period of a few > > dozens > > >>> clients sending a continuous stream of such interactions, I could see > > it > > >>> getting slower and slower and the source appears to be the > softCommit. > > >>> > > >>> No changes in the XML config have been provided in terms of commit > > >> timings. > > >>> The JVM is given 20GB heap space, of which it seems to hang steady at > > >> 10GB > > >>> at all times throughout usage, and there's some 25M documents getting > > up > > >> to > > >>> a total of 150GB of data on disk. Everything is on 1 shard, with 2 > > hosts > > >>> each having 1 instance of the collection. The underlying disk is an > SSD > > >> on > > >>> both hosts. > > >>> > > >>> Before I dive into documentation or code, I was wondering if anyone > > here > > >>> might have immediate ideas of what could cause such behavior for soft > > >>> commits. > > >>> > > >>> If someone has an immediate bit of knowledge towards what causes > > >>> softCommits to take up to 30 seconds, that'd be appreciated. > > >> > > >> Can you share the whole config -- solrconfig.xml, the schema, and any > > >> file(s) referenced by either of those. The schema may be named > > >> managed-schema.xml, managed-schema, or schema.xml (or even something > > >> different) depending on Solr version and the rest of the config. > > >> > > >> Normally there isn't anything sensitive in these files, but if you do > > >> have something, redact it as minimally as possible, don't just delete > > >> the whole section with the sensitive data. > > >> > > >> Thanks, > > >> Shawn > > >> > > > > >
