Re: Slow softCommits under heavy load?

[Koen De Groote]

Shawn,

After having a look at these files: No, I cannot share them.

What I can say is that there's a couple hundred fields, dynamicFields and
copyFields(each).

The updatehandler uses solr.DirectUpdateHandler2(the only one I can see in
the source code extending the regular updateHandler), with a max autoCommit
time of 6 and a max autoSoftCommit time of 1000

Regards,
Koen


On Sun, Jul 23, 2023 at 1:43 AM Shawn Heisey  wrote:

> On 7/22/23 17:09, Koen De Groote wrote:
> > Recently, I experienced softCommits taking up to 30 seconds to return.
> The
> > application in question was creating a document per interaction and
> doing a
> > soft commit at the end of the interaction. After a period of a few dozens
> > clients sending a continuous stream of such interactions, I could see it
> > getting slower and slower and the source appears to be the softCommit.
> >
> > No changes in the XML config have been provided in terms of commit
> timings.
> > The JVM is given 20GB heap space, of which it seems to hang steady at
> 10GB
> > at all times throughout usage, and there's some 25M documents getting up
> to
> > a total of 150GB of data on disk. Everything is on 1 shard, with 2 hosts
> > each having 1 instance of the collection. The underlying disk is an SSD
> on
> > both hosts.
> >
> > Before I dive into documentation or code, I was wondering if anyone here
> > might have immediate ideas of what could cause such behavior for soft
> > commits.
> >
> > If someone has an immediate bit of knowledge towards what causes
> > softCommits to take up to 30 seconds, that'd be appreciated.
>
> Can you share the whole config -- solrconfig.xml, the schema, and any
> file(s) referenced by either of those.  The schema may be named
> managed-schema.xml, managed-schema, or schema.xml (or even something
> different) depending on Solr version and the rest of the config.
>
> Normally there isn't anything sensitive in these files, but if you do
> have something, redact it as minimally as possible, don't just delete
> the whole section with the sensitive data.
>
> Thanks,
> Shawn
>

Re: Slow softCommits under heavy load?

[ufuk yılmaz]

Can it be related to file descriptor/open file handles limit?

—

> On 23 Jul 2023, at 14:24, Koen De Groote  wrote:
> 
> Shawn,
> 
> After having a look at these files: No, I cannot share them.
> 
> What I can say is that there's a couple hundred fields, dynamicFields and
> copyFields(each).
> 
> The updatehandler uses solr.DirectUpdateHandler2(the only one I can see in
> the source code extending the regular updateHandler), with a max autoCommit
> time of 6 and a max autoSoftCommit time of 1000
> 
> Regards,
> Koen
> 
> 
>> On Sun, Jul 23, 2023 at 1:43 AM Shawn Heisey  wrote:
>> 
>>> On 7/22/23 17:09, Koen De Groote wrote:
>>> Recently, I experienced softCommits taking up to 30 seconds to return.
>> The
>>> application in question was creating a document per interaction and
>> doing a
>>> soft commit at the end of the interaction. After a period of a few dozens
>>> clients sending a continuous stream of such interactions, I could see it
>>> getting slower and slower and the source appears to be the softCommit.
>>> 
>>> No changes in the XML config have been provided in terms of commit
>> timings.
>>> The JVM is given 20GB heap space, of which it seems to hang steady at
>> 10GB
>>> at all times throughout usage, and there's some 25M documents getting up
>> to
>>> a total of 150GB of data on disk. Everything is on 1 shard, with 2 hosts
>>> each having 1 instance of the collection. The underlying disk is an SSD
>> on
>>> both hosts.
>>> 
>>> Before I dive into documentation or code, I was wondering if anyone here
>>> might have immediate ideas of what could cause such behavior for soft
>>> commits.
>>> 
>>> If someone has an immediate bit of knowledge towards what causes
>>> softCommits to take up to 30 seconds, that'd be appreciated.
>> 
>> Can you share the whole config -- solrconfig.xml, the schema, and any
>> file(s) referenced by either of those.  The schema may be named
>> managed-schema.xml, managed-schema, or schema.xml (or even something
>> different) depending on Solr version and the rest of the config.
>> 
>> Normally there isn't anything sensitive in these files, but if you do
>> have something, redact it as minimally as possible, don't just delete
>> the whole section with the sensitive data.
>> 
>> Thanks,
>> Shawn
>> 

Re: Slow softCommits under heavy load?

[Koen De Groote]

According to monitoring, there's no increase in use of file handles or file
descriptors in the period of heavy load, on the entire system.

On Sun, Jul 23, 2023 at 3:30 PM ufuk yılmaz 
wrote:

> Can it be related to file descriptor/open file handles limit?
>
> —
>
> > On 23 Jul 2023, at 14:24, Koen De Groote  wrote:
> >
> > Shawn,
> >
> > After having a look at these files: No, I cannot share them.
> >
> > What I can say is that there's a couple hundred fields, dynamicFields and
> > copyFields(each).
> >
> > The updatehandler uses solr.DirectUpdateHandler2(the only one I can see
> in
> > the source code extending the regular updateHandler), with a max
> autoCommit
> > time of 6 and a max autoSoftCommit time of 1000
> >
> > Regards,
> > Koen
> >
> >
> >> On Sun, Jul 23, 2023 at 1:43 AM Shawn Heisey 
> wrote:
> >>
> >>> On 7/22/23 17:09, Koen De Groote wrote:
> >>> Recently, I experienced softCommits taking up to 30 seconds to return.
> >> The
> >>> application in question was creating a document per interaction and
> >> doing a
> >>> soft commit at the end of the interaction. After a period of a few
> dozens
> >>> clients sending a continuous stream of such interactions, I could see
> it
> >>> getting slower and slower and the source appears to be the softCommit.
> >>>
> >>> No changes in the XML config have been provided in terms of commit
> >> timings.
> >>> The JVM is given 20GB heap space, of which it seems to hang steady at
> >> 10GB
> >>> at all times throughout usage, and there's some 25M documents getting
> up
> >> to
> >>> a total of 150GB of data on disk. Everything is on 1 shard, with 2
> hosts
> >>> each having 1 instance of the collection. The underlying disk is an SSD
> >> on
> >>> both hosts.
> >>>
> >>> Before I dive into documentation or code, I was wondering if anyone
> here
> >>> might have immediate ideas of what could cause such behavior for soft
> >>> commits.
> >>>
> >>> If someone has an immediate bit of knowledge towards what causes
> >>> softCommits to take up to 30 seconds, that'd be appreciated.
> >>
> >> Can you share the whole config -- solrconfig.xml, the schema, and any
> >> file(s) referenced by either of those.  The schema may be named
> >> managed-schema.xml, managed-schema, or schema.xml (or even something
> >> different) depending on Solr version and the rest of the config.
> >>
> >> Normally there isn't anything sensitive in these files, but if you do
> >> have something, redact it as minimally as possible, don't just delete
> >> the whole section with the sensitive data.
> >>
> >> Thanks,
> >> Shawn
> >>
>
>

Re: Slow softCommits under heavy load?

[Rahul Goswami]

“The application in question was creating a document per interaction and
doing a
soft commit at the end of the interaction.“

You also mentioned your autoSoftCommit interval is 1 sec. If you really
need NRT, I would suggest the client stop sending a softCommit upon each
insert since the (extremely) short autoSoftCommit interval is anyway taking
care of making the writes available immediately .

If that is not possible, try increasing the autoSoftCommit interval in
solrconfig. You don’t need both. Also, do you have any autoWarm cache
queries?

-Rahul



On Sun, Jul 23, 2023 at 1:00 PM Koen De Groote  wrote:

> According to monitoring, there's no increase in use of file handles or file
> descriptors in the period of heavy load, on the entire system.
>
> On Sun, Jul 23, 2023 at 3:30 PM ufuk yılmaz 
> wrote:
>
> > Can it be related to file descriptor/open file handles limit?
> >
> > —
> >
> > > On 23 Jul 2023, at 14:24, Koen De Groote  wrote:
> > >
> > > Shawn,
> > >
> > > After having a look at these files: No, I cannot share them.
> > >
> > > What I can say is that there's a couple hundred fields, dynamicFields
> and
> > > copyFields(each).
> > >
> > > The updatehandler uses solr.DirectUpdateHandler2(the only one I can see
> > in
> > > the source code extending the regular updateHandler), with a max
> > autoCommit
> > > time of 6 and a max autoSoftCommit time of 1000
> > >
> > > Regards,
> > > Koen
> > >
> > >
> > >> On Sun, Jul 23, 2023 at 1:43 AM Shawn Heisey 
> > wrote:
> > >>
> > >>> On 7/22/23 17:09, Koen De Groote wrote:
> > >>> Recently, I experienced softCommits taking up to 30 seconds to
> return.
> > >> The
> > >>> application in question was creating a document per interaction and
> > >> doing a
> > >>> soft commit at the end of the interaction. After a period of a few
> > dozens
> > >>> clients sending a continuous stream of such interactions, I could see
> > it
> > >>> getting slower and slower and the source appears to be the
> softCommit.
> > >>>
> > >>> No changes in the XML config have been provided in terms of commit
> > >> timings.
> > >>> The JVM is given 20GB heap space, of which it seems to hang steady at
> > >> 10GB
> > >>> at all times throughout usage, and there's some 25M documents getting
> > up
> > >> to
> > >>> a total of 150GB of data on disk. Everything is on 1 shard, with 2
> > hosts
> > >>> each having 1 instance of the collection. The underlying disk is an
> SSD
> > >> on
> > >>> both hosts.
> > >>>
> > >>> Before I dive into documentation or code, I was wondering if anyone
> > here
> > >>> might have immediate ideas of what could cause such behavior for soft
> > >>> commits.
> > >>>
> > >>> If someone has an immediate bit of knowledge towards what causes
> > >>> softCommits to take up to 30 seconds, that'd be appreciated.
> > >>
> > >> Can you share the whole config -- solrconfig.xml, the schema, and any
> > >> file(s) referenced by either of those.  The schema may be named
> > >> managed-schema.xml, managed-schema, or schema.xml (or even something
> > >> different) depending on Solr version and the rest of the config.
> > >>
> > >> Normally there isn't anything sensitive in these files, but if you do
> > >> have something, redact it as minimally as possible, don't just delete
> > >> the whole section with the sensitive data.
> > >>
> > >> Thanks,
> > >> Shawn
> > >>
> >
> >
>

Re: Slow softCommits under heavy load?

[Koen De Groote]

Point taken.

Going over the code, I am seeing *autowarm*Count="0" a few times in the
config XML near various LRU and fastLRU cache definitions. Not seeing
specific queries defined in any XML.

Regards,
Koen

On Sun, Jul 23, 2023 at 7:10 PM Rahul Goswami  wrote:

> “The application in question was creating a document per interaction and
> doing a
> soft commit at the end of the interaction.“
>
> You also mentioned your autoSoftCommit interval is 1 sec. If you really
> need NRT, I would suggest the client stop sending a softCommit upon each
> insert since the (extremely) short autoSoftCommit interval is anyway taking
> care of making the writes available immediately .
>
> If that is not possible, try increasing the autoSoftCommit interval in
> solrconfig. You don’t need both. Also, do you have any autoWarm cache
> queries?
>
> -Rahul
>
>
>
> On Sun, Jul 23, 2023 at 1:00 PM Koen De Groote  wrote:
>
> > According to monitoring, there's no increase in use of file handles or
> file
> > descriptors in the period of heavy load, on the entire system.
> >
> > On Sun, Jul 23, 2023 at 3:30 PM ufuk yılmaz  >
> > wrote:
> >
> > > Can it be related to file descriptor/open file handles limit?
> > >
> > > —
> > >
> > > > On 23 Jul 2023, at 14:24, Koen De Groote  wrote:
> > > >
> > > > Shawn,
> > > >
> > > > After having a look at these files: No, I cannot share them.
> > > >
> > > > What I can say is that there's a couple hundred fields, dynamicFields
> > and
> > > > copyFields(each).
> > > >
> > > > The updatehandler uses solr.DirectUpdateHandler2(the only one I can
> see
> > > in
> > > > the source code extending the regular updateHandler), with a max
> > > autoCommit
> > > > time of 6 and a max autoSoftCommit time of 1000
> > > >
> > > > Regards,
> > > > Koen
> > > >
> > > >
> > > >> On Sun, Jul 23, 2023 at 1:43 AM Shawn Heisey  >
> > > wrote:
> > > >>
> > > >>> On 7/22/23 17:09, Koen De Groote wrote:
> > > >>> Recently, I experienced softCommits taking up to 30 seconds to
> > return.
> > > >> The
> > > >>> application in question was creating a document per interaction and
> > > >> doing a
> > > >>> soft commit at the end of the interaction. After a period of a few
> > > dozens
> > > >>> clients sending a continuous stream of such interactions, I could
> see
> > > it
> > > >>> getting slower and slower and the source appears to be the
> > softCommit.
> > > >>>
> > > >>> No changes in the XML config have been provided in terms of commit
> > > >> timings.
> > > >>> The JVM is given 20GB heap space, of which it seems to hang steady
> at
> > > >> 10GB
> > > >>> at all times throughout usage, and there's some 25M documents
> getting
> > > up
> > > >> to
> > > >>> a total of 150GB of data on disk. Everything is on 1 shard, with 2
> > > hosts
> > > >>> each having 1 instance of the collection. The underlying disk is an
> > SSD
> > > >> on
> > > >>> both hosts.
> > > >>>
> > > >>> Before I dive into documentation or code, I was wondering if anyone
> > > here
> > > >>> might have immediate ideas of what could cause such behavior for
> soft
> > > >>> commits.
> > > >>>
> > > >>> If someone has an immediate bit of knowledge towards what causes
> > > >>> softCommits to take up to 30 seconds, that'd be appreciated.
> > > >>
> > > >> Can you share the whole config -- solrconfig.xml, the schema, and
> any
> > > >> file(s) referenced by either of those.  The schema may be named
> > > >> managed-schema.xml, managed-schema, or schema.xml (or even something
> > > >> different) depending on Solr version and the rest of the config.
> > > >>
> > > >> Normally there isn't anything sensitive in these files, but if you
> do
> > > >> have something, redact it as minimally as possible, don't just
> delete
> > > >> the whole section with the sensitive data.
> > > >>
> > > >> Thanks,
> > > >> Shawn
> > > >>
> > >
> > >
> >
>

Re: [ANNOUNCE] Apache Solr 9.3.0 released

[David Smiley]

Yes; thanks Houston!
This is a notable Solr release.

~ David


On Fri, Jul 21, 2023 at 4:08 PM Alessandro Benedetti 
wrote:

> Thanks Houston and everyone involved, for the release!
>
>
> On Fri, 21 Jul 2023, 21:56 Houston Putman,  wrote:
>
> > The Solr PMC is pleased to announce the release of Apache Solr 9.3.0.
> >
> > Solr is the popular, blazing fast, open source NoSQL search platform from
> > the Apache Solr project. Its major features include powerful full-text
> > search, hit highlighting, faceted search, dynamic clustering, database
> > integration, rich document handling, and geospatial search. Solr is
> highly
> > scalable, providing fault tolerant distributed search and indexing, and
> > powers the search and navigation features of many of the world's largest
> > internet sites.
> >
> > Solr 9.3.0 is available for immediate download at:
> >
> >   
> >
> > *Solr 9.3.0 Release Highlights:*
> >
> >- The Lucene version used by Solr has been upgraded to 9.7.
> >- Solr releases now have a slim variant, both for the binary release
> and
> >the docker image.
> >   - The Slim variant is the same as the normal variant, except that
> it
> >   does not include Solr modules or the Prometheus exporter.
> >- Vector Search
> >   - Added support for byte vector encoding in DenseVectorField and
> >   KnnQParser
> >   - High dimensional vectors are now supported in Solr
> >   - Solr can now take advantage of SIMD optimizations for Vector
> >   calculations, when run with Java 20 or 21.
> >   - A new "vectorSimilarity" function query has been added to
> calculate
> >   similarity scores for DenseVectorFields
> >- Solr now provides an "Install Shard" API
> ><
> >
> https://solr.apache.org/guide/solr/9_3/deployment-guide/shard-management.html#installsharddata
> > >
> >to allow users who have built (per-shard) indices offline to import
> them
> >into SolrCloud shards.
> >- Solr’s experimental "v2" API has seen a number of improvements in
> the
> >9.3 release.
> >   - It is now approaching parity with the functionality offered by
> >   Solr’s v1 API.
> >   - The v2 API as a whole is being redesigned to be more REST-ful and
> >   intuitive
> >   See the Changelog and upgrade notes for information on which v2
> APIs
> >   have backward-incompatible changes.
> >- New APIs for MigrateReplicas and BalanceReplicas. These work
> >out-of-the-box with the built-in PlacementPlugins.
> >   - The AffinityPlacementPlugin now supports co-location of shards
> >   between collections, using the "withCollectionShards" parameter.
> >- Join Queries may handle equally sharded collections on both sides.
> >   - Collections shards should be collocated via
> >   AffinityPlacementPlugin.withCollectionShards
> >   - This operation doesn't support SplitShard
> >- Unknown cores are no longer deleted by default when Solr starts. Use
> >"solr.deleteUnknownCores=true" to use the previous behavior.
> >- Warning: Solr cannot be used with Java 20 on MacOS with the Java
> >Security Manager.
> >Please use the environment variable
> SOLR_SECURITY_MANAGER_ENABLED=false
> >when running with Java 20 on MacOS.
> >
> >
> > Please refer to the Upgrade Notes in the Solr Ref Guide for information
> on
> > upgrading from previous Solr versions:
> >
> >   <
> >
> >
> https://solr.apache.org/guide/solr/9_3/upgrade-notes/solr-upgrade-notes.html
> > >
> >
> > Please read CHANGES.txt for a full list of new features, changes and
> > bugfixes:
> >
> >   
> >
>

Re: Slow softCommits under heavy load?

[Rahul Goswami]

Ok if the count for cache autoWarm for the various caches is 0 then there
is no cache warmup, so that shouldn’t be contributing to the slowness.

For now, I would recommend increasing the autoSoftCommit interval to a
higher number like 6 (1 min) and see if you observe any difference in
performance, although stopping those softCommits upon each client update
call (especially during heavy write load) is what should help more.

-Rahul

On Sun, Jul 23, 2023 at 3:54 PM Koen De Groote  wrote:

> Point taken.
>
> Going over the code, I am seeing *autowarm*Count="0" a few times in the
> config XML near various LRU and fastLRU cache definitions. Not seeing
> specific queries defined in any XML.
>
> Regards,
> Koen
>
> On Sun, Jul 23, 2023 at 7:10 PM Rahul Goswami 
> wrote:
>
> > “The application in question was creating a document per interaction and
> > doing a
> > soft commit at the end of the interaction.“
> >
> > You also mentioned your autoSoftCommit interval is 1 sec. If you really
> > need NRT, I would suggest the client stop sending a softCommit upon each
> > insert since the (extremely) short autoSoftCommit interval is anyway
> taking
> > care of making the writes available immediately .
> >
> > If that is not possible, try increasing the autoSoftCommit interval in
> > solrconfig. You don’t need both. Also, do you have any autoWarm cache
> > queries?
> >
> > -Rahul
> >
> >
> >
> > On Sun, Jul 23, 2023 at 1:00 PM Koen De Groote 
> wrote:
> >
> > > According to monitoring, there's no increase in use of file handles or
> > file
> > > descriptors in the period of heavy load, on the entire system.
> > >
> > > On Sun, Jul 23, 2023 at 3:30 PM ufuk yılmaz
>  > >
> > > wrote:
> > >
> > > > Can it be related to file descriptor/open file handles limit?
> > > >
> > > > —
> > > >
> > > > > On 23 Jul 2023, at 14:24, Koen De Groote 
> wrote:
> > > > >
> > > > > Shawn,
> > > > >
> > > > > After having a look at these files: No, I cannot share them.
> > > > >
> > > > > What I can say is that there's a couple hundred fields,
> dynamicFields
> > > and
> > > > > copyFields(each).
> > > > >
> > > > > The updatehandler uses solr.DirectUpdateHandler2(the only one I can
> > see
> > > > in
> > > > > the source code extending the regular updateHandler), with a max
> > > > autoCommit
> > > > > time of 6 and a max autoSoftCommit time of 1000
> > > > >
> > > > > Regards,
> > > > > Koen
> > > > >
> > > > >
> > > > >> On Sun, Jul 23, 2023 at 1:43 AM Shawn Heisey <
> elyog...@elyograg.org
> > >
> > > > wrote:
> > > > >>
> > > > >>> On 7/22/23 17:09, Koen De Groote wrote:
> > > > >>> Recently, I experienced softCommits taking up to 30 seconds to
> > > return.
> > > > >> The
> > > > >>> application in question was creating a document per interaction
> and
> > > > >> doing a
> > > > >>> soft commit at the end of the interaction. After a period of a
> few
> > > > dozens
> > > > >>> clients sending a continuous stream of such interactions, I could
> > see
> > > > it
> > > > >>> getting slower and slower and the source appears to be the
> > > softCommit.
> > > > >>>
> > > > >>> No changes in the XML config have been provided in terms of
> commit
> > > > >> timings.
> > > > >>> The JVM is given 20GB heap space, of which it seems to hang
> steady
> > at
> > > > >> 10GB
> > > > >>> at all times throughout usage, and there's some 25M documents
> > getting
> > > > up
> > > > >> to
> > > > >>> a total of 150GB of data on disk. Everything is on 1 shard, with
> 2
> > > > hosts
> > > > >>> each having 1 instance of the collection. The underlying disk is
> an
> > > SSD
> > > > >> on
> > > > >>> both hosts.
> > > > >>>
> > > > >>> Before I dive into documentation or code, I was wondering if
> anyone
> > > > here
> > > > >>> might have immediate ideas of what could cause such behavior for
> > soft
> > > > >>> commits.
> > > > >>>
> > > > >>> If someone has an immediate bit of knowledge towards what causes
> > > > >>> softCommits to take up to 30 seconds, that'd be appreciated.
> > > > >>
> > > > >> Can you share the whole config -- solrconfig.xml, the schema, and
> > any
> > > > >> file(s) referenced by either of those.  The schema may be named
> > > > >> managed-schema.xml, managed-schema, or schema.xml (or even
> something
> > > > >> different) depending on Solr version and the rest of the config.
> > > > >>
> > > > >> Normally there isn't anything sensitive in these files, but if you
> > do
> > > > >> have something, redact it as minimally as possible, don't just
> > delete
> > > > >> the whole section with the sensitive data.
> > > > >>
> > > > >> Thanks,
> > > > >> Shawn
> > > > >>
> > > >
> > > >
> > >
> >
>