Sorry to have dropped the ball on this past couple of days. I created the
JIRA with information shared on this mail thread.
https://issues.apache.org/jira/browse/SOLR-16838
Running further benchmarks reveals that the slowness is in
the searcher.getFirstMatch() call inside getInputDocument() .
*S
> Solr unfortunately do not yet have a comprehensive official nightly
benchmark run that would catch such regressions. But the community is
working on it, there are some tests on http://mostly.cool maintained by
Ishan, but I have no idea how to add new tests..
I'll publish steps to add new tests s
Hi,
Next step I believe is for you to open a Solr JIRA issue for this regression,
as it is obviously an issue.
Then we can use that JIRA to coordinate, even if the fix ends up being in
Lucene code.
Once we have a Solr JIRA, it may make sense to open a mail thread on
users@lucene list to get so
So I ran the test to index 5 million docs this time (batches of 1000 docs
in 15 parallel threads). To eliminate the network overhead and get as
accurate a benchmark as possible, I used an AtomicLong to measure the time
around the RTG call in DistibutedUpdateProcessor across all calls (
https://gith
Yes, the numbers should be similar. Thanks for digging in. If there is a big
regression, it could be worthwile running the same test on v8.0, 8.1, 8.2 etc
to plot in what version the regression happened. Perhaps it could even be
scripted :)
Solr unfortunately do not yet have a comprehensive off
Sure, I can do that. Let me create an index with a few million docs, call
RTG with a few million iterations on it and note the times between 7.x and
8.x. I assume this should be sufficient (?)
On Wed, May 31, 2023 at 5:19 PM Jan Høydahl wrote:
> Would be nice to determine whether RTG is orders o
Would be nice to determine whether RTG is orders of magnitude slower in 8.x
than 7.x and is the main culprit. Then we could isolate the testing to RTG
only and not involce Atomic Update?
Jan
> 31. mai 2023 kl. 21:33 skrev Rahul Goswami :
>
> I don’t have any nested documents. And the results
Lots of work has been done in this area in 8.x.
See https://issues.apache.org/jira/browse/SOLR-12638 as an example (Support
atomic updates of nested/child documents for nested-enabled schema, fixed in
8.1), which adds support for atomic update of nested docs. Touching all that
code may of cours
I don’t have any nested documents. And the results are consistent across
multiple runs. I tried looking for similar issues in the mailing list, but
couldn’t find anything relevant . So if you do happen to find any JIRAs
addressing it that would be really helpful (thanks!).
To Jan’s question about
I would love some profiling as well. I know 8.8 or 8.9 had some performance
problems with atomic update but this was later addressed. I cant find the
jira atm though. Also I am on 8.11.1 and atomic update is not an issue for
me.
By the way, do you happen to have nested docs?
On Wed, May 31, 2023
Hi
MMap is most important for searching. Indexing bypasses the cache by using
direct IO.
I have noticed slow real time get on Solr 8.x during atomic update myself.
Would be interesting with a comparison with profiling. RTG gets the document
from transaction log I believe? Could there be some R
Thanks for the response Shawn. We are using Windows server with pretty huge
indexes (multiple TB cores). With Mmap, I have observed that the machine
just completely freezes with high CPU and memory usage to a point where it
becomes impossible to even connect to it. SimpleFS works out well for us in
On 5/30/23 15:34, Rahul Goswami wrote:
Environment details: - Java 11 on Windows server - Xms1536m Xmx3072m -
Indexing client code running 15 parallel threads indexing in batches of
1000 - using SimpleFSDirectoryFactory (since Mmap doesn't quite work
well on Windows for our index sizes which co
Adding another detail if it matters...all indexing is happening on
standalone Solr on a single core.
On Tue, May 30, 2023 at 5:34 PM Rahul Goswami wrote:
> Hi,
> We started experiencing slowness with updates in production after
> upgrading from Solr 7.7.2 to 8.11.1. Upon comparing the performanc
Hi,
We started experiencing slowness with updates in production after upgrading
from Solr 7.7.2 to 8.11.1. Upon comparing the performance it turns out that
indexing 20 million docs via atomic updates through the same client program
(running 15 parallel threads indexing in batches of 1000) takes bel
15 matches
Mail list logo
