End of Life of Solr 8.11?
Dear All, According to https://solr.apache.org/downloads.html#about-versions-and-support, all Solr versions <8.11 are End of Life. Is it already known when Solr 8.11 will have its End Of Life? Best regards, Thomas Heldmann -- Thomas Heldmann Bayerische Staatsbibliothek Verbundzentrale des Bibliotheksverbunds Bayern Leopoldstraße 240 80807 München Tel.: 089/28638-4153 E-Mail: thomas.heldm...@bsb-muenchen.de
Re: Atomic updates too slow in Solr 8 vs Solr 7
Yes, the numbers should be similar. Thanks for digging in. If there is a big regression, it could be worthwile running the same test on v8.0, 8.1, 8.2 etc to plot in what version the regression happened. Perhaps it could even be scripted :) Solr unfortunately do not yet have a comprehensive official nightly benchmark run that would catch such regressions. But the community is working on it, there are some tests on http://mostly.cool maintained by Ishan, but I have no idea how to add new tests.. Jan > 31. mai 2023 kl. 23:50 skrev Rahul Goswami : > > Sure, I can do that. Let me create an index with a few million docs, call > RTG with a few million iterations on it and note the times between 7.x and > 8.x. I assume this should be sufficient (?) > > On Wed, May 31, 2023 at 5:19 PM Jan Høydahl wrote: > >> Would be nice to determine whether RTG is orders of magnitude slower in >> 8.x than 7.x and is the main culprit. Then we could isolate the testing to >> RTG only and not involce Atomic Update? >> >> Jan >> >>> 31. mai 2023 kl. 21:33 skrev Rahul Goswami : >>> >>> I don’t have any nested documents. And the results are consistent across >>> multiple runs. I tried looking for similar issues in the mailing list, >> but >>> couldn’t find anything relevant . So if you do happen to find any JIRAs >>> addressing it that would be really helpful (thanks!). >>> >>> To Jan’s question about RTG taking more time in Solr 8.x, I can say with >>> good certainty that this is the case. Although it does look into >>> transaction logs first, thread dumps suggest that it is the next phase >>> (when it doesn't find the doc in tlog) which seems to be time consuming . >>> It tries to look up the document via the current searcher >>> (searcher.getFirstMatch() ). Proceeding further in the stack, it is this >>> call where many threads are spending time: >>> >>> >> https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.11.1/lucene/core/src/java/org/apache/lucene/codecs/blocktree/SegmentTermsEnum.java#L485 >>> >>> Although this call is the same in 7.7.2 and 8.11.1 quite likely >>> something changed in Lucene's FST.java which is causing the slowness. I >> am >>> trying to dig further and might also ask folks on the Lucene mailing >> list. >>> Thanks. >>> >>> >>> >>> On Wed, May 31, 2023 at 11:36 AM Srijan wrote: >>> I would love some profiling as well. I know 8.8 or 8.9 had some >> performance problems with atomic update but this was later addressed. I cant find >> the jira atm though. Also I am on 8.11.1 and atomic update is not an issue >> for me. By the way, do you happen to have nested docs? On Wed, May 31, 2023, 11:20 Jan Høydahl wrote: > Hi > > MMap is most important for searching. Indexing bypasses the cache by using > direct IO. > > I have noticed slow real time get on Solr 8.x during atomic update myself. > Would be interesting with a comparison with profiling. RTG gets the > document from transaction log I believe? Could there be some RTG >> changes in > 8.x that caused such slowdown? > > Jan Høydahl > >> 31. mai 2023 kl. 16:57 skrev Rahul Goswami : >> >> Thanks for the response Shawn. We are using Windows server with >> pretty > huge >> indexes (multiple TB cores). With Mmap, I have observed that the machine >> just completely freezes with high CPU and memory usage to a point >> where > it >> becomes impossible to even connect to it. SimpleFS works out well for us > in >> this case. >> >> As noted in my first email, even with SimpleFS, Solr 7 completes the > crawl >> in nearly 1/5th the time taken in Solr 8. Hence there should be something >> OUTSIDE the directory factory in the code which is causing this. >> >> Thanks, >> Rahul >> >> >>> On Tue, May 30, 2023 at 10:47 PM Shawn Heisey > wrote: >>> On 5/30/23 15:34, Rahul Goswami wrote: Environment details: - Java 11 on Windows server - Xms1536m Xmx3072m - Indexing client code running 15 parallel threads indexing in batches of 1000 - using SimpleFSDirectoryFactory (since Mmap doesn't quite work well on Windows for our index sizes which commonly run north of 1 >> TB) >>> >>> Don't change the directoryFactory. You *WANT* Solr to use MMAP for your >>> indexes. Not using MMAP is likely to slow things down considerably. >>> MMAP should work just fine on 64-bit Windows with 64-bit Java. Which of >>> course requires 64-bit hardware. >>> >>> 32 bit systems and software cannot properly deal with data larger >> than >>> about 2GB. >>> >>> Thanks, >>> Shawn >>> > >> >>
Re: End of Life of Solr 8.11?
Hi, Yes, it is EOL the same date that 10.0 is shippped. The release date of 10.0 is yet to be determined. I expect a few more 9.x releases to happen first though, and most likely we'll wait for Lucene 10 to ship first. Jan > 1. jun. 2023 kl. 09:36 skrev Thomas Heldmann > : > > Dear All, > > According to > https://solr.apache.org/downloads.html#about-versions-and-support, all Solr > versions <8.11 are End of Life. Is it already known when Solr 8.11 will have > its End Of Life? > > Best regards, > Thomas Heldmann > > -- > Thomas Heldmann > Bayerische Staatsbibliothek > Verbundzentrale des Bibliotheksverbunds Bayern > Leopoldstraße 240 > 80807 München > > Tel.: 089/28638-4153 > E-Mail: thomas.heldm...@bsb-muenchen.de > > >
Re: End of Life of Solr 8.11?
Do we have a rough idea of when we think Solr 10 will be? A few months? A year? I just have an upgrade project to Solr 9 on the horizon and might hold off on it a bit if Solr 10 is imminent. > Yes, it is EOL the same date that 10.0 is shippped. The release date of 10.0 > is yet to be determined. I expect a few more 9.x releases to happen first > though, and most likely we'll wait for Lucene 10 to ship first.
RE: Issue in connecting zookeeper host from node js service
Classification: Public It would be great If anyone can help on this. Thanks for the assistance. Regards Manoj Jain -Original Message- From: Manoj Jain Sent: Wednesday, May 31, 2023 4:22 PM To: users@solr.apache.org; Ishan Chattopadhyaya Subject: RE: Issue in connecting zookeeper host from node js service Classification: Public Thanks for the quick response. As far as ZK connection string is concerned it is valid and accessible and with the same I can successfully connect to the java service (via Solrj library - CloudSolrCleint) We are using external zookeeper in our solr cloud setup in Kubernetes. Our headless zookeeper internally has three zookeeper pods (to make a quorum) which stores the configuration data for solr in a hierarchical data structure (Znode tree). Further solr cluster have three solr pods. The reason we want to connect to ZK instead of Solr cluster directly because(as per my understanding) its Zk only which is aware of the true state of Solr cluster. If we directly connect to Solr cluster instead of going through the ZK then I believe in production there are chances to encounter some weird scenarios, for instance the request may go to a Solr node which may not have the latest data. Please share your feedback/inputs on the same. Regards Manoj Jain -Original Message- From: Ishan Chattopadhyaya Sent: Wednesday, May 31, 2023 1:56 PM To: manoj.j...@hcl.com.invalid Cc: users@solr.apache.org Subject: Re: Issue in connecting zookeeper host from node js service [CAUTION: This Email is from outside the Organization. Unless you trust the sender, Don’t click links or open attachments as it may be a Phishing email, which can steal your Information and compromise your Computer.] If you're using Solr in Kubernetes, you can continue connecting to Solr pods without connecting to ZK. We don't have any officially supported Solr client for NodeJS, but the HTTP endpoints of Solr are supported and can be used directly. If you're having trouble connecting to ZK via a third party (unofficial) Solr client in NodeJS, maybe you can reach out to the support for that client library. Also, you can make sure the ZK connection string you're using is valid and accessible from outside the Kubernetes pods. On Wed, 31 May, 2023, 12:00 am Manoj Jain, wrote: > Classification: Confidential > Hello, > > We are migrating from Solr-standalone to Solr-cloud of version 8.11 > using external zookeeper. We have the node and java microservices > which are connected to Solr for indexing and query and everything is > deployed within same GKE cluster. > > For the migration of solr cloud we have the task to modify the > connection string so that it can point to Solr Cloud via zookeeper. > For the java services we use 'CloudSolrClient' class of SolrJ library > to connect to the zookeeper host and it worked well. And for the node > service we are using 'Solr-client' library, but we are not able to > connect to the zookeeper host using the same. For testing purpose, we > give solr-cluster I.P. and port instead after which the service can > connect but the same failed when we try to connect to zookeeper host > (neither from DNS name or I.P. and port). We do not see any other > specific solr library for node js to connect to Solr Cloud. Can anyone > please suggest how we can connect to zookeeper host from node js service? > > Regards > Manoj Jain > > ::DISCLAIMER:: > > The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. E-mail transmission is not > guaranteed to be secure or error-free as information could be > intercepted, corrupted, lost, destroyed, arrive late or incomplete, or > may contain viruses in transmission. The e mail and its contents (with > or without referred errors) shall therefore not attach any liability > on the originator or HCL or its affiliates. Views or opinions, if any, > presented in this email are solely those of the author and may not > necessarily reflect the views or opinions of HCL or its affiliates. > Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of this message > without the prior written consent of authorized representative of HCL > is strictly prohibited. If you have received this email in error > please delete it and notify the sender immediately. Before opening any > email and/or attachments, please check them for viruses and other defects. > >
Re: End of Life of Solr 8.11?
At least a year, but I would not be surprised if it's 2 years or longer. Recently Solr has had a 2-3 year span between major releases, and Solr 9 was released a bit over a year ago. - Houston On Thu, Jun 1, 2023 at 8:36 AM Andy Lester wrote: > Do we have a rough idea of when we think Solr 10 will be? A few months? A > year? > > I just have an upgrade project to Solr 9 on the horizon and might hold off > on it a bit if Solr 10 is imminent. > > > Yes, it is EOL the same date that 10.0 is shippped. The release date of > 10.0 is yet to be determined. I expect a few more 9.x releases to happen > first though, and most likely we'll wait for Lucene 10 to ship first. > >
Re: Streaming expr as LTR feature
Thank you for links to the docs, they were super helpful and the expression
syntax also works perfectly. Appreciate it.
Just another question inline to the streaming api. Is it possible to invoke
the stream expression via solr local param standard query syntax such as
{!stream expr=select()}. Any specific handler that can allow stream expr go
through standard solr request and perhaps also returns stream response into
standard solr response?
On Wed, May 31, 2023 at 1:53 PM Joel Bernstein wrote:
> The select function is documented here:
>
> https://solr.apache.org/guide/solr/latest/query-guide/transform.html
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Wed, May 31, 2023 at 1:51 PM Joel Bernstein wrote:
>
> > The array function doesn't operate in the way its being used here. Here
> > are the docs on arrays:
> >
> >
> >
> https://solr.apache.org/guide/solr/latest/query-guide/vector-math.html#arrays
> >
> > Using a multi-valued field you could do something like this where test_fs
> > is a multi-valued float field. The select function evaluates arrays
> > automatically so they can be operated on by math expressions in a
> streaming
> > context.
> >
> > select(search(jdata, fl=test_fs),
> > dotProduct(test_fs, test_fs) as p)
> >
> > Which returns:
> >
> > { "result-set": { "docs": [ { "p": 1.9350813535659377 }, { "p":
> > 2.2449532856850816 }, { "p": 1.7212359783803421 }, { "p":
> 2.761290822044021
> > },
> >
> >
> >
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Wed, May 31, 2023 at 12:59 PM Alessandro Benedetti <
> > a.benede...@sease.io> wrote:
> >
> >> Hi,
> >> we are working on contributing the possibility of having
> vector-similarity
> >> features, in Apache Solr Learning To Rank.
> >> We are starting from the Lucene contribution of related function
> queries,
> >> which we are close to merging.
> >> Then we'll do the Solr part.
> >>
> >> What you are trying to do has not been tested, it may work but there's
> no
> >> dedicated design for that so it may be quite clunky and expensive.
> >> And by the way, Images are not visible in the mailing list.
> >>
> >> Cheers
> >> --
> >> *Alessandro Benedetti*
> >> Director @ Sease Ltd.
> >> *Apache Lucene/Solr Committer*
> >> *Apache Solr PMC Member*
> >>
> >> e-mail: a.benede...@sease.io
> >>
> >>
> >> *Sease* - Information Retrieval Applied
> >> Consulting | Training | Open Source
> >>
> >> Website: Sease.io | Twitter
> >> | Youtube
> >> | Github
> >>
> >>
> >>
> >> On Wed, 31 May 2023 at 16:05, rajani m wrote:
> >>
> >> > Validating the expression to begin with, it doesn't work. Vector math
> >> > supports reading from an array of values so I tried the following
> >> > expression.
> >> >
> >> > dotProduct(array(search(v9,
> >> >q="id:1",
> >> >fl="numeric_field_dfd",
> >> >sort="numeric_field_dfd asc",
> >> >qt="/export")),array(2))
> >> >
> >> >
> >> > where numeric_field_dfd - single valued dynamic field double type. I
> >> > tried the multivalued double type assuming that it converts to an
> array
> >> of
> >> > values but it didn't work, so tried the single value to start with.
> >> >
> >> > output of the expression is an exception -
> >> >
> >> > [image: image.png]
> >> >
> >> >
> >> > The value is not null as seen below, so am I wrong in terms of
> >> expression
> >> > syntax then, any suggestions?
> >> >
> >> >
> >> > [image: image.png]
> >> >
> >> >
> >> >
> >> > On Tue, May 30, 2023 at 4:47 PM rajani m
> wrote:
> >> >
> >> >> Hi Solr Users,
> >> >>
> >> >>Does LTR Solr Feature
> >> >> <
> >>
> https://solr.apache.org/guide/8_7/learning-to-rank.html#feature-engineering
> >
> >> support
> >> >> streaming expressions? Steaming expr supports vector math
> >> >> <
> >>
> https://solr.apache.org/guide/7_5/vector-math.html#dot-product-and-cosine-similarity
> >> >,
> >> >> I am trying to configure stream apis vector math as a solr feature
> >> which
> >> >> would fetch a vector from a document field and another from query
> >> param and
> >> >> compute cosine or dot product.
> >> >>
> >> >> For example, a LTR feature definition that would look like below, is
> >> this
> >> >> supported? Does LTR solr feature support parsing streaming api
> >> requests and
> >> >> its somewhat unique response that is not same as standard solr
> >> response?
> >> >>
> >> >>
> >> >> {
> >> >> "name": "vector_sim_score",
> >> >> "class": "org.apache.solr.ltr.feature.SolrFeature",
> >> >> "params": {
> >> >> "q":
> >> "expr=dotProduct(search(collection_name,q="id:$uniq_id",fl="doc_vector",
> >> sort="from asc", qt="/export"), ${query_vector})"
> >> >> },
> >> >> "store
Re: Streaming expr as LTR feature
There currently is not a way to inline a streaming expression into a
traditional Solr search. Part of the reason why is that they have different
performance and design goals. A standard search is designed to scale to a
high qps with sub-second performance. The design of Solr's
traditional search supports that. Streaming expressions were never designed
to perform at scale like that. So it would be hard to integrate streaming
expressions into a system that required high qps and sub-second
performance.
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Jun 1, 2023 at 10:28 AM rajani m wrote:
> Thank you for links to the docs, they were super helpful and the expression
> syntax also works perfectly. Appreciate it.
> Just another question inline to the streaming api. Is it possible to invoke
> the stream expression via solr local param standard query syntax such as
> {!stream expr=select()}. Any specific handler that can allow stream expr go
> through standard solr request and perhaps also returns stream response into
> standard solr response?
>
> On Wed, May 31, 2023 at 1:53 PM Joel Bernstein wrote:
>
> > The select function is documented here:
> >
> > https://solr.apache.org/guide/solr/latest/query-guide/transform.html
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Wed, May 31, 2023 at 1:51 PM Joel Bernstein
> wrote:
> >
> > > The array function doesn't operate in the way its being used here. Here
> > > are the docs on arrays:
> > >
> > >
> > >
> >
> https://solr.apache.org/guide/solr/latest/query-guide/vector-math.html#arrays
> > >
> > > Using a multi-valued field you could do something like this where
> test_fs
> > > is a multi-valued float field. The select function evaluates arrays
> > > automatically so they can be operated on by math expressions in a
> > streaming
> > > context.
> > >
> > > select(search(jdata, fl=test_fs),
> > > dotProduct(test_fs, test_fs) as p)
> > >
> > > Which returns:
> > >
> > > { "result-set": { "docs": [ { "p": 1.9350813535659377 }, { "p":
> > > 2.2449532856850816 }, { "p": 1.7212359783803421 }, { "p":
> > 2.761290822044021
> > > },
> > >
> > >
> > >
> > >
> > > Joel Bernstein
> > > http://joelsolr.blogspot.com/
> > >
> > >
> > > On Wed, May 31, 2023 at 12:59 PM Alessandro Benedetti <
> > > a.benede...@sease.io> wrote:
> > >
> > >> Hi,
> > >> we are working on contributing the possibility of having
> > vector-similarity
> > >> features, in Apache Solr Learning To Rank.
> > >> We are starting from the Lucene contribution of related function
> > queries,
> > >> which we are close to merging.
> > >> Then we'll do the Solr part.
> > >>
> > >> What you are trying to do has not been tested, it may work but there's
> > no
> > >> dedicated design for that so it may be quite clunky and expensive.
> > >> And by the way, Images are not visible in the mailing list.
> > >>
> > >> Cheers
> > >> --
> > >> *Alessandro Benedetti*
> > >> Director @ Sease Ltd.
> > >> *Apache Lucene/Solr Committer*
> > >> *Apache Solr PMC Member*
> > >>
> > >> e-mail: a.benede...@sease.io
> > >>
> > >>
> > >> *Sease* - Information Retrieval Applied
> > >> Consulting | Training | Open Source
> > >>
> > >> Website: Sease.io | Twitter
> > >> | Youtube
> > >> | Github
> > >>
> > >>
> > >>
> > >> On Wed, 31 May 2023 at 16:05, rajani m wrote:
> > >>
> > >> > Validating the expression to begin with, it doesn't work. Vector
> math
> > >> > supports reading from an array of values so I tried the following
> > >> > expression.
> > >> >
> > >> > dotProduct(array(search(v9,
> > >> >q="id:1",
> > >> >fl="numeric_field_dfd",
> > >> >sort="numeric_field_dfd asc",
> > >> >qt="/export")),array(2))
> > >> >
> > >> >
> > >> > where numeric_field_dfd - single valued dynamic field double type.
> I
> > >> > tried the multivalued double type assuming that it converts to an
> > array
> > >> of
> > >> > values but it didn't work, so tried the single value to start with.
> > >> >
> > >> > output of the expression is an exception -
> > >> >
> > >> > [image: image.png]
> > >> >
> > >> >
> > >> > The value is not null as seen below, so am I wrong in terms of
> > >> expression
> > >> > syntax then, any suggestions?
> > >> >
> > >> >
> > >> > [image: image.png]
> > >> >
> > >> >
> > >> >
> > >> > On Tue, May 30, 2023 at 4:47 PM rajani m
> > wrote:
> > >> >
> > >> >> Hi Solr Users,
> > >> >>
> > >> >>Does LTR Solr Feature
> > >> >> <
> > >>
> >
> https://solr.apache.org/guide/8_7/learning-to-rank.html#feature-engineering
> > >
> > >> support
> > >> >> streaming expressions? Steaming expr supports vector math
> > >> >> <
> > >>
> >
> ht
Re: End of Life of Solr 8.11?
If Lucene releases v10 this autumn, I may propose Solr 10 to follow quickly, in part to EOL the aging 8.x line with ant build system. You’ll just have to wait and see. If I needed to upgrade soon I’d go for 9.x, which will be supported and patched for a long long time. Jan Høydahl > 1. jun. 2023 kl. 16:23 skrev Houston Putman : > > At least a year, but I would not be surprised if it's 2 years or longer. > Recently Solr has had a 2-3 year span between major releases, and Solr 9 > was released a bit over a year ago. > > - Houston > >> On Thu, Jun 1, 2023 at 8:36 AM Andy Lester wrote: >> >> Do we have a rough idea of when we think Solr 10 will be? A few months? A >> year? >> >> I just have an upgrade project to Solr 9 on the horizon and might hold off >> on it a bit if Solr 10 is imminent. >> >>> Yes, it is EOL the same date that 10.0 is shippped. The release date of >> 10.0 is yet to be determined. I expect a few more 9.x releases to happen >> first though, and most likely we'll wait for Lucene 10 to ship first. >> >>
Re: Atomic updates too slow in Solr 8 vs Solr 7
So I ran the test to index 5 million docs this time (batches of 1000 docs in 15 parallel threads). To eliminate the network overhead and get as accurate a benchmark as possible, I used an AtomicLong to measure the time around the RTG call in DistibutedUpdateProcessor across all calls ( https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.2/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L1416). Did this for both Solr 7.7.2 and Solr 8.11.1 and built the solr-core.jar to replace it in the solr webapp lib. RTG in Solr 8.x is ~10x slower. Here are the numbers (times are in milliseconds): *Solr 7.7.2 *: 2023-06-01 15:39:48.272 WARN (qtp1034094674-24) [ x:techproducts] o.a.s.u.p.LogUpdateProcessorFactory *Total rtg time:7293486* *Solr 8.11.1: *2023-06-01 04:46:24.758 WARN (qtp391506011-71) [ x:techproducts] o.a.s.u.p.LogUpdateProcessorFactory *Total rtg time:72029877* Let me know if I can help further with debugging. At this point I am suspecting the slowness to be on Lucene's side with the findTargetArc() method in FST.java. I am running more analysis on my side in parallel and will share if/when I find more. -Rahul On Thu, Jun 1, 2023 at 3:53 AM Jan Høydahl wrote: > Yes, the numbers should be similar. Thanks for digging in. If there is a > big regression, it could be worthwile running the same test on v8.0, 8.1, > 8.2 etc to plot in what version the regression happened. Perhaps it could > even be scripted :) > > Solr unfortunately do not yet have a comprehensive official nightly > benchmark run that would catch such regressions. But the community is > working on it, there are some tests on http://mostly.cool maintained by > Ishan, but I have no idea how to add new tests.. > > Jan > > > 31. mai 2023 kl. 23:50 skrev Rahul Goswami : > > > > Sure, I can do that. Let me create an index with a few million docs, call > > RTG with a few million iterations on it and note the times between 7.x > and > > 8.x. I assume this should be sufficient (?) > > > > On Wed, May 31, 2023 at 5:19 PM Jan Høydahl > wrote: > > > >> Would be nice to determine whether RTG is orders of magnitude slower in > >> 8.x than 7.x and is the main culprit. Then we could isolate the > testing to > >> RTG only and not involce Atomic Update? > >> > >> Jan > >> > >>> 31. mai 2023 kl. 21:33 skrev Rahul Goswami : > >>> > >>> I don’t have any nested documents. And the results are consistent > across > >>> multiple runs. I tried looking for similar issues in the mailing list, > >> but > >>> couldn’t find anything relevant . So if you do happen to find any JIRAs > >>> addressing it that would be really helpful (thanks!). > >>> > >>> To Jan’s question about RTG taking more time in Solr 8.x, I can say > with > >>> good certainty that this is the case. Although it does look into > >>> transaction logs first, thread dumps suggest that it is the next phase > >>> (when it doesn't find the doc in tlog) which seems to be time > consuming . > >>> It tries to look up the document via the current searcher > >>> (searcher.getFirstMatch() ). Proceeding further in the stack, it is > this > >>> call where many threads are spending time: > >>> > >>> > >> > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.11.1/lucene/core/src/java/org/apache/lucene/codecs/blocktree/SegmentTermsEnum.java#L485 > >>> > >>> Although this call is the same in 7.7.2 and 8.11.1 quite likely > >>> something changed in Lucene's FST.java which is causing the slowness. I > >> am > >>> trying to dig further and might also ask folks on the Lucene mailing > >> list. > >>> Thanks. > >>> > >>> > >>> > >>> On Wed, May 31, 2023 at 11:36 AM Srijan wrote: > >>> > I would love some profiling as well. I know 8.8 or 8.9 had some > >> performance > problems with atomic update but this was later addressed. I cant find > >> the > jira atm though. Also I am on 8.11.1 and atomic update is not an issue > >> for > me. > > By the way, do you happen to have nested docs? > > > On Wed, May 31, 2023, 11:20 Jan Høydahl > wrote: > > > Hi > > > > MMap is most important for searching. Indexing bypasses the cache by > using > > direct IO. > > > > I have noticed slow real time get on Solr 8.x during atomic update > myself. > > Would be interesting with a comparison with profiling. RTG gets the > > document from transaction log I believe? Could there be some RTG > >> changes > in > > 8.x that caused such slowdown? > > > > Jan Høydahl > > > >> 31. mai 2023 kl. 16:57 skrev Rahul Goswami : > >> > >> Thanks for the response Shawn. We are using Windows server with > >> pretty > > huge > >> indexes (multiple TB cores). With Mmap, I have observed that the > machine > >> just completely freezes with high CPU and memory usage to a point > >> where > > it > >> becomes impossible to even connect to it. Sim
Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)
Hi, I am running into the following issue while creating a custom docker image on top of the official Solr docker image (9.2.1). The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key Because of this, it further fails with the following error message: http://archive.ubuntu.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C It fails on the RUN step. What might be the problem? Is this a genuine issue with the base image? FROM solr:9.2.1 USER root RUN apt-get update && \ apt-get upgrade --yes && \ apt-get install --yes locales
Re: Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)
Maybe apt-get was outdated by the apt command. Can you post the error message? On Thu, 1 Jun 2023 at 18:39, gnandre wrote: > Hi, > > I am running into the following issue while creating a custom docker image > on top of the official Solr docker image (9.2.1). > > The key(s) in the keyring > /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the > file is not readable by user '_apt' executing apt-key > > Because of this, it further fails with the following error message: > > http://archive.ubuntu.com/ubuntu jammy InRelease: The following > signatures > couldn't be verified because the public key is not available: NO_PUBKEY > 871920D1991BC93C > > It fails on the RUN step. What might be the problem? Is this a genuine > issue with the base image? > > FROM solr:9.2.1 > > USER root > > RUN apt-get update && \ > apt-get upgrade --yes && \ > apt-get install --yes locales > -- Vincenzo D'Amore
Re: Atomic updates too slow in Solr 8 vs Solr 7
Hi, Next step I believe is for you to open a Solr JIRA issue for this regression, as it is obviously an issue. Then we can use that JIRA to coordinate, even if the fix ends up being in Lucene code. Once we have a Solr JIRA, it may make sense to open a mail thread on users@lucene list to get some Lucene expertice involved in digging. Jan > 1. jun. 2023 kl. 18:15 skrev Rahul Goswami : > > So I ran the test to index 5 million docs this time (batches of 1000 docs > in 15 parallel threads). To eliminate the network overhead and get as > accurate a benchmark as possible, I used an AtomicLong to measure the time > around the RTG call in DistibutedUpdateProcessor across all calls ( > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.2/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L1416). > Did this for both Solr 7.7.2 and Solr 8.11.1 and built the solr-core.jar to > replace it in the solr webapp lib. > > RTG in Solr 8.x is ~10x slower. Here are the numbers (times are in > milliseconds): > > *Solr 7.7.2 *: 2023-06-01 15:39:48.272 WARN (qtp1034094674-24) [ > x:techproducts] o.a.s.u.p.LogUpdateProcessorFactory *Total rtg time:7293486* > *Solr 8.11.1: *2023-06-01 04:46:24.758 WARN (qtp391506011-71) [ > x:techproducts] o.a.s.u.p.LogUpdateProcessorFactory *Total rtg > time:72029877* > > Let me know if I can help further with debugging. At this point I am > suspecting the slowness to be on Lucene's side with the findTargetArc() > method in FST.java. I am running more analysis on my side in parallel and > will share if/when I find more. > > -Rahul > > > > > > > > > > > > On Thu, Jun 1, 2023 at 3:53 AM Jan Høydahl wrote: > >> Yes, the numbers should be similar. Thanks for digging in. If there is a >> big regression, it could be worthwile running the same test on v8.0, 8.1, >> 8.2 etc to plot in what version the regression happened. Perhaps it could >> even be scripted :) >> >> Solr unfortunately do not yet have a comprehensive official nightly >> benchmark run that would catch such regressions. But the community is >> working on it, there are some tests on http://mostly.cool maintained by >> Ishan, but I have no idea how to add new tests.. >> >> Jan >> >>> 31. mai 2023 kl. 23:50 skrev Rahul Goswami : >>> >>> Sure, I can do that. Let me create an index with a few million docs, call >>> RTG with a few million iterations on it and note the times between 7.x >> and >>> 8.x. I assume this should be sufficient (?) >>> >>> On Wed, May 31, 2023 at 5:19 PM Jan Høydahl >> wrote: >>> Would be nice to determine whether RTG is orders of magnitude slower in 8.x than 7.x and is the main culprit. Then we could isolate the >> testing to RTG only and not involce Atomic Update? Jan > 31. mai 2023 kl. 21:33 skrev Rahul Goswami : > > I don’t have any nested documents. And the results are consistent >> across > multiple runs. I tried looking for similar issues in the mailing list, but > couldn’t find anything relevant . So if you do happen to find any JIRAs > addressing it that would be really helpful (thanks!). > > To Jan’s question about RTG taking more time in Solr 8.x, I can say >> with > good certainty that this is the case. Although it does look into > transaction logs first, thread dumps suggest that it is the next phase > (when it doesn't find the doc in tlog) which seems to be time >> consuming . > It tries to look up the document via the current searcher > (searcher.getFirstMatch() ). Proceeding further in the stack, it is >> this > call where many threads are spending time: > > >> https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.11.1/lucene/core/src/java/org/apache/lucene/codecs/blocktree/SegmentTermsEnum.java#L485 > > Although this call is the same in 7.7.2 and 8.11.1 quite likely > something changed in Lucene's FST.java which is causing the slowness. I am > trying to dig further and might also ask folks on the Lucene mailing list. > Thanks. > > > > On Wed, May 31, 2023 at 11:36 AM Srijan wrote: > >> I would love some profiling as well. I know 8.8 or 8.9 had some performance >> problems with atomic update but this was later addressed. I cant find the >> jira atm though. Also I am on 8.11.1 and atomic update is not an issue for >> me. >> >> By the way, do you happen to have nested docs? >> >> >> On Wed, May 31, 2023, 11:20 Jan Høydahl >> wrote: >> >>> Hi >>> >>> MMap is most important for searching. Indexing bypasses the cache by >> using >>> direct IO. >>> >>> I have noticed slow real time get on Solr 8.x during atomic update >> myself. >>> Would be interesting with a comparison with profiling. RTG gets the >>> document from transaction log I believe? Could there
Re: Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)
I cannot reproduce this. Copied your Dockerfile and all succeeds (MacBook). You should consider adding USER solr as the last line of your file. Jan > 1. jun. 2023 kl. 18:38 skrev gnandre : > > Hi, > > I am running into the following issue while creating a custom docker image > on top of the official Solr docker image (9.2.1). > > The key(s) in the keyring > /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the > file is not readable by user '_apt' executing apt-key > > Because of this, it further fails with the following error message: > > http://archive.ubuntu.com/ubuntu jammy InRelease: The following signatures > couldn't be verified because the public key is not available: NO_PUBKEY > 871920D1991BC93C > > It fails on the RUN step. What might be the problem? Is this a genuine > issue with the base image? > > FROM solr:9.2.1 > > USER root > > RUN apt-get update && \ >apt-get upgrade --yes && \ >apt-get install --yes locales
Re: Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)
Thanks for the replies. @Vincenzo: Please find the full error at the bottom. @Jan: I am using Debian 10 (Buster). I am switching to the solr user at the end of the dockerfile. I did not paste the whole dockerfile here but only the relevant part. It fails on that RUN instruction and does not go any further. So, I think the later part in the Dockerfile becomes irrelevant for this error. 37597 [INFO] DOCKER> Step 1/32 : FROM solr:9.2.1 37612 [INFO] DOCKER> 37615 [INFO] DOCKER> ---> 1834fb31cef8 37615 [INFO] DOCKER> Step 2/32 : USER root 37615 [INFO] DOCKER> 39769 [INFO] DOCKER> ---> Running in 923bd0b0f3c8 39959 [INFO] DOCKER> Removing intermediate container 923bd0b0f3c8 39959 [INFO] DOCKER> ---> 2b5502f09462 39967 [INFO] DOCKER> Step 3/32 : RUN apt-get update && apt-get upgrade --yes && apt-get install --yes locales 39967 [INFO] DOCKER> 40054 [INFO] DOCKER> ---> Running in b20c1b8bebbc 40416 [INFO] DOCKER> Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 40549 [INFO] DOCKER> Err:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 40550 [INFO] DOCKER> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C 40562 [INFO] DOCKER> Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB] 41157 [INFO] DOCKER> Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 41251 [INFO] DOCKER> Err:2 http://archive.ubuntu.com/ubuntu jammy InRelease 41252 [INFO] DOCKER> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C 41361 [INFO] DOCKER> Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB] 41415 [INFO] DOCKER> Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 41416 [INFO] DOCKER> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C 41554 [INFO] DOCKER> Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 41554 [INFO] DOCKER> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C 41556 [INFO] DOCKER> Reading package lists... 41578 [INFO] DOCKER> 41589 [INFO] DOCKER> [91mW 41589 [INFO] DOCKER> [91m: 41593 [INFO] DOCKER> [91mhttp:// security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key. 41601 [INFO] DOCKER> [91m 41605 [INFO] DOCKER> [91mW 41605 [INFO] DOCKER> [91m: 41605 [INFO] DOCKER> [91mhttp:// security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key. 41605 [INFO] DOCKER> [91m 41606 [INFO] DOCKER> [91mW 41606 [INFO] DOCKER> [91m: 41606 [INFO] DOCKER> [91mGPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C41606 [INFO] DOCKER> [91m 41606 [INFO] DOCKER> [91mE 41607 [INFO] DOCKER> [91m: 41607 [INFO] DOCKER> [91mThe repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed. 41607 [INFO] DOCKER> [91m 41607 [INFO] DOCKER> [91mW 41607 [INFO] DOCKER> [91m: 41608 [INFO] DOCKER> [91mhttp:// archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key. 41613 [INFO] DOCKER> [91m 41613 [INFO] DOCKER> [91mW 41613 [INFO] DOCKER> [91m: 41613 [INFO] DOCKER> [91mhttp:// archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key. 41613 [INFO] DOCKER> [91m 41617 [INFO] DOCKER> [91mW 41617 [INFO] DOCKER> [91m: 41617 [INFO] DOCKER> [91mGPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C 41623 [INFO] DOCKER> [91m E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed. W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key. W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key. W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C E: The rep
Re: Atomic updates too slow in Solr 8 vs Solr 7
> Solr unfortunately do not yet have a comprehensive official nightly benchmark run that would catch such regressions. But the community is working on it, there are some tests on http://mostly.cool maintained by Ishan, but I have no idea how to add new tests.. I'll publish steps to add new tests soon. For now, check out solr-bench from GitHub.com/fullstorydev/solr-bench Run an existing test as follows: mvn clean compile assembly:single ./stress.sh -c suites/cluster-test.json Results will be written out in suites/results/. On Fri, 2 Jun, 2023, 2:44 am Jan Høydahl, wrote: > Hi, > > Next step I believe is for you to open a Solr JIRA issue for this > regression, as it is obviously an issue. > Then we can use that JIRA to coordinate, even if the fix ends up being in > Lucene code. > > Once we have a Solr JIRA, it may make sense to open a mail thread on > users@lucene list to get some Lucene expertice involved in digging. > > Jan > > > 1. jun. 2023 kl. 18:15 skrev Rahul Goswami : > > > > So I ran the test to index 5 million docs this time (batches of 1000 docs > > in 15 parallel threads). To eliminate the network overhead and get as > > accurate a benchmark as possible, I used an AtomicLong to measure the > time > > around the RTG call in DistibutedUpdateProcessor across all calls ( > > > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.2/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L1416 > ). > > Did this for both Solr 7.7.2 and Solr 8.11.1 and built the solr-core.jar > to > > replace it in the solr webapp lib. > > > > RTG in Solr 8.x is ~10x slower. Here are the numbers (times are in > > milliseconds): > > > > *Solr 7.7.2 *: 2023-06-01 15:39:48.272 WARN (qtp1034094674-24) [ > > x:techproducts] o.a.s.u.p.LogUpdateProcessorFactory *Total rtg > time:7293486* > > *Solr 8.11.1: *2023-06-01 04:46:24.758 WARN (qtp391506011-71) [ > > x:techproducts] o.a.s.u.p.LogUpdateProcessorFactory *Total rtg > > time:72029877* > > > > Let me know if I can help further with debugging. At this point I am > > suspecting the slowness to be on Lucene's side with the findTargetArc() > > method in FST.java. I am running more analysis on my side in parallel and > > will share if/when I find more. > > > > -Rahul > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 1, 2023 at 3:53 AM Jan Høydahl > wrote: > > > >> Yes, the numbers should be similar. Thanks for digging in. If there is a > >> big regression, it could be worthwile running the same test on v8.0, > 8.1, > >> 8.2 etc to plot in what version the regression happened. Perhaps it > could > >> even be scripted :) > >> > >> Solr unfortunately do not yet have a comprehensive official nightly > >> benchmark run that would catch such regressions. But the community is > >> working on it, there are some tests on http://mostly.cool maintained by > >> Ishan, but I have no idea how to add new tests.. > >> > >> Jan > >> > >>> 31. mai 2023 kl. 23:50 skrev Rahul Goswami : > >>> > >>> Sure, I can do that. Let me create an index with a few million docs, > call > >>> RTG with a few million iterations on it and note the times between 7.x > >> and > >>> 8.x. I assume this should be sufficient (?) > >>> > >>> On Wed, May 31, 2023 at 5:19 PM Jan Høydahl > >> wrote: > >>> > Would be nice to determine whether RTG is orders of magnitude slower > in > 8.x than 7.x and is the main culprit. Then we could isolate the > >> testing to > RTG only and not involce Atomic Update? > > Jan > > > 31. mai 2023 kl. 21:33 skrev Rahul Goswami : > > > > I don’t have any nested documents. And the results are consistent > >> across > > multiple runs. I tried looking for similar issues in the mailing > list, > but > > couldn’t find anything relevant . So if you do happen to find any > JIRAs > > addressing it that would be really helpful (thanks!). > > > > To Jan’s question about RTG taking more time in Solr 8.x, I can say > >> with > > good certainty that this is the case. Although it does look into > > transaction logs first, thread dumps suggest that it is the next > phase > > (when it doesn't find the doc in tlog) which seems to be time > >> consuming . > > It tries to look up the document via the current searcher > > (searcher.getFirstMatch() ). Proceeding further in the stack, it is > >> this > > call where many threads are spending time: > > > > > > >> > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.11.1/lucene/core/src/java/org/apache/lucene/codecs/blocktree/SegmentTermsEnum.java#L485 > > > > Although this call is the same in 7.7.2 and 8.11.1 quite likely > > something changed in Lucene's FST.java which is causing the > slowness. I > am > > trying to dig further and might also ask folks on the Lucene mailing > list. > > Thanks. > > > > > > > > On Wed, May 31, 2023 at 11:36
Re: Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)
: @Jan: I am using Debian 10 (Buster). I am switching to the solr user at the based on the logs you posted, you seem to be building the docker container inside of some other build system... 1) are you certain you know exactly what command/options your build system is using to invoke 'docker build' ? 2) have you tried just building manually on the command line? 3) are you certain you are using the 'solr:9.2.1' from docker hub (and not sme other weird locally built version) ... have you tried 'docker build --pull ...' 4) do you have any firewall rules that might be allowing apt to talk to the ubuntu distribution servers, but not the keyserver? 5) This particular set of error msgs strike me as evience of something very, very wrong happening with your docker build, since the files in question is world readable inside the solr docker image... : 41593 [INFO] DOCKER> [91mhttp:// : security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in : the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are : ignored as the file is not readable by user '_apt' executing apt-key. $ docker run --rm -it --entrypoint /bin/bash solr:9.2.1 -c 'ls -al /etc/apt/trusted.gpg.d/' total 16 drwxr-xr-x 2 root root 4096 Apr 25 14:06 . drwxr-xr-x 8 root root 4096 Apr 25 14:03 .. -rw-r--r-- 1 root root 2794 Mar 26 2021 ubuntu-keyring-2012-cdimage.gpg -rw-r--r-- 1 root root 1733 Mar 26 2021 ubuntu-keyring-2018-archive.gpg 6) As jan mentioned - what are you are trying to do seems to work fine for both of us hossman@slate:~/tmp/dockerbase$ ls -l total 4 -rw-rw-r-- 1 hossman hossman 117 Jun 1 21:08 Dockerfile hossman@slate:~/tmp/dockerbase$ cat Dockerfile FROM solr:9.2.1 USER root RUN apt-get update && \ apt-get upgrade --yes && \ apt-get install --yes locales hossman@slate:~/tmp/dockerbase$ docker build -t tmp-extened-solr . Sending build context to Docker daemon 2.048kB Step 1/3 : FROM solr:9.2.1 9.2.1: Pulling from library/solr 1bc677758ad7: Pull complete 0d0e0ecb256a: Pull complete 212512b6dedf: Pull complete 648d9d544695: Pull complete ae4d1091c493: Pull complete 5848e44a1a2f: Pull complete ddd2db2e437b: Pull complete e68c34067326: Pull complete 9ab7d551d9e8: Pull complete Digest: sha256:a46e8f2cc8b0e3e1a50be289cc0b4718e61fa5a5a745454c2da9e51a0312b612 Status: Downloaded newer image for solr:9.2.1 ---> 1834fb31cef8 Step 2/3 : USER root ---> Running in 3a0727367865 Removing intermediate container 3a0727367865 ---> 0592892b6da1 Step 3/3 : RUN apt-get update && apt-get upgrade --yes && apt-get install --yes locales ---> Running in a25792bbd3ae Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB] Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] ... -Hoss
