I didn't see these responses because they were buried in my clutter folder.
We have 12,541,505 docs for calls, 9,144,862 form fills, 53,838 SMS and 12,752
social leads. These are all a single Solr 9.1 cluster of three nodes with PROD
and UAT all on a single server. As follows:
[cid:image001.png@01DA6A4F.59817D30]
The three nodes are r5.xlarge and we’re not sure if those are large enough. The
documents are not huge, from 1K to 25K each.
samisolrcld.aws01.hibu.int is a load-balancer
The request is
async function getCalls(businessId, limit) {
const config = {
method: 'GET',
url: http://samisolrcld.aws01.hibu.int:8983/solr/calls/select,
params: {
q: `business_id:${businessId} AND call_day:[20230101 TO 20240101}`,
fl: "business_id, call_id, call_day, call_date, dialog_merged,
call_callerno, call_duration, call_status, caller_name, caller_address,
caller_state, caller_city, caller_zip",
rows: limit,
start: 0,
group: true,
"group.main": true,
"group.field": "call_callerno",
sort: "call_day desc"
}
};
//console.log(config);
let rval = [];
while(true) {
try {
//console.log(config.params.start);
const rsp = await axios(config);
if(rsp.data && rsp.data.response) {
let docs = rsp.data.response.docs;
if(docs.length == 0) break;
config.params.start += limit;
rval = rval.concat(docs);
}
} catch (err) {
console.log("Error: " + err.message);
}
}
return rval;
}
You wrote:
Note that EFS is encrypted file system, and stunnel is encrypted transport, so
for each disk read you likely causing:
- read raw encrypted data from disk to memory (at AWS)
- decrypt the disk data in memory (at AWS)
- encrypt the memory data for stunnel transport (at AWS)
- send the data over the wire
- decrypt the data for use by solr. (Hardware you specify)
That's guaranteed to be slow, and worse yet, you have no control at all over
the size or loading of the hardware performing anything but the last step. You
are completely at the mercy of AWS's cost/speed tradeoffs which are unlikely to
be targeting the level of performance usually desired for search disk IO.
This is interesting. I can copy the data to local and try it from there.
Jim Beale
Lead Software Engineer
hibu.com
2201 Renaissance Boulevard, King of Prussia, PA, 19406
Office: 610-879-3864
Mobile: 610-220-3067
-----Original Message-----
From: Gus Heck <gus.h...@gmail.com>
Sent: Sunday, February 25, 2024 9:15 AM
To: users@solr.apache.org
Subject: [EXTERNAL] Re: Is this list alive? I need help
Caution! Attachments and links (urls) can contain deceptive and/or
malicious content.
Hi Jim,
Welcome to the Solr user list, not sure why your are asking about list
liveliness? I don't see prior messages from you?
https://lists.apache.org/list?users@solr.apache.org:lte=1M:jim
Probably the most important thing you haven't told us is the current size of
your indexes. You said 20k/day input, but at the start do you have 0days, 1
day, 10 days, 100 days, 1000 days, or 10000 days (27y) on disk already?
If you are starting from zero, then there is likely a 20x or more growth in the
size of the index between the first and second measurement.. indexes do get
slower with size though you would need fantastically large documents or some
sort of disk problem to explain it that way.
However, maybe you do have huge documents or disk issues since your query time
at time1 is already abysmal? Either you are creating a fantastically expensive
query, or your system is badly overloaded. New systems, properly sized with
moderate sized documents ought to be serving simple queries in tens of
milliseconds.
As others have said it is *critical you show us the entire query request*.
If you are doing something like attempting to return the entire index with
rows=999999, that would almost certainly explain your issues...
How large are your average documents (in terms of bytes)?
Also what version of Solr?
r5.xlarge only has 4 cpu and 32 GB of memory. That's not very large (despite
the name). However since it's unclear what your total index size looks like, it
might be OK.
What are your IOPS constraints with EFS? Are you running out of a quota there?
(bursting mode?)
Note that EFS is encrypted file system, and stunnel is encrypted transport, so
for each disk read you likely causing:
- read raw encrypted data from disk to memory (at AWS)
- decrypt the disk data in memory (at AWS)
- encrypt the memory data for stunnel transport (at AWS)
- send the data over the wire
- decrypt the data for use by solr. (Hardware you specify)
That's guaranteed to be slow, and worse yet, you have no control at all over
the size or loading of the hardware performing anything but the last step. You
are completely at the mercy of AWS's cost/speed tradeoffs which are unlikely to
be targeting the level of performance usually desired for search disk IO.
I'll also echo others and say that it's a bad idea to allow solr instances to
compete for disk IO in any way. I've seen people succeed with setups that use
invisibly provisioned disks, but one typically has to run more hardware to
compensate. Having a shared disk creates competition, and it also creates a
single point of failure partially invalidating the notion of running 3 servers
in cloud mode for high availability. If you can't have more than one disk, then
you might as well run a single node, especially at small data sizes like
20k/day. A single node on well chosen hardware can usually serve tens of
millions of normal sized documents, which would be several years of data for
you. (assuming low query rates, handling high rates of course starts to require
hardware)
Finally, you will want to get away from using single queries as a measurement
of latency. If you care about response time I HIGHLY suggest you watch this
YouTube video on how NOT to measure latency:
https://www.youtube.com/watch?v=lJ8ydIuPFeU
On Fri, Feb 23, 2024 at 6:44 PM Jan Høydahl
<jan....@cominvent.com<mailto:jan....@cominvent.com>> wrote:
> I think EFS is a terribly slow file system to use for Solr, who
> recommended it? :) Better use one EBS per node.
> Not sure if the gradually slower performance is due to EFS though. We
> need to know more about your setup to get a clue. What role does
> stunnel play here? How are you indexing the content etc.
>
> Jan
>
> > 23. feb. 2024 kl. 19:58 skrev Walter Underwood
> > <wun...@wunderwood.org<mailto:wun...@wunderwood.org>>:
> >
> > First, a shared disk is not a good idea. Each node should have its
> > own
> local disk. Solr makes heavy use of the disk.
> >
> > If the indexes are shared, I’m surprised it works at all. Solr is
> > not
> designed to share indexes.
> >
> > Please share the full query string.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org<mailto:wun...@wunderwood.org>
> > http://observer.wunderwood.org/ (my blog)
> >
> >> On Feb 23, 2024, at 10:01 AM, Beale, Jim (US-KOP)
> <jim.be...@hibu.com.INVALID<mailto:jim.be...@hibu.com.INVALID>> wrote:
> >>
> >> I have a Solrcloud installation of three servers on three r5.xlarge
> >> EC2
> with a shared disk drive using EFS and stunnel.
> >>
> >> I have documents coming in about 20000 per day and I am trying to
> perform indexing along with some regular queries and some special
> queries for some new functionality.
> >>
> >> When I just restart Solr, these queries run very fast but over time
> become slower and slower.
> >>
> >> This is typical for the numbers. At time1, the request only took
> >> 2.16
> sec but over night the response took 18.137 sec. That is just typical.
> >>
> >> businessId, all count, reduced count, time1, time2
> >> 7016274253,8433,4769,2.162,18.137
> >>
> >> The same query is so far different. Overnight the Solr servers slow
> down and give terrible response. I don’t even know if this list is alive.
> >>
> >>
> >> Jim Beale
> >> Lead Software Engineer
> >> hibu.com
> >> 2201 Renaissance Boulevard, King of Prussia, PA, 19406
> >> Office: 610-879-3864
> >> Mobile: 610-220-3067
> >>
> >>
> >>
> >> The information contained in this email message, including any
> attachments, is intended solely for use by the individual or entity
> named above and may be confidential. If the reader of this message is
> not the intended recipient, you are hereby notified that you must not
> read, use, disclose, distribute or copy any part of this
> communication. If you have received this communication in error,
> please immediately notify me by email and destroy the original message,
> including any attachments. Thank you.
> **Hibu IT Code:1414593000000**
> >
>
>
--
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)
