We've lowered our autoCommit from 1 min to 3 min and that's help quite a lot
with the number of small segments being constantly merged and has lowered the
overall load on solr. Will continue to monitor. We're tempted to go to 5
minutes, but the size of tlogs then would be a bit uncomfortable (at 3mins
under peak write load they're about 1.5gb each).
On 08/04/2021, 19:31, "Karl Stoney" <karl.sto...@autotrader.co.uk> wrote:
The documents are pretty large yes, 650 fields, circa 20kb/document so at
peak (300/sec) that's circa 6meg/sec. ramBufferSizeMB is 512 so we'd be
averaging 1 segment every 90 seconds (ish)?
> This means you never explicitly commits from the client? But You
autoCommit openSearcher=false every minute to flush transLog, and then
autoSoftCommit every 15min to make changes visible?
Yes, this is correct
> Have you considered using PULL replicas for reading? Then you could
tailor the HW on those servers to only serve reads, and they would replicate
index from leader. You'd have e.g. 2xNRT + 7xPULL.
I did, but we unfortunately rely on RealTimeGet so need NRT.
On 08/04/2021, 15:10, "Jan Høydahl" <jan....@cominvent.com> wrote:
> - Update rate, and how you do commits?
>
> Update rates very throughout the day, but range from 20ops/sec to
300ops/sec. Commits are done using autoCommit on 1 min interval, softCommit on
15min interval.
This means you never explicitly commits from the client? But You
autoCommit openSearcher=false every minute to flush transLog, and then
autoSoftCommit every 15min to make changes visible?
I cannot see why these seeings would cause constant merging, unless you
commit more frequently? Your documents are large so the RAM-buffer will fill up
quickly and cause a flush, perhaps try increasing ramBufferSizeMb will lower
number of flushes and merges.
Have you considered using PULL replicas for reading? Then you could
tailor the HW on those servers to only serve reads, and they would replicate
index from leader. You'd have e.g. 2xNRT + 7xPULL.
Jan
Unless expressly stated otherwise in this email, this e-mail is sent on behalf
of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester,
Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited
is part of the Auto Trader Group Plc group. This email and any files
transmitted with it are confidential and may be legally privileged, and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the sender.
This email message has been swept for the presence of computer viruses.
