[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494444#comment-15494444
]
Russell Black commented on SOLR-9521:
-------------------------------------
We are not restarting often. We have been running a large solr 6 cluster (80
shards) for a few days. We have a socketTimeout of 1500 ms on the SolrClient,
a socketTimeout of 1000 ms on the shardHandlerFactory, and we are sending
timeAllowed=1000 with every search. Average response time is 30 ms. We
restarted solr on all the shards for the first time under a live load
yesterday. We restarted them 3 at a time.
After the restart, the cold firstSearcher immediately started taking traffic.
The load on the servers went way up, compared to its pre-restart load, and all
queries were completing in 1000 ms, presumably with partial results. This high
load continued for 20 minutes until finally I switched our site to point to our
old search system. We were later able to get them back online by gradually
transitioning the traffic back to the new system. (I can control what
percentage of searches are directed to the new solr cluster and change it in
real time).
My belief (and I could be wrong) is that the caches could never get populated
after the restart because queries were being aborted early by the timeAllowed
parameter and caches can't populate with partial results. Hence my desire to
properly warm the first searcher.
> Warm first searcher with queries from log file
> ----------------------------------------------
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
> Affects Versions: 6.2
> Reporter: Russell Black
> Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml}<listener event="firstSearcher"
> class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
> <listener event="firstSearcher" class="solrplugin.LogFileWarmer">
> <!-- The arguments shown below are the defaults and could be omitted -->
> <!-- A list of filenames to search. They are searched in the order
> listed, and the lines in each file are searched in reverse
> order, so that the most recent queries are used to warm the
> cache. Searching stops when it has gathered enough queries.
> -->
> <str name="file">logs/solr.log</str>
> <str name="file">logs/solr.log.1</str>
> <str name="file">logs/solr.log.2</str>
> <str name="file">logs/solr.log.3</str>
> <str name="file">logs/solr.log.4</str>
> <str name="file">logs/solr.log.5</str>
> <str name="file">logs/solr.log.6</str>
> <str name="file">logs/solr.log.7</str>
> <str name="file">logs/solr.log.8</str>
> <str name="file">logs/solr.log.9</str>
>
> <!-- The maximum number of queries to use from logs -->
> <int name="count">100</int>
> <!-- The maximum number of log lines to search in order to find <count>
> queries-->
> <int name="maxLines">1000000</int>
> <!-- The regular expression that matches the query string from the log
> line. It expects the query string to be in capture group 1. -->
> <str name="regex">path=/select params=\{([^ ]+)\} </str>
> <!-- if no queries can be found in the log files, fall back to this -->
> <arr name="fallback">
> <lst><str name="q">solr</str><str name="sort">price asc</str></lst>
> <lst><str name="q">rocks</str><str name="sort">weight asc</str></lst>
> </arr>
> <listener>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
