reubent opened a new pull request, #3250: URL: https://github.com/apache/solr/pull/3250
I don't have access to JIRA to create a ticket but please see below. I've posted in the users list but not received any replies to I've attempted to fix myself. <!-- _(If you are a project committer then you may remove some/all of the following template.)_ Before creating a pull request, please file an issue in the ASF Jira system for Solr: * https://issues.apache.org/jira/projects/SOLR For something minor (i.e. that wouldn't be worth putting in release notes), you can skip JIRA. To create a Jira issue, you will need to create an account there first. The title of the PR should reference the Jira issue number in the form: * SOLR-####: <short description of problem or changes> SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on. Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. --> # Description We noticed increasing memory usage over long periods of time in our Solr Clouds. Running some synthetic tests against such showed that it was directly linked to the number of abnormally terminated requests received - either exceeding resource limits or client timeout closed connection. Investigation showed that the length of the response received to the tasks/list endpoints was growing. The resource leak was minimal but the time spent processing them was rising therefore a cleanup mechanism was needed. Further investigation of why they were not being removed showed that when code ran out of time to run the calls to `checkLimitsBefore` would cause them to return before the finaliser that removes the active query from the list runs. This issue only appears to affect 1. Explicitly set query IDs 2. Queries sent to shards which get the node name prepended to the generated query ID when adding to the list but not when being removed # Solution 1. Prevent CancellableQueryTracker from filling with abnormally terminated requests by adding a timeout mechanism and making it configurable 3. Improve logging in that class 4. Add tests for that class 5. Add an additional release for activeQueries when the request has been finished early Additionally there are clearly issues with the cancellation mechanism more generally that I'm not going to tackle now but am more than happy to look at in more depth later. Namely: 1. Multithreaded queries are not cancellable 2. Cancellability is dependent on the cancellation being sent to the same node that created the task to be cancelled. It also seems there's something weird with shard selection going on here too but I've not got to the bottom of it. 3. activeTaskList is again node dependent so the same UUID can be simultaneously sent to two nodes # Tests I've added tests for the class and run nearly half a million "broken" and half a million non broken requests against a cloud of four nodes # Checklist Please review the following and check all that apply: - [X ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [X ] I have developed this patch against the `main` branch. - [X ] I have run `./gradlew check`. - [X ] I have added tests for my changes. - [ n/a] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
