We have had a situation where Solr node was in distress due to hard drive being full and the queries became very slow. Since our Solr cluster has two nodes with indexes being fully available on both the nodes, we think that one good solution would be to just stop the Solr instance on a distressed node. That way the other node will continue to serve queries until the distressed node is fixed.
Can folks please share their experiences in such a situation? What are some good ways to handle this? How/when should we detect that the host (linux) is in distress and then stop the solr node? Thanks,