Hi All
For some of our collections when we run a collections API DELETE
synchronously followed immediately by a configset API DELETE for the
underlying configset we end up in a messed up state.
I have been unable to reproduce this issue in a test environment, it only
happens on the live production instances inconsistently, so it may be
load/race condition related.
Running a COLSTATUS against the broken collection provides the following
response,
{"responseHeader": {"status": 404,"QTime": 33},"collection_19744": {"
stateFormat": 2,"znodeVersion": 51,"properties": {"autoAddReplicas": "false"
,"maxShardsPerNode": "1","nrtReplicas": "3","pullReplicas": "0","
replicationFactor": "3","router": {"name": "compositeId"},"tlogReplicas":
"0"},"activeShards": 1,"inactiveShards": 0},"error": {"metadata": [
"error-class",
"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException",
"root-error-class",
"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException"],"msg
": "Error from server at
http://solr1.mysooqr2prod-internal:8983/solr/collection_19744_shard1_replica_n4:
Expected mime type application/octet-stream but got text/html.
<html>\n<head>\n<meta http-equiv=\"Content-Type\"
content=\"text/html;charset=utf-8\"/>\n<title>Error 404 Not
Found</title>\n</head>\n<body><h2>HTTP ERROR 404</h2>\n<p>Problem accessing
/solr/collection_19744_shard1_replica_n4/admin/segments. Reason:\n<pre> Not
Found</pre></p>\n</body>\n</html>\n","code": 404}}
The underlying shard data for the collection has been successfully removed
from disk and is not present on any of the solr nodes, the
/collections/collection_19744 node has also been successfully deleted from
Zookeeper, which I tested using the zkcli script. Receiving a `NoNode for
/collections/collection_19744` message.
As the COLSTATUS is broken we cannot delete the associated configset, doing
so results in a "Can not delete ConfigSet as it is currently being used by
collection [collection_19744]" message. Which is false.
Where exactly does the COLSTATUS get its collection meta information, as
the /collections/collection_19744 node is missing in zookeeper?
I want to remove the broken collection metadata so I can remove the
configset and recreate the collection with the original naming.
Thanks
--
Martin Graney
Lead Developer
http://sooqr.com <http://www.sooqr.com/>
http://twitter.com/sooqrcom
Office: +31 (0) 88 766 7700
Mobile: +31 (0) 64 660 8543
--
<https://www.linkedin.com/company/sooqr-com/>
