sarkaramrit1993 opened a new pull request, #5161: URL: https://github.com/apache/solr/pull/5161
https://issues.apache.org/jira/browse/SOLR-17895 With the Overseer disabled, every collection and configset command takes a lock under `/distributedapi/collectionlocks` or `/distributedapi/configsetlocks`. The lock itself is ephemeral, but the dirs holding it were created persistent and nothing ever removed them, not even DELETE collection. They pile up for every collection a cluster has ever had. Changes: - Lock dirs are now container znodes, so ZooKeeper removes them once they're empty. - Since a dir can now go away at any time, a lock is created first and the dir chain is only (re)built on `NoNode`, at most 3 attempts. This also does what the old TODO in `doCreateLock` asked for. - After DELETE collection releases its lock, that collection's empty lock dirs are deleted. This cleans up the persistent dirs older versions left behind. Dirs holding a live lock are skipped, and a failure only logs a warning. Not changed: - Old persistent dirs are only removed on a DELETE collection request. Dirs of collections deleted before upgrading, deleted indirectly (routed alias maintenance, REINDEXCOLLECTION), or of deleted configsets stay. It's a one-time leftover since new dirs clean themselves up, and they're safe to remove by hand while no collection commands are running. - Curator's `creatingParentContainersIfNeeded` could replace `makeLockPath`, but it means calling Curator directly from the lock code, so I kept it on `SolrZkClient`. Things to know: - Rolling upgrade: an older node checks the dir exists, then creates its lock in it. If the dir gets swept or cleaned up in between, that older node fails the command with `NoNode`. It's a one round trip window and a retry works. Upgraded nodes retry on their own. - Cost: when the dirs exist, taking a lock is now 1 ZK call instead of 2. When they were swept (idle for about a minute), the first command rebuilds them, a few extra creates. The DELETE cleanup does 3 calls per legacy dir, so a big old collection adds some time to its DELETE, once. Tests: three new tests in `ZkDistributedLockTest` (dirs are containers, legacy cleanup keeps live locks, a parent removed mid-create still gets the lock). The container test fails without the fix, and the race test fails with a single retry. Ran the lock tests plus `CollectionsAPIDistributedZkTest`, `CollectionsAPISolrJTest`, `DeleteReplicaTest` and `TestCollectionAPI` with the Overseer disabled, and `gradlew check -x test`. Also checked on a real Solr that the dirs are swept after a delete, and that planted persistent dirs are removed by DELETE. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
