sarkaramrit1993 opened a new pull request, #5124: URL: https://github.com/apache/solr/pull/5124
https://issues.apache.org/jira/browse/SOLR-18525 Creating a core can hang forever if its replica is removed from state.json while `ZkController.register` is running. `register` installs `UnloadCoreOnDeletedWatcher`, and `registerDocCollectionWatcher` runs it inline on the creating thread. The watcher sees the replica is gone and calls `CoreContainer.unload`, which since SOLR-17700 waits on the pending core op for that name. That op is held by the same `CoreContainer.create` call, so it never finishes. Other notification threads then pile up on the synchronized `onStateChanged`. Changes: - The watcher now queues the unload with `CoreContainer.runAsync` instead of calling it directly. The unload waits for the create to release its op, then removes the core. - It only queues once per watcher, and skips the unload if the core is already gone or now belongs to a different coreNodeName. - At shutdown, a rejected task is logged and dropped. Not changed: - The coreNodeName check happens before `unload` takes the op, so a core unloaded and recreated under the same name in that window could still be hit. The old code had the same gap. Closing it needs a new CoreContainer method, so I left it out. - If an unload is queued behind a slow create during shutdown, the async executor can wait up to 60s before interrupting it. Bounded, no hang. Tests: new `CoreCreateReplicaRemovedDeadlockTest`, two cases (collection already watched on the node, and first replica on the node, where the inline call races the async notification). It pauses register with a custom UpdateLog in `recoverFromLog`, since `testing_beforeRegisterInZk` fires before register checks cluster state. Both cases hang without the fix and pass with it. Also ran DeleteReplicaTest, DeleteShardTest, MoveReplicaTest, ReplaceNodeTest, DeleteNodeTest, and `:solr:core:check -x test`. @HoustonPutman this touches the unload path from SOLR-17700, would appreciate a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
