peterxcli opened a new pull request, #11349: URL: https://github.com/apache/ozone/pull/11349
## What changes were proposed in this pull request? SCM bugs that involve datanode failures, decommission or maintenance, and SCM restarts are hard to find and hard to reproduce. Whether they happen depends on the order in which the node health check, the event handlers, ReplicationManager and the pipeline services see each change. MiniOzoneCluster tests are too slow to try many such orders, and a failed run usually cannot be repeated in the same order. This PR adds `TestScmSimulation`, a deterministic simulation test for SCM. It runs the real `StorageContainerManager` in-process against simulated datanodes: - Simulated datanodes register and heartbeat through `SCMDatanodeProtocolServer`, send container and pipeline reports, and carry out the commands SCM sends back. - SCM is built through `SCMConfigurator` with a simulated clock, seeded random sources and an event queue whose handlers run on the simulation's scheduler. SCM's background services are stopped, and the simulation runs their tasks at the same intervals: the node health check, ReplicationManager, the background pipeline creator, the pipeline scrubber and the decommission monitor. - Everything runs on one thread, so a run is fully determined by its seed: rerunning a seed gives exactly the same trace. Three hours of simulated time take about 1-2 seconds. - During the chaos phase, a client writes to containers while datanodes crash, get partitioned, are decommissioned (one at a time or a whole rack), enter maintenance or join, replicas are lost or corrupted, and SCM restarts. Then the faults stop, every datanode comes back, and SCM gets time to repair the cluster. - Checked during the run: SCM never deletes the last replica, the last healthy replica, or a replica newer than every other healthy one; open containers are on open pipelines; healthy datanodes are in the network topology; no event handler or service throws. - Checked at the end: every container is correctly replicated, every decommission and maintenance finishes, SCM and the datanodes agree on the replicas, and SCM sends no replicate or delete commands for the next 15 simulated minutes. Not simulated: SCM HA, block deletion, EC containers, the container balancer and security. ### Production changes They only let the test inject what it needs. When nothing is injected, SCM behaves as before. - `SCMConfigurator` can set the event queue, the clock, the pipeline ID generator, and the `Random` that ReplicationManager uses to shuffle vulnerable UNHEALTHY replicas. - SCM's clock is passed down to `ContainerManagerImpl`, `PipelineManagerImpl` and the container and pipeline state maps, which record when containers and pipelines are created and change state. Pipelines loaded at startup take their times from SCM's clock instead of `Instant.now()`. - `SCMNodeManager`, `NodeStateManager`, `DatanodeInfo` and `PendingContainerTracker` can take a clock for heartbeat and health check times. The default is still a monotonic clock. - New pipelines get their IDs from a `Supplier<PipelineID>` passed through `PipelineFactory` to each `PipelineProvider`. The default is `PipelineID::randomId`. - `NetworkTopologyImpl` has a constructor that takes a seeded `Random`. - `PipelinePlacementPolicy` is no longer `final`, so that the test can seed it, and `BackgroundPipelineCreator#createPipelines` is package-private so that the test can run it. The `NodeStateManager` constructor change overlaps with #11343 (HDDS-16623), which adds the network topology to it; whichever is merged second needs a small merge. ### Bugs found so far Found by the simulation, or while fixing what it found. Each is linked to HDDS-16627 as "Discovered while testing": - HDDS-16623: Datanode can be missing from NetworkTopology after it recovers from DEAD (#11343) - HDDS-16628: SCMContainerPlacementRackAware gives up before trying the other used node's rack - HDDS-16629: Decommissioning a whole rack in a 2-rack cluster leaves containers under-replicated - HDDS-16630: SCM does not update the network topology when a datanode re-registers with new ports or a new version - HDDS-16631: NetworkTopology chooseRandom returns null early when one rack name is a prefix of another - HDDS-16632: SCM loses a datanode's container list when the datanode re-registers with a new address, ports or version - HDDS-16633: Datanode can go missing from the network topology if it re-registers while SCM is removing it as DEAD The fault that restarts a datanode with new ports stays off until HDDS-16630 is fixed. ### Running it ``` mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation # rerun a failed seed mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos -Dscm.simulation.seed=SEED # try 100 new seeds with 2, 3, 4 or 12 racks mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos -Dscm.simulation.random.seeds=100 -Dscm.simulation.racks=random ``` The trace of a failed run is written to `target/scm-simulation`. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16627 ## How was this patch tested? - New unit tests for the production changes: `TestSCMConfigurator` (new), `TestContainerStateMap`, `TestPipelineStateMap`, `TestNodeStateManager`, `TestPipelineManagerImpl`, `TestRatisUnderReplicationHandler` and `TestNetworkTopologyImpl`. - `TestScmSimulation` runs seeds 1, 2 and 3, a run without faults, and a check that the same seed gives the same trace. It takes about 10 seconds and passes on master. - 100 seeds (2000-2099) with 2, 3, 4 or 12 racks: 8 fail on master. 6 end with a decommission that never finishes and containers left under-replicated (HDDS-16628, HDDS-16629), and 2 hit the DeadNodeHandler crash fixed by #11343 (HDDS-16623). With the fixes for HDDS-16623, HDDS-16628, HDDS-16629, HDDS-16630 and HDDS-16631 applied, all 100 pass. - All `hdds-server-scm` unit tests pass. Recon and the integration test modules compile. Checkstyle, PMD and RAT are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
